Vision-based unmanned aerial vehicle positioning method and device, and storage medium

Abstract

The application discloses a visual-based unmanned aerial vehicle positioning method and device and a storage medium, which comprises the following steps: collecting high-altitude images through a camera carried by an unmanned aerial vehicle, and preprocessing the high-altitude images to obtain preprocessed high-altitude images; performing feature point selection on the preprocessed high-altitude images to obtain a target feature point array; performing optical flow calculation and displacement vector secondary processing based on the target feature point array to obtain an overall displacement vector of a current frame; performing self-rotation state judgment on the unmanned aerial vehicle; performing emergency stop state judgment on the unmanned aerial vehicle under the condition that the unmanned aerial vehicle is in a non-self-rotation state; and performing yaw correction and accumulation based on the overall displacement vector of the current frame to obtain the current position of the unmanned aerial vehicle. By introducing the self-rotation and emergency stop judgment mechanisms, errors are avoided, and the errors caused by the displacement vectors of the feature points in different regions are corrected, thereby further improving the positioning accuracy of the unmanned aerial vehicle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle positioning, in particular to an unmanned aerial vehicle positioning method and device based on vision and a storage medium. BACKGROUND

[0002] The existing technology adopts image recognition to first identify people or ground vehicles, and then performs key point detection on other areas by using ORB algorithm, and then uses optical flow algorithm to track the key points to inversely deduce the movement of the unmanned aerial vehicle. First, the required computing power of the image recognition and the ORB algorithm detection is not small, which is not suitable for the working environment of the unmanned aerial vehicle on-board platform. Second, because the unmanned aerial vehicle is located at a high altitude with a large image taking range, the covered range is large, and the ground objects are complex, which cannot guarantee that all moving units can be completely identified. Third, compared with static objects, the edges of moving objects are more likely to be identified as key points. Once the object identification fails, too many false key points will introduce a lot of false displacement information. Finally, although the gimbal camera is used, the shaking range is greatly reduced, but when the unmanned aerial vehicle is suddenly stopped, a large angle of elevation will occur, causing the gimbal to tilt. Because the obtained image is two-dimensional, when the camera tilts, the displacement information measured will be greatly distorted, resulting in poor positioning accuracy of the unmanned aerial vehicle. SUMMARY

[0003] The main purpose of the present application is to provide an unmanned aerial vehicle positioning method and device based on vision and a storage medium, which aims to solve the technical problem of poor positioning accuracy of the existing unmanned aerial vehicle.

[0004] To achieve the above purpose, the present application provides an unmanned aerial vehicle positioning method based on vision, which comprises:

[0005] An aerial image is collected by a camera carried by the unmanned aerial vehicle, and the aerial image is preprocessed to obtain a preprocessed aerial image, wherein the preprocessing at least includes gray scale conversion, image equalization and noise reduction operation;

[0006] Feature point selection is performed on the preprocessed aerial image to obtain a target feature point array;

[0007] Based on the target feature point array, optical flow calculation and displacement vector secondary processing are performed to obtain the overall displacement vector of the current frame;

[0008] The self-rotation state of the unmanned aerial vehicle is judged to obtain a first judgment result;

[0009] In the case that the first judgment result is that the unmanned aerial vehicle is in a non-self-rotation state, the sudden stop state of the unmanned aerial vehicle is judged to obtain a second judgment result;

[0010] In a case where the second determination result is that the UAV is in a non-emergency stop state, a yaw correction is performed based on the overall displacement vector of the current frame and is accumulated to obtain a current position of the UAV.

[0011] In an embodiment, the feature point selection on the preprocessed high-altitude image to obtain a target feature point array comprises:

[0012] A preset number of rows and a preset number of columns of candidate feature points are selected in the preprocessed high-altitude image based on a preset coordinate point and according to a preset step size, wherein the preset coordinate point is a center point of the image, the preset step size is adjusted according to the resolution of the preprocessed high-altitude image, the preset number of rows is determined according to the height and the preset step size of the preprocessed high-altitude image, and the preset number of columns is determined according to the height and the width of the preprocessed high-altitude image.

[0013] A feature descriptor calculation is performed on the candidate feature points to obtain a feature descriptor of each candidate feature point.

[0014] A feature matching is performed based on the feature descriptors to obtain a matched feature point pair.

[0015] A filtering is performed according to the geometric relationship between the matched feature point pairs to obtain a target feature point array, wherein the feature points in the target feature point array are uniformly distributed in the preprocessed high-altitude image.

[0016] In an embodiment, the optical flow calculation and the displacement vector secondary processing based on the target feature point array to obtain an overall displacement vector of the current frame comprises:

[0017] A Lucas-Kanade optical flow algorithm is used to perform an optical flow calculation on the target feature point array to obtain an optical flow vector of each target feature point array from a previous frame to a current frame.

[0018] A displacement vector of each target feature point array is calculated according to the direction and size of the optical flow vector.

[0019] A secondary processing is performed on the displacement vectors of the target feature point arrays to obtain an overall displacement vector of the current frame.

[0020] In an embodiment, the secondary processing on the displacement vectors of the target feature point arrays to obtain an overall displacement vector of the current frame comprises:

[0021] A horizontal component of the displacement vector of each target feature point array is extracted, and an average horizontal component and a horizontal component standard deviation are determined based on the horizontal component.

[0022] screening the displacement vectors of each of the target feature point arrays based on the average horizontal component and the horizontal component standard deviation, eliminating displacement vectors whose horizontal components are more than a preset multiple of the horizontal component standard deviation away from the average horizontal component, to obtain displacement vectors after a first screening;

[0023] extracting the vertical components of the displacement vectors after the first screening, and determining an average vertical component and a vertical component standard deviation based on the vertical components;

[0024] screening the displacement vectors after the first screening again based on the average vertical component and the vertical component standard deviation, eliminating displacement vectors whose vertical components are more than a preset multiple of the vertical component standard deviation away from the average vertical component, to obtain displacement vectors after a second screening;

[0025] performing average value calculation based on the displacement vectors after the second screening, to obtain overall displacement vectors of the current frame.

[0026] In an embodiment, the spinning state judgment on the unmanned aerial vehicle to obtain a first judgment result comprises:

[0027] performing quadrant division on the high-altitude image to obtain high-altitude images of four quadrants;

[0028] respectively calculating average displacement vectors in the high-altitude images of the four quadrants;

[0029] judging whether the unmanned aerial vehicle is in a spinning state according to the signs of the horizontal direction components and the signs of the vertical direction components of the average displacement vectors in the high-altitude images of the four quadrants;

[0030] if the signs of the horizontal direction components and the signs of the vertical direction components of the average displacement vectors in the high-altitude images of the four quadrants satisfy a preset condition and the duration exceeds a preset time threshold, determining that the first judgment result is that the unmanned aerial vehicle is in a spinning state;

[0031] if the signs of the horizontal direction components and the signs of the vertical direction components of the average displacement vectors in the high-altitude images of the four quadrants do not satisfy the preset condition and / or the duration does not exceed the preset time threshold, determining that the first judgment result is that the unmanned aerial vehicle is in a non-spinning state.

[0032] In an embodiment, the spinning state judgment on the unmanned aerial vehicle to obtain a first judgment result comprises:

[0033] performing region division on the high-altitude image to obtain a left region and a right region in the case that the first judgment result is that the unmanned aerial vehicle is in a non-spinning state;

[0034] obtain a displacement vector matrix, and perform element extraction and processing on the left side region and the right side region based on the displacement vector matrix to obtain an average displacement vector of the left side region and an average displacement vector of the right side region;

[0035] If the difference between the average displacement vector of the left side region and the average displacement vector of the right side region exceeds a preset threshold, it is determined that the second judgment result is that the UAV is in an emergency stop state.

[0036] If the difference between the average displacement vector of the left side region and the average displacement vector of the right side region does not exceed the preset threshold, it is determined that the second judgment result is that the UAV is in a non-emergency stop state.

[0037] In an embodiment, in the case that the second judgment result is that the UAV is in a non-emergency stop state, the yaw correction is performed based on the overall displacement vector of the current frame and is accumulated to obtain the current position of the UAV, which comprises:

[0038] In the case that the second judgment result is that the UAV is in a non-emergency stop state, the yaw angle is calculated based on the overall displacement vector of the current frame to obtain the yaw angle.

[0039] The overall displacement vector of the current frame is corrected according to the yaw angle and a reference displacement vector to obtain a corrected overall displacement vector, wherein the reference displacement vector is a fixed reference value obtained by optical flow calculation and displacement vector processing according to a preset feature point array.

[0040] The current position of the UAV is determined based on the corrected overall displacement vector.

[0041] In an embodiment, the yaw angle is calculated based on the overall displacement vector of the current frame to obtain the yaw angle, which comprises:

[0042] The heading angle offset in the vertical direction and the heading angle offset in the horizontal direction are determined based on the overall displacement vector of the current frame.

[0043] The interference displacement vector is determined based on the heading angle offset in the vertical direction and the heading angle offset in the horizontal direction.

[0044] The yaw angle is determined based on the interference displacement vector.

[0045] In addition, in order to achieve the above-mentioned purpose, the present application further provides a UAV positioning device based on vision, which comprises:

[0046] A preprocessing module is configured to collect high-altitude images by a camera carried by a UAV, and to perform preprocessing on the high-altitude images to obtain preprocessed high-altitude images, wherein the preprocessing at least includes grayscale conversion, image equalization and noise reduction operation.

[0047] a selection module configured to select feature points from the preprocessed high-altitude image to obtain a target feature point array, wherein the feature points in the target feature point array are uniformly distributed in the preprocessed high-altitude image;

[0048] a screening module configured to perform optical flow calculation and displacement vector secondary processing based on the target feature point array to obtain an overall displacement vector of the current frame;

[0049] a judgment module configured to judge a self-rotation state of the unmanned aerial vehicle to obtain a first judgment result;

[0050] The judgment module is further configured to, when the first judgment result indicates that the unmanned aerial vehicle is in a non-self-rotation state, judge an emergency stop state of the unmanned aerial vehicle to obtain a second judgment result.

[0051] a correction module configured to, when the second judgment result indicates that the unmanned aerial vehicle is in a non-emergency stop state, perform yaw correction and accumulation based on the overall displacement vector of the current frame to obtain a current position of the unmanned aerial vehicle.

[0052] In addition, to achieve the above object, the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the method as described above.

[0053] In addition, to achieve the above object, the present application further provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the method as described above.

[0054] In addition, to achieve the above object, the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the method as described above.

[0055] One or more technical solutions provided in the application, through the camera carried by the unmanned aerial vehicle, high-altitude images are collected, and the high-altitude images are preprocessed to obtain preprocessed high-altitude images, wherein the preprocessing at least includes gray scale conversion, image equalization and noise reduction operation; feature point selection is performed on the preprocessed high-altitude images to obtain a target feature point array; based on the target feature point array, optical flow calculation and displacement vector secondary processing are performed to obtain the overall displacement vector of the current frame; the self-rotation state of the unmanned aerial vehicle is judged to obtain a first judgment result; in the case that the first judgment result is that the unmanned aerial vehicle is in a non-self-rotation state, the emergency stop state of the unmanned aerial vehicle is judged to obtain a second judgment result; in the case that the second judgment result is that the unmanned aerial vehicle is in a non-emergency stop state, the yaw correction and accumulation are performed based on the overall displacement vector of the current frame to obtain the current position of the unmanned aerial vehicle. Through the above-mentioned manner, by introducing the self-rotation and emergency stop judgment mechanism, the error is avoided to be introduced, and then the error caused by the displacement vector of the feature points in different regions is corrected, and the positioning accuracy of the unmanned aerial vehicle is further improved. BRIEF DESCRIPTION OF DRAWINGS

[0056] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and serve to explain the principles of the application together with the specification.

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, below will briefly introduce the drawings needed to be used in the embodiments or prior art description, and obviously, for those skilled in the art, without creative labor, other drawings can also be obtained according to these drawings.

[0058] Figure 1 The flowchart provided by the first embodiment of the visual-based unmanned aerial vehicle positioning method of the application;

[0059] Figure 2 The average displacement direction diagram of the clockwise rotation of the unmanned aerial vehicle (counterclockwise rotation of the image) provided by the first embodiment of the visual-based unmanned aerial vehicle positioning method of the application;

[0060] Figure 3 The flowchart provided by the second embodiment of the visual-based unmanned aerial vehicle positioning method of the application;

[0061] Figure 4 The diagram of the image screen divided into left and right sides provided by the first embodiment of the visual-based unmanned aerial vehicle positioning method of the application;

[0062] Figure 5 The diagram of the displacement vector of each feature point when there is no yaw provided by the first embodiment of the visual-based unmanned aerial vehicle positioning method of the application;

[0063] Figure 6 The schematic diagram of the displacement vector of different feature points in the yawing state of the embodiment of the visual-based UAV positioning method of the present application;

[0064] Figure 7 The schematic diagram of the x component of the displacement vector in the A region about the X axis symmetry of the embodiment of the visual-based UAV positioning method of the present application;

[0065] Figure 8 The schematic diagram of the image screen divided into upper and lower sides of the embodiment of the visual-based UAV positioning method of the present application;

[0066] Figure 9 The schematic diagram of the module structure of the visual-based UAV positioning device of the embodiment of the present application;

[0067] Figure 10 The schematic diagram of the device structure of the hardware running environment of the visual-based UAV positioning device of the embodiment of the present application.

[0068] Explanation of the reference numerals:

[0069] 1001 - processing device, 1002 - ROM, 1003 - storage device, 1004 - RAM, 1005 - bus, 1006 - I / O interface, 1007 - input device, 1008 - output device, 1009 - communication device.

[0070] The purpose of the present application, the functional characteristics and advantages will be further explained in conjunction with the embodiments, with reference to the accompanying drawings. DETAILED DESCRIPTION

[0071] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0072] In order to better understand the technical solutions of the present application, the specific embodiments will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0073] The main solution of the embodiment of the application is: collecting a high-altitude image by a camera carried by a UAV, and pre-processing the high-altitude image to obtain a pre-processed high-altitude image, wherein the preprocessing at least includes gray conversion, image equalization and noise reduction operation; performing feature point selection on the pre-processed high-altitude image to obtain a target feature point array; performing optical flow calculation and displacement vector secondary processing based on the target feature point array to obtain an overall displacement vector of a current frame; performing self-rotation state judgment on the UAV to obtain a first judgment result; in the case that the first judgment result is that the UAV is in a non-self-rotation state, performing emergency stop state judgment on the UAV to obtain a second judgment result; in the case that the second judgment result is that the UAV is in a non-emergency stop state, performing yaw correction and accumulation based on the overall displacement vector of the current frame to obtain a current position of the UAV.

[0074] The existing technology mainly adopts image recognition to recognize a person or a ground vehicle first, and then performs key point detection on other regions by using an ORB algorithm, and then uses an optical flow algorithm to track the key points to deduce the movement of the UAV itself, and the positioning accuracy of the UAV is poor.

[0075] The application provides a solution, which introduces a self-rotation and emergency stop judgment mechanism to avoid introducing errors, and then corrects the errors caused by yaw based on the displacement vectors of feature points in different regions, thereby improving the positioning accuracy of the UAV.

[0076] It should be noted that the execution subject of the embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a vision-based UAV positioning device, etc. The embodiment and the following embodiments will be described below taking the vision-based UAV positioning device as an example.

[0077] Based on this, the embodiment of the application provides a vision-based UAV positioning method, which refers to Figure 1 , Figure 1 The flowchart of the first embodiment of the vision-based UAV positioning method of the application is shown in the figure.

[0078] In the embodiment, the vision-based UAV positioning method includes steps S10-S60:

[0079] Step S10: collecting a high-altitude image by a camera carried by a UAV, and pre-processing the high-altitude image to obtain a pre-processed high-altitude image, wherein the preprocessing at least includes gray conversion, image equalization and noise reduction operation.

[0080] It should be noted that the unmanned aerial vehicle uses the camera to capture stable high-altitude images to determine its own motion trajectory, and the most important thing is to capture the overall movement of the image.

[0081] It can be understood that when the unmanned aerial vehicle flies at high altitude, due to the complexity of the environment, the high-altitude images collected may contain a large amount of noise and interference information, which will affect the subsequent feature point selection and optical flow calculation. Therefore, before feature point selection, the high-altitude image needs to be preprocessed to improve the quality and clarity of the image. Gray conversion can convert color images to grayscale images, reduce the amount of image data, and improve processing speed; image equalization can enhance the contrast of the image, making the features in the image more obvious; noise reduction operation can remove noise and interference information in the image, improving the clarity of the image. Through these preprocessing operations, the preprocessed high-altitude image can be obtained, making the picture more stable, while reducing the interference of noise points in the optical flow calculation, providing a good foundation for subsequent feature point selection and optical flow calculation.

[0082] Step S20: Selecting feature points from the preprocessed high-altitude image to obtain a target feature point array.

[0083] It should be noted that feature point selection refers to pre-setting feature points for optical flow calculation in the entire image to form a target feature point array.

[0084] It can be understood that the target feature point array should include the following features: 1. The target feature points in the target feature point array are uniformly distributed in the picture, that is, the feature points are uniformly distributed in the entire picture, and the overall displacement information of the picture is extracted uniformly; 2. The distance between the target feature points in the target feature point array is appropriate, that is, the distance between the two target feature points should be far enough to enable the ground person and the small moving unit to not affect or only affect the optical flow calculation of one target feature point, and the distance should not be too far to ignore too much ground feature information, so the spacing or number of target feature points needs to be balanced between these two mutually exclusive factors; 3. The target feature points in the target feature point array are fixed pixels, that is, the feature points calculated between each frame are the pixel points of the target feature point position, that is, the displacement vector calculated by each optical flow measures the displacement of the fixed pixel. Therefore, the target feature points in the target feature point array cannot move with the current frame calculation pixel points, and will not be lost due to the effective displacement vector obtained by a frame optical flow calculation.

[0085] It is worth noting that after setting the feature point distribution reasonably, the calculation resources are saved while the interference of moving information such as ground personnel or vehicles is greatly reduced.

[0086] In an implementation, step S20 can include: selecting candidate feature points in the preprocessed high-altitude image according to a preset coordinate point as a reference and a preset step length, a preset number of rows, and a preset number of columns, wherein the preset coordinate point is a center point of the image, the preset step length is adjusted according to a resolution of the preprocessed high-altitude image, the preset number of rows is determined according to a height and the preset step length of the preprocessed high-altitude image, and the preset number of columns is determined according to a height and a width of the preprocessed high-altitude image; performing feature descriptor calculation on the candidate feature points to obtain a feature descriptor of each candidate feature point; performing feature matching based on the feature descriptors to obtain matched feature point pairs; and performing screening according to geometric relationships between the matched feature point pairs to obtain a target feature point array, wherein the feature points in the target feature point array are uniformly distributed in the preprocessed high-altitude image.

[0087] It should be noted that the feature point selection rule starts from a preset coordinate point (X, Y), and takes (H / L-1) rows and (W / L-1) columns of candidate feature points with a step length of l , wherein W and H are the width and height of the preprocessed high-altitude image, respectively. The preset coordinate point can be the center point of the image or any other position of the image, which is set according to actual needs. The step length l is adjusted according to the resolution of the image and the density requirement of the feature points to ensure the uniform distribution of the feature points in the image.

[0088] It can be understood that the feature descriptor calculation of the candidate feature points can use SIFT, SURF, or other algorithms to obtain a feature descriptor that can uniquely identify the feature point by statistically analyzing the gradient and direction information of the pixels around the candidate feature point. Feature matching is based on the similarity between the feature descriptors to obtain matched feature point pairs. Finally, according to the geometric relationships between the matched feature point pairs, the false matched feature point pairs are removed to obtain a target feature point array, and the feature points in the target feature point array are uniformly distributed in the preprocessed high-altitude image, which not only saves computing power but also reduces the interference of the ground movable unit. The feature points will be used for subsequent optical flow calculation to achieve accurate positioning of the unmanned aerial vehicle.

[0089] In a specific implementation, when SIFT (Scale-Invariant Feature Transform) is used for feature descriptor calculation, Gaussian blur processing is performed on the image, and gradient values (G x and G y ) of the image are calculated. The gradient calculation formula is as follows:

[0090]

[0091] where I(x, y) is the gray value of the image at position (x, y).

[0092] The gradient direction θ and the magnitude m are calculated:

[0093] θ = atan2(G y ,G x )

[0094]

[0095] The surrounding gradient directions are then counted to form a histogram.

[0096] A 16x16 neighborhood is selected around the key point, and the region is divided into 4x4 sub-regions, and a gradient direction histogram (8 directions) is calculated for each sub-region. The final descriptor is a 4x4x8 = 128-dimensional vector.

[0097] It should be noted that the K-Nearest Neighbor algorithm can be used to match the feature descriptors to obtain the initial matching feature point pairs, and the Euclidean distance ratio between the initial matching feature point pairs is calculated, and the feature point pairs with a Euclidean distance ratio less than a preset threshold are taken as the matching feature point pairs.

[0098] Step S30: optical flow calculation and displacement vector secondary processing based on the target feature point array to obtain the overall displacement vector of the current frame.

[0099] It should be noted that after the target feature point array is set, the optical flow calculation is performed on each pixel point at a preset fixed point position. Because the target feature point array is not a high-quality feature point calculated by feature point detection, the number of layers, accuracy, and other parameters of the optical flow calculation function need to be increased to ensure the calculation accuracy.

[0100] It can be understood that in the present embodiment, the Lucas-Kanade optical flow calculation algorithm of opencv for a single pixel point is used, which is an open source algorithm for observing the overall movement of an image. The opencv optical flow calculation function can directly configure parameters to improve the number of calculation layers to improve the calculation accuracy of the displacement of a single pixel.

[0101] In a possible implementation, step S30 can include: performing optical flow calculation on the target feature point array using the Lucas-Kanade optical flow algorithm to obtain the optical flow vector of each target feature point array from the previous frame to the current frame; calculating the displacement vector of each target feature point array according to the direction and size of the optical flow vector; and performing secondary processing on the displacement vectors of the target feature point arrays to obtain the overall displacement vector of the current frame.

[0102] It's important to note that optical flow is an algorithm for calculating the motion of objects in an image over time. In computer vision and image processing, optical flow is used to estimate the motion (i.e., velocity) of each pixel in an image, and it is commonly used to analyze dynamic information in video sequences. Its basic assumptions include:

[0103] Brightness Constancy Assumption: This assumes that the brightness on the surface of an object remains constant across adjacent time frames.

[0104]

[0105] in, Indicates the position at time T ( Image brightness at ().

[0106] Small Motion Assumption: This assumption assumes that the motion of an object is relatively small over a short period of time, so as to make a linear approximation of the displacement.

[0107] It is worth noting that the derivation of the optical flow equation is based on the assumptions of constant brightness and small displacement. According to the linear approximation of the Taylor expansion, the following relationship is obtained:

[0108]

[0109] Where I(x,y,t) is the brightness value of the image at position (x,y) and time (t), and U and V are the horizontal and vertical optical flow components at position (x,y) in the image, respectively. These are the partial derivatives of image brightness I with respect to space and time, respectively.

[0110] The optical flow equation represents the spatial and temporal changes in the displacement of a pixel in an image. The goal is to estimate the optical flow vector for each pixel by solving this equation.

[0111] It is worth noting that optical flow calculation is a typical inverse problem; it is not a univariate equation but contains multiple unknowns (the horizontal and vertical optical flow components (u) and (v) for each pixel). Therefore, optical flow calculation usually requires additional assumptions or constraints to solve.

[0112] The Lucas-Kanade method is another widely used optical flow computation method, which assumes that the optical flow in a local region of an image is constant, and the optical flow of each pixel can be calculated by the brightness change of the pixels in the neighborhood. It uses the gradient information of the local region of the image to estimate the optical flow:

[0113]

[0114] where, , , , (Ix, Iy, It) are the partial derivatives of the image brightness function I(x, y, t) with respect to (x, y, t).

[0115] The Lucas-Kanade algorithm can estimate the horizontal and vertical optical flow components (u) and (v) for each pixel by solving this equation set.

[0116] In an implementation, the displacement vectors of the target feature point arrays are processed twice to obtain the overall displacement vector of the current frame, including: extracting the horizontal components of the displacement vectors of the target feature point arrays, and determining the average horizontal component and the horizontal component standard deviation based on the horizontal components; screening the displacement vectors of the target feature point arrays based on the average horizontal component and the horizontal component standard deviation, removing displacement vectors whose horizontal components are more than a preset multiple of the horizontal component standard deviation away from the average horizontal component, to obtain displacement vectors after the first screening; extracting the vertical components of the displacement vectors after the first screening, and determining the average vertical component and the vertical component standard deviation based on the vertical components; screening the displacement vectors after the first screening again based on the average vertical component and the vertical component standard deviation, removing displacement vectors whose vertical components are more than a preset multiple of the vertical component standard deviation away from the average vertical component, to obtain displacement vectors after the second screening; and performing average value calculation based on the displacement vectors after the second screening to obtain the overall displacement vector of the current frame.

[0117] It should be noted that in this embodiment, the displacement vectors calculated between each frame are stored in an array, and two standard deviation screening processes are performed.

[0118] The X components of each displacement vector are extracted first, the average of the X components is calculated, and then the standard deviation is calculated. Then, the displacement vectors whose X components are more than A times the standard deviation away from the average are removed, where A is an adjustable natural number.

[0119] Then, the Y components of the remaining displacement vectors are extracted, the average of the Y components is calculated, and then the standard deviation is calculated. Then, the displacement vectors whose Y components are more than A times the standard deviation away from the average are removed, where A is an adjustable natural number.

[0120] It is worth mentioning that the displacement vectors after screening are averaged to obtain the overall displacement vector of the current frame. Taking the initial frame as the origin (0, 0), the average displacement calculated between each frame is accumulated to obtain the current coordinates.

[0121] Step S40: judging the spin state of the unmanned aerial vehicle to obtain a first judgment result.

[0122] It should be noted that the spin state refers to the state of the unmanned aerial vehicle rotating around its own axis during flight. This state may be caused by external environmental factors (such as wind shear) or faults in the unmanned aerial vehicle's own control system. The spin state not only affects the flight stability and flight path accuracy of the unmanned aerial vehicle, but also may pose a threat to the safety of the unmanned aerial vehicle. Therefore, it is crucial to timely and accurately judge the spin state of the unmanned aerial vehicle and take appropriate control measures.

[0123] In a possible implementation, step S40 can include: dividing the high-altitude image into four quadrants to obtain high-altitude images of the four quadrants; calculating the average displacement vector in the high-altitude images of the four quadrants respectively; judging whether the unmanned aerial vehicle is in a spin state according to the signs of the horizontal component and the vertical component of the average displacement vector in the high-altitude images of the four quadrants; if the signs of the horizontal component and the vertical component of the average displacement vector in the high-altitude images of the four quadrants satisfy a preset condition and the duration exceeds a preset time threshold, determining that the first judgment result is that the unmanned aerial vehicle is in a spin state; if the signs of the horizontal component and the vertical component of the average displacement vector in the high-altitude images of the four quadrants do not satisfy the preset condition and / or the duration does not exceed the preset time threshold, determining that the first judgment result is that the unmanned aerial vehicle is in a non-spin state.

[0124] It should be noted that if the environmental wind direction is unstable in the hovering state of the unmanned aerial vehicle, it will cause the working unmanned aerial vehicle to appear a continuous spin that cannot be resolved by itself, so it is necessary to identify this state and implement appropriate measures.

[0125] It can be understood that when the unmanned aerial vehicle hovers and spins, the average displacement vector direction in the high-altitude images of the four quadrants is fixed. Taking the case of clockwise rotation of the unmanned aerial vehicle, i.e., counterclockwise rotation of the image, as an example, the average displacement direction features that the horizontal displacement of the first quadrant is negative, the vertical displacement is positive, the horizontal displacement of the second quadrant is negative, the vertical displacement is negative, the horizontal displacement of the third quadrant is positive, the vertical displacement is negative, and the horizontal displacement of the fourth quadrant is positive, the vertical displacement is positive, as shown in Figure 2 , and Figure 2 is a schematic diagram of the average displacement direction when the unmanned aerial vehicle rotates clockwise (the image rotates counterclockwise).

[0126] It is worth noting that when the unmanned aerial vehicle rotates counterclockwise (the image rotates clockwise), the component directions in the corresponding directions of adjacent quadrants are opposite, and therefore the signs of the horizontal component and the vertical component of the average displacement vector in the high-altitude images of the four quadrants can be used to determine the rotation direction of the unmanned aerial vehicle. If the preset conditions are met, i.e., the horizontal displacement of the first quadrant is positive, the vertical displacement is negative, the horizontal displacement of the second quadrant is positive, the vertical displacement is positive, the horizontal displacement of the third quadrant is negative, the vertical displacement is positive, and the horizontal displacement of the fourth quadrant is negative, the vertical displacement is positive, and the duration exceeds the preset time threshold, it is determined that the rotation direction of the unmanned aerial vehicle is counterclockwise, and the spin occurs. The preset time threshold can be 10s. By determining the rotation direction of the unmanned aerial vehicle, the flight state of the unmanned aerial vehicle can be further understood, and important basis for subsequent control decisions can be provided.

[0127] In a specific implementation, in the process of determining the spin state of the unmanned aerial vehicle, since the spin state includes clockwise and counterclockwise, the corresponding preset conditions can be that the signs of the horizontal component and the vertical component of the average displacement vector in the high-altitude images of the four quadrants correspond to the direction characteristics when rotating clockwise and counterclockwise, respectively. For example, when rotating clockwise, the preset conditions can be that the horizontal displacement of the first quadrant is negative, the vertical displacement is positive, the horizontal displacement of the second quadrant is negative, the vertical displacement is negative, the horizontal displacement of the third quadrant is positive, the vertical displacement is negative, and the horizontal displacement of the fourth quadrant is positive, the vertical displacement is positive. When rotating counterclockwise, the preset conditions are opposite. This preset condition is derived from the change rule of the displacement vector in the image when the unmanned aerial vehicle is spinning. When the unmanned aerial vehicle is in the spin state, due to the influence of rotation, the average displacement vector in the four quadrants will exhibit a specific directionality. Therefore, by detecting these directional characteristics, the spin state of the unmanned aerial vehicle can be effectively determined.

[0128] The preset time threshold is set to ensure the accuracy of the determination result. Since external environmental factors or temporary fluctuations of the unmanned aerial vehicle's own control system can cause temporary changes in the displacement vector, such changes are not sufficient to indicate that the unmanned aerial vehicle is in a spin state. Therefore, a time threshold needs to be set. Only when the duration of the change in the displacement vector exceeds the threshold, it is considered that the unmanned aerial vehicle is in a spin state.

[0129] By accurately determining the spin state of the unmanned aerial vehicle, potential flight safety problems can be discovered in time, and appropriate control measures can be taken to ensure the flight stability and safety of the unmanned aerial vehicle.

[0130] Further, the X and Y components of the average displacement vector of each quadrant are detected for a sign, and if the preset condition is met for more than 10 seconds, it is determined that the current UAV enters a spin state, and a self-resolving process is enabled. When the UAV is in a turbulent airflow area, a spin state that the flight control cannot resolve itself may occur. If the state is not resolved in time, the UAV spin will become faster and faster. After the state occurs, the joystick is manually manipulated to move the UAV forward and backward by a small distance, which can resolve the state. Therefore, after the UAV is identified to be in a long-time spin state by the above method, a command is sent to the flight control to move forward and backward by the same distance, which can resolve the state.

[0131] Step S50: In the case that the first judgment result is that the UAV is in a non-spin state, an emergency stop state of the UAV is judged to obtain a second judgment result.

[0132] It should be noted that when the UAV is in an emergency stop state, an excessive pitch angle may be generated. If the maximum compensation angle of the gimbal is less than the pitch angle in the emergency stop state, the camera will introduce perspective interference when capturing a picture. The emergency stop state can be judged according to the characteristics of the perspective interference.

[0133] In a possible implementation, step S50 can include: in the case that the first judgment result is that the UAV is in a non-spin state, the high-altitude image is divided into a left region and a right region; a displacement vector matrix is obtained, and the left region and the right region are respectively subjected to element extraction and processing based on the displacement vector matrix to obtain an average displacement vector of the left region and an average displacement vector of the right region; if the difference between the average displacement vector of the left region and the average displacement vector of the right region exceeds a preset threshold, it is determined that the second judgment result is that the UAV is in an emergency stop state; and if the difference between the average displacement vector of the left region and the average displacement vector of the right region does not exceed the preset threshold, it is determined that the second judgment result is that the UAV is in a non-emergency stop state.

[0134] It should be noted that in the present embodiment, optical flow calculation is performed only once for each frame, and the result of the optical flow calculation, i.e., the displacement vector matrix, is directly extracted in the subsequent process. The elements of different regions can be directly extracted from the displacement vector matrix for processing, which effectively improves the processing efficiency.

[0135] It can be understood that when the camera generates a pitch angle (i.e., the camera is not perpendicular to the ground) due to the emergency stop of the UAV, the higher side of the camera records a larger range due to the perspective principle, which makes the average displacement vector of the preset feature points on the side much larger than that on the other side. When the high-altitude image is divided into a left region (A) and a right region (B), the average displacement vectors of the two regions A and B have a relationship as follows:

[0136]

[0137] or

[0138]

[0139] The calculation method of obtaining the average displacement of the region is that when the difference between the average displacement of the A region and the B region exceeds a certain threshold, that is:

[0140]

[0141] It is judged that the UAV has an emergency stop.

[0142] In a specific implementation, the high-altitude image can also be divided into an upper region (C) and a lower region (D), and when the difference between the average displacement of the C region and the D region exceeds a certain threshold, that is:

[0143]

[0144] It is judged that the UAV has an emergency stop.

[0145] Step S60: In the case where the second judgment result is that the UAV is in a non-emergency stop state, yaw correction is performed based on the overall displacement vector of the current frame and is accumulated to obtain the current position of the UAV.

[0146] It should be noted that the yaw correction refers to correcting the interference factors introduced by the self-rotation of the UAV relative to the ground after the self-rotation, so that the overall displacement vector calculated in each frame is more accurate, thereby ensuring the position accuracy of the UAV.

[0147] In a feasible implementation, step S60 can further include: calculating the angle of the self-rotation movement of the UAV other than the horizontal movement according to the overall displacement vector of the current frame; and correcting the calculated overall displacement vector according to the calculated angle. Through the yaw correction, the overall displacement vector calculated in each frame can be made closer to the actual displacement vector of the UAV, thereby improving the positioning accuracy of the UAV.

[0148] In a specific implementation, the speed, height, and other information of the UAV can also be combined for comprehensive judgment to further improve the accuracy of the yaw correction.

[0149] It should be noted that in the case where the second judgment result is that the UAV is in an emergency stop state, the current frame displacement vector is set to 0, thereby reducing the calculation error caused by the lens pitch caused by the emergency stop.

[0150] The embodiment provides a visual-based unmanned aerial vehicle positioning method, which comprises the following steps: collecting a high-altitude image by a camera carried by an unmanned aerial vehicle, and pre-processing the high-altitude image to obtain a pre-processed high-altitude image, wherein the pre-processing at least comprises grayscale conversion, image equalization and noise reduction operation; performing feature point selection on the pre-processed high-altitude image to obtain a target feature point array; performing optical flow calculation and displacement vector secondary processing based on the target feature point array to obtain an overall displacement vector of a current frame; performing self-rotation state judgment on the unmanned aerial vehicle to obtain a first judgment result; in the case that the first judgment result is that the unmanned aerial vehicle is in a non-self-rotation state, performing emergency stop state judgment on the unmanned aerial vehicle to obtain a second judgment result; in the case that the second judgment result is that the unmanned aerial vehicle is in a non-emergency stop state, performing yaw correction and accumulation based on the overall displacement vector of the current frame to obtain a current position of the unmanned aerial vehicle. Through the above method, the self-rotation and emergency stop judgment mechanism is introduced, the error is avoided, and then the error caused by the displacement vector of the feature points in different regions is corrected, so that the unmanned aerial vehicle positioning accuracy is further improved.

[0151] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above embodiment one can refer to the above introduction, and the subsequent will not be described in detail. On this basis, please refer to Figure 3 , step S60 comprises steps S601-S603:

[0152] Step S601: in the case that the second judgment result is that the unmanned aerial vehicle is in a non-emergency stop state, performing yaw angle calculation based on the overall displacement vector of the current frame to obtain a yaw angle.

[0153] It should be noted that the yaw angle is the self-rotation angle of the unmanned aerial vehicle, that is, the angle of left and right yaw rotation of the unmanned aerial vehicle. By calculating the yaw angle, the self-rotation of the unmanned aerial vehicle in the flight process can be understood, and key data is provided for subsequent positioning correction. The calculation of the yaw angle can be based on the overall displacement vector of the current frame, and the flight attitude and speed of the unmanned aerial vehicle are comprehensively analyzed to obtain the yaw angle.

[0154] In a specific implementation, the calculation of the yaw angle can be realized by a geometric method or a vector method. The geometric method mainly calculates the yaw angle by measuring the included angle between the current position of the unmanned aerial vehicle and the predetermined route; and the vector method calculates the yaw angle by calculating the included angle between the vector of the current flight direction of the unmanned aerial vehicle and the vector of the direction of the predetermined route. The two methods have advantages and disadvantages, and the appropriate method can be selected to calculate the yaw angle according to the actual situation.

[0155] In a possible implementation, step S601 can comprise: determining a yaw angle offset in the vertical direction and a yaw angle offset in the horizontal direction based on the overall displacement vector of the current frame; determining an interference displacement vector based on the yaw angle offset in the vertical direction and the yaw angle offset in the horizontal direction; and determining a yaw angle based on the interference displacement vector.

[0156] It should be noted that the UAV will actively or passively yaw in the working state. Yaw will introduce an interference displacement vector to the optical flow calculation of the target feature point array, and therefore the average displacement vector obtained by calculation needs to be yaw-corrected and accumulated.

[0157] It can be understood that the displacement calculation method is essentially to detect the displacement of points on an image. The displacement vector calculated according to the pixel points on the image captured by the camera is the combined displacement vector of the displacement components respectively affected by translation and yaw-induced rotation. What we need is the translation displacement vector (also the actual displacement vector of the camera), and the displacement vector calculated due to yaw is an error term or an interference term. Therefore, reducing the displacement vector caused by yaw aims to reduce the calculation error and thus improve the calculation accuracy.

[0158] In a specific implementation, the image captured by the camera is equally divided into two parts, i.e., a left region (A) and a right region (B), as shown in Figure 4 . Figure 4 is a schematic diagram in which an image frame is divided into left and right sides.

[0159] If there is no yaw angle change, the displacement vectors calculated by the two parts should be parallel and the vector modules should not differ much, i.e., when there is no yaw, the displacement vectors of each feature point are parallel, as shown in Figure 5 . Figure 5 is a schematic diagram of the displacement vectors of each feature point when there is no yaw.

[0160] If yaw occurs, an additional interference displacement vector will be introduced for the calculation of the displacement vector between each frame, so that the calculated displacement vector is offset, as shown in Figure 6 . Figure 6 is a schematic diagram in which the displacement vectors calculated by different feature points are offset to different degrees when yaw occurs.

[0161] The displacement vector calculated when yaw occurs is:

[0162]

[0163] The actual displacement vector can be obtained as follows:

[0164]

[0165] In the left region (region A), when the UAV only has yaw, the x components of the displacement vectors calculated by the target feature point array symmetrical about the X axis are the same in size and opposite in direction, as shown in FIG. 8. Figure 7 Figure 7 FIG. 8 is a schematic diagram of the x components of the displacement vectors in region A symmetrical about the X axis.

[0166] Therefore, the average displacement of all displacement vectors in region A is (0, y). Similarly, the average displacement of all displacement vectors in region B is (0, -y).

[0167] According to the above algorithm, the calculated average displacement of regions A and B can be obtained, and the actual displacement of regions A and B is known to be the same, so the following can be calculated:

[0168]

[0169] That is, the yaw angle offset in the Y axis direction is:

[0170]

[0171] If the image is 2L long and 2W wide (i.e., region A is L long and 2W wide), the Y component of the displacement vector occurs at a distance of r = 0.5 * L from the center of rotation.

[0172] Similarly, the image is divided into two parts, i.e., region C and region D, as shown in FIG. 9. Figure 8 Figure 8 is a schematic diagram of the image divided into two parts.

[0173] The following can be calculated:

[0174]

[0175] If the image is 2L long and 2W wide (i.e., region C is 2L long and W wide), the X component of the displacement vector occurs at a distance of r = 0.5 * W from the center of rotation.

[0176] Because the X and Y components calculated above occur at different distances from the center (r), the interference displacement vector introduced by yaw at a distance of r = 0.5 * L from the center of rotation is (x, y), and the angle of deflection can be calculated as:

[0177]

[0178] Step S602: correcting the overall displacement vector of the current frame according to the yaw angle and a reference displacement vector to obtain a corrected overall displacement vector, wherein the reference displacement vector is a fixed reference value obtained by optical flow calculation and displacement vector processing according to a preset feature point array. ​​​

[0179] It should be noted that when the UAV occurs yaw (relative to the ground self-rotation), the yaw angle needs to be calculated, which means that the coordinate system of the UAV and the coordinate system of the ground have been deflected by an angle, and the calculation error caused by the angle needs to be corrected once. The correction is essentially to convert the displacement vector of the UAV into a displacement vector in the ground coordinate system.

[0180] It can be understood that the reference displacement vector is a fixed reference value obtained by optical flow calculation and displacement vector processing according to the preset feature point array, which plays a role of a reference in the entire positioning process. By comparing the overall displacement vector of the current frame with the reference displacement vector, and combining the calculated yaw angle, the error caused by yaw can be more accurately corrected.

[0181] In specific implementation, according to the reference displacement vector and the overall displacement vector of the current frame, the average displacement vector calculated during yaw is corrected, and the correction value is:

[0182]

[0183] Wherein, N is the modulus of (X, Y), , is the deflection angle.

[0184] The corrected overall displacement vector obtained by calculation can be obtained by accumulation, and the continuous position information of the UAV in the flight process can be obtained, so as to realize accurate positioning of the UAV. The angle is also accumulated and calculated, and needs to be taken in reverse when used. Because the calculated relative to the original direction is deflected by angle, it needs to be corrected back in the opposite direction. Once the UAV occurs yaw, the influence of the angle is always present, so it needs to be accumulated.

[0185] Optionally, in the case of judging the need for yaw correction, the size of the yaw angle and the current flight speed, height and other information of the UAV are also calculated according to the preset yaw correction model or algorithm to calculate the corrected overall displacement vector. The corrected overall displacement vector can more accurately reflect the actual position of the UAV on the predetermined route.

[0186] Step S603: based on the corrected overall displacement vector, accumulation is carried out to determine the current position of the UAV.

[0187] ​​It should be noted that the current position of the unmanned aerial vehicle is updated based on the corrected overall displacement vector. After obtaining the corrected overall displacement vector, it is applied to the position updating algorithm of the unmanned aerial vehicle to calculate the current position of the unmanned aerial vehicle in three-dimensional space. This position information can be used for navigation, path planning, obstacle avoidance and other aspects of the unmanned aerial vehicle, and is the basis for autonomous flight and task completion of the unmanned aerial vehicle.

[0188] In a specific implementation, the current position of the unmanned aerial vehicle can be gradually updated by accumulation based on the initial position of the unmanned aerial vehicle and the corrected overall displacement vector. The initial position of the unmanned aerial vehicle can be obtained by a positioning device such as GPS, and the corrected overall displacement vector reflects the actual movement of the unmanned aerial vehicle during flight. By continuously updating the current position of the unmanned aerial vehicle, real-time tracking and monitoring of the flight trajectory of the unmanned aerial vehicle can be realized, providing strong support for autonomous flight and remote control of the unmanned aerial vehicle. In addition, after determining the current position of the unmanned aerial vehicle, path planning, obstacle avoidance decision and other operations can be further combined with task requirements, environmental information and other factors of the unmanned aerial vehicle to realize more intelligent and autonomous flight control of the unmanned aerial vehicle.

[0189] In this embodiment, by separately processing the preset feature points in different regions, the error caused by yaw can be corrected, thereby effectively improving the accuracy of unmanned aerial vehicle point positioning.

[0190] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the visual-based unmanned aerial vehicle positioning method of the present application. More forms of simple transformation based on this technical concept are within the protection scope of the present application.

[0191] The present application also provides a visual-based unmanned aerial vehicle positioning device, which is described in detail in the Figure 9 The device comprises:

[0192] The preprocessing module 10 is configured to collect high-altitude images by the camera carried by the unmanned aerial vehicle, and pre-process the high-altitude images to obtain pre-processed high-altitude images, wherein the preprocessing at least includes gray scale conversion, image equalization and noise reduction operation.

[0193] The selection module 20 is configured to select feature points from the pre-processed high-altitude images to obtain a target feature point array.

[0194] The screening module 30 is configured to perform optical flow calculation and displacement vector secondary processing based on the target feature point array to obtain an overall displacement vector of the current frame.

[0195] The judgment module 40 is configured to judge the spin state of the unmanned aerial vehicle to obtain a first judgment result.

[0196] The judgment module 40 is further configured to, when the first judgment result indicates that the UAV is in a non-spinning state, perform emergency stop state judgment on the UAV to obtain a second judgment result.

[0197] The correction module 50 is configured to, when the second judgment result indicates that the UAV is in a non-emergency stop state, perform yaw correction and accumulation based on the overall displacement vector of the current frame to obtain the current position of the UAV.

[0198] The visual-based UAV positioning device provided in the present application adopts the visual-based UAV positioning method in the above embodiments, and can solve the technical problem of poor UAV positioning accuracy in the prior art. Compared with the prior art, the visual-based UAV positioning device provided in the present application has the same beneficial effects as the visual-based UAV positioning method provided in the above embodiments, and other technical features in the visual-based UAV positioning device are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0199] The present application provides a visual-based UAV positioning device, which comprises at least one processor and a memory in communication connection with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the visual-based UAV positioning method in Embodiment I.

[0200] Reference will now be made to the drawings, and Figure 10 which shows a structural schematic diagram of a visual-based UAV positioning device suitable for implementing the embodiments of the present application. The visual-based UAV positioning device in the embodiments of the present application can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle-mounted terminals (for example, vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 10 The visual-based UAV positioning device shown is merely an example, and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0201] As Figure 10As shown, the vision-based UAV positioning device can include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to programs stored in a ROM 1002 (Read Only Memory) or loaded from a storage device 1003 into a RAM 1004 (Random Access Memory). In the RAM 1004, various programs and data required for the operation of the vision-based UAV positioning device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An I / O interface 1006 (input / output interface) is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, an LCD (Liquid Crystal Display), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the vision-based UAV positioning device to communicate with other devices wirelessly or by wire to exchange data. Although the vision-based UAV positioning device with various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or possessed. More or fewer systems can be alternatively implemented or possessed.

[0202] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are performed.

[0203] The vision-based UAV positioning device provided in the present application adopts the vision-based UAV positioning method in the above-mentioned embodiments, and can solve the technical problem of poor positioning accuracy of the UAV in the prior art. Compared with the prior art, the vision-based UAV positioning device provided in the present application has the same beneficial effects as the vision-based UAV positioning method provided in the above-mentioned embodiments, and other technical features in the vision-based UAV positioning device are the same as the features disclosed in the previous embodiment method, which will not be described here.

[0204] It should be understood that various aspects of the disclosure can be implemented in hardware, software, firmware, or a combination thereof. In the description of the embodiments above, specific features, structures, materials or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0205] The above description is merely illustrative of the application and is not intended to limit the scope of the application. Any modifications or equivalents of the technology disclosed herein should be construed as falling within the scope of the application. Therefore, the scope of the application should be determined by the scope of the claims.

[0206] The application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e., computer programs) for performing the visual-based UAV positioning method in the above-described embodiments.

[0207] The computer readable storage medium provided by the application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination thereof. More specific examples of the computer readable storage medium can include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a RAM (Random Access Memory), a ROM (Read Only Memory), an EPROM (Erasable Programmable Read Only Memory or flash memory), an optical fiber, a CD-ROM (CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), etc., or any suitable combination thereof.

[0208] The above computer readable storage medium can be included in the visual-based UAV positioning device; or can exist separately and not be assembled into the visual-based UAV positioning device.

[0209] The computer readable storage medium carries one or more programs, when the one or more programs are executed by the visual-based unmanned aerial vehicle positioning device, the visual-based unmanned aerial vehicle positioning device is caused to: collect high-altitude images through a camera carried by an unmanned aerial vehicle, and pre-process the high-altitude images to obtain pre-processed high-altitude images, wherein the pre-processing at least includes grayscale conversion, image equalization, and noise reduction operation; perform feature point selection on the pre-processed high-altitude images to obtain a target feature point array; perform optical flow calculation and displacement vector secondary processing based on the target feature point array to obtain an overall displacement vector of a current frame; perform self-rotation state judgment on the unmanned aerial vehicle to obtain a first judgment result; in a case where the first judgment result is that the unmanned aerial vehicle is in a non-self-rotation state, perform emergency stop state judgment on the unmanned aerial vehicle to obtain a second judgment result; in a case where the second judgment result is that the unmanned aerial vehicle is in a non-emergency stop state, perform yaw correction and accumulation based on the overall displacement vector of the current frame to obtain a current position of the unmanned aerial vehicle.

[0210] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0211] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or combinations of hardware and software.

[0212] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the unit itself.

[0213] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer programs) for executing the above-mentioned visual-based UAV positioning method, and can solve the technical problem of poor UAV positioning accuracy in the prior art. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the visual-based UAV positioning method provided by the above-mentioned embodiments, which will not be repeated here.

[0214] The present application also provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the visual-based UAV positioning method as described above.

[0215] The computer program product provided by the present application can solve the technical problem of poor UAV positioning accuracy in the prior art. Compared with the prior art, the computer program product provided by the present application has the same beneficial effects as the visual-based UAV positioning method provided by the above-mentioned embodiments, which will not be repeated here.

[0216] The above only describes some embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structural transformation made by using the content of the present application specification and drawings, or direct / indirect application in other related technical fields is included in the patent protection scope of the present application.

Claims

1. A vision-based UAV localization method, characterized in that, The method includes: High-altitude images are captured by a camera mounted on a drone, and the high-altitude images are preprocessed to obtain preprocessed high-altitude images. The preprocessing includes at least grayscale conversion, image equalization, and noise reduction. Feature points are selected from the preprocessed high-altitude image to obtain a target feature point array; Optical flow calculation and secondary processing of displacement vector are performed based on the target feature point array to obtain the overall displacement vector of the current frame; The spin state of the drone is determined to obtain a first determination result; If the first judgment result indicates that the drone is in a non-spinning state, the drone is then subjected to an emergency stop judgment to obtain a second judgment result. If the second determination result indicates that the UAV is in a non-emergency stop state, yaw correction is performed based on the overall displacement vector of the current frame and accumulated to obtain the current position of the UAV; The process of performing optical flow calculation and secondary displacement vector processing based on the target feature point array to obtain the overall displacement vector of the current frame includes: The Lucas-Kanade optical flow algorithm is used to calculate the optical flow of the target feature point array to obtain the optical flow vector of each target feature point array from the previous frame to the current frame. Based on the direction and magnitude of the optical flow vector, calculate the displacement vector of each target feature point array; The displacement vectors of each of the target feature point arrays are processed twice to obtain the overall displacement vector of the current frame; The secondary processing of the displacement vectors of each of the target feature point arrays to obtain the overall displacement vector of the current frame includes: Extract the horizontal component of the displacement vector of each of the target feature point arrays, and determine the horizontal average component and the horizontal component standard deviation based on the horizontal component; The displacement vectors of each target feature point array are filtered based on the average level component and the standard deviation of the level component. Displacement vectors whose horizontal component is more than a preset multiple of the standard deviation of the level component are removed, and the displacement vectors after the initial screening are obtained. Extract the vertical component of the displacement vector after the initial screening, and determine the average vertical component and the standard deviation of the vertical component based on the vertical component. Based on the average vertical component and the standard deviation of the vertical component, the displacement vectors after the initial screening are screened again, and displacement vectors whose vertical component is more than a preset multiple of the standard deviation of the vertical component are removed, resulting in displacement vectors after secondary screening. The average value of the displacement vector after the secondary filtering is calculated to obtain the overall displacement vector of the current frame. The step of determining the spin state of the UAV to obtain a first determination result includes: The high-altitude images are divided into quadrants to obtain high-altitude images in four quadrants; Calculate the average displacement vector in the aerial images of the four quadrants respectively; Based on the horizontal and vertical component signs of the average displacement vector in the high-altitude images of the four quadrants, determine whether the UAV is in a spin state; If the horizontal and vertical component symbols of the average displacement vector in the high-altitude images of the four quadrants meet the preset conditions and the duration exceeds the preset time threshold, then the first judgment result is determined to be that the UAV is in a spin state. If the horizontal and vertical component symbols of the average displacement vector in the high-altitude images of the four quadrants do not meet the preset conditions and / or the duration does not exceed the preset time threshold, then the first judgment result is determined to be that the UAV is in a non-spinning state.

2. The method as described in claim 1, characterized in that, The step of selecting feature points from the preprocessed high-altitude image to obtain a target feature point array includes: Using a preset coordinate point as a reference, candidate feature points of a preset number of rows and columns are selected in the preprocessed aerial image according to a preset step size. The preset coordinate point is the center point of the image. The preset step size is adjusted according to the resolution of the preprocessed aerial image. The preset number of rows is determined according to the height of the preprocessed aerial image and the preset step size. The preset number of columns is determined according to the height and width of the preprocessed aerial image. Feature descriptors are calculated for the candidate feature points to obtain the feature descriptor for each candidate feature point; Based on the feature descriptor, feature matching is performed to obtain matched feature point pairs; The target feature point array is obtained by filtering based on the geometric relationship between the matched feature point pairs, wherein the feature points in the target feature point array are uniformly distributed in the preprocessed high-altitude image.

3. The method as described in claim 1, characterized in that, The step of determining the emergency stop state of the drone when the first determination result indicates that the drone is in a non-spinning state, and obtaining a second determination result, includes: If the first judgment result indicates that the UAV is in a non-spinning state, the high-altitude image is divided into regions to obtain a left region and a right region. Obtain the displacement vector matrix, and extract and process elements of the left region and the right region based on the displacement vector matrix to obtain the average displacement vector of the left region and the average displacement vector of the right region. If the difference between the average displacement vector of the left region and the average displacement vector of the right region exceeds a preset threshold, then the second judgment result is determined to be that the drone is in an emergency stop state. If the difference between the average displacement vector of the left region and the average displacement vector of the right region does not exceed a preset threshold, then the second judgment result is determined to be that the drone is in a non-emergency stop state.

4. The method as described in claim 1, characterized in that, When the second determination result indicates that the UAV is in a non-emergency stop state, the step of performing yaw correction based on the overall displacement vector of the current frame and accumulating the results to obtain the current position of the UAV includes: If the second judgment result indicates that the UAV is in a non-emergency stop state, the yaw angle is calculated based on the overall displacement vector of the current frame to obtain the yaw angle. The overall displacement vector of the current frame is corrected based on the yaw angle and the reference displacement vector to obtain the corrected overall displacement vector. The reference displacement vector is a fixed reference value obtained by optical flow calculation and displacement vector processing based on a preset feature point array. The current position of the UAV is determined by accumulating the corrected overall displacement vector.

5. The method as described in claim 4, characterized in that, The calculation of the yaw angle based on the overall displacement vector of the current frame to obtain the yaw angle includes: The vertical and horizontal heading angle offsets are determined based on the overall displacement vector of the current frame. The disturbance displacement vector is determined based on the vertical and horizontal heading angle offsets. The yaw angle is determined based on the disturbance displacement vector.

6. A vision-based unmanned aerial vehicle (UAV) positioning device, characterized in that, include: The preprocessing module is used to acquire high-altitude images through the camera mounted on the drone and preprocess the high-altitude images to obtain preprocessed high-altitude images. The preprocessing includes at least grayscale conversion, image equalization and noise reduction operations. The selection module is used to select feature points from the preprocessed high-altitude image to obtain a target feature point array. The filtering module is used to perform optical flow calculation and secondary processing of displacement vector based on the target feature point array to obtain the overall displacement vector of the current frame; The judgment module is used to judge the spin state of the UAV and obtain a first judgment result; The judgment module is further configured to determine the emergency stop state of the drone when the first judgment result indicates that the drone is in a non-spinning state, and obtain a second judgment result. The correction module is used to perform yaw correction and accumulate the yaw based on the overall displacement vector of the current frame when the second judgment result is that the UAV is in a non-emergency stop state, so as to obtain the current position of the UAV. The filtering module is used to perform optical flow calculation on the target feature point array using the Lucas-Kanade optical flow algorithm to obtain the optical flow vector of each target feature point array from the previous frame to the current frame; calculate the displacement vector of each target feature point array according to the direction and magnitude of the optical flow vector; and perform secondary processing on the displacement vector of each target feature point array to obtain the overall displacement vector of the current frame. The filtering module is further configured to: extract the horizontal component of the displacement vector of each target feature point array; determine the average horizontal component and the standard deviation of the horizontal component based on the horizontal component; filter the displacement vectors of each target feature point array based on the average horizontal component and the standard deviation of the horizontal component, removing displacement vectors whose horizontal component is more than a preset multiple of the standard deviation of the horizontal component, to obtain the displacement vectors after initial filtering; extract the vertical component of the displacement vectors after initial filtering, and determine the average vertical component and the standard deviation of the vertical component based on the vertical component; filter the displacement vectors after initial filtering again based on the average vertical component and the standard deviation of the vertical component, removing displacement vectors whose vertical component is more than a preset multiple of the standard deviation of the vertical component, to obtain the displacement vectors after secondary filtering; and calculate the average value of the displacement vectors after secondary filtering to obtain the overall displacement vector of the current frame. The judgment module is used to divide the high-altitude image into four quadrants to obtain high-altitude images in four quadrants; calculate the average displacement vector in the high-altitude images in each of the four quadrants; and determine whether the UAV is in a spin state based on the horizontal and vertical component signs of the average displacement vector in the high-altitude images in the four quadrants. If the horizontal and vertical component signs of the average displacement vector in the high-altitude images in the four quadrants meet preset conditions and the duration exceeds a preset time threshold, then the first judgment result is determined to be that the UAV is in a spin state; if the horizontal and vertical component signs of the average displacement vector in the high-altitude images in the four quadrants do not meet preset conditions and / or the duration does not exceed the preset time threshold, then the first judgment result is determined to be that the UAV is in a non-spin state.

7. A storage medium, characterized in that, The storage medium stores a vision-based drone positioning program, which, when executed by a processor, implements the method as described in any one of claims 1 to 4.

Images