Vision-based unmanned aerial vehicle positioning method and device and storage medium

Abstract

The invention discloses a vision-based unmanned aerial vehicle positioning method and device and a storage medium, and the method comprises the steps: collecting a high-altitude image through a camera carried by an unmanned aerial vehicle, and carrying out the preprocessing of the high-altitude image, and obtaining a preprocessed high-altitude image; performing feature point selection on the preprocessed high-altitude image to obtain a target feature point array; performing optical flow calculation and displacement vector secondary processing based on the target feature point array to obtain an overall displacement vector of the current frame; the spinning state of the unmanned aerial vehicle is judged; under the condition that the unmanned aerial vehicle is in the non-spinning state, the sudden stop state of the unmanned aerial vehicle is judged; under the condition that the unmanned aerial vehicle is in a non-sudden-stop state, yaw correction and accumulation are carried out based on the overall displacement vector of the current frame, the current position of the unmanned aerial vehicle is obtained, errors are prevented from being introduced by introducing a spinning and sudden-stop judgment mechanism, then errors caused by yaw are corrected based on displacement vectors of feature points of different areas, and the accuracy of yaw correction is improved. And the positioning accuracy of the unmanned aerial vehicle is further improved.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) positioning technology, and in particular to vision-based UAV positioning methods, devices, and storage media. Background Technology

[0002] Existing technologies often employ image recognition to first identify people or ground vehicles, then use ORB (Objective-Oriented Base) algorithms to detect key points in other areas, and finally use optical flow algorithms to track these key points and deduce the drone's own movement. First, this approach requires significant computing power for both image recognition and ORB algorithms, making it unsuitable for the working environment of a drone's onboard platform. Second, because drones operate at a high altitude with a large image capture range, the complex terrain makes it difficult to guarantee the complete identification of all moving objects. Third, when using ORB algorithms to find key points, the edges of moving objects are more easily identified as key points than stationary objects; if object identification fails, the introduction of too many erroneous key points will lead to a lot of incorrect displacement information. Finally, although gimbal-based cameras are used, significantly reducing shaking, sudden stops by the drone can cause large tilt angles, resulting in gimbal tilt. Because the acquired image is two-dimensional, when the camera tilts, perspective principles cause significant distortion in the measured displacement information, leading to poor drone positioning accuracy. Summary of the Invention

[0003] The main objective of this application is to provide a vision-based drone positioning method, device, and storage medium, aiming to solve the technical problem of poor drone positioning accuracy in the prior art.

[0004] To achieve the above objectives, this application proposes a vision-based UAV localization method, the method comprising: High-altitude images are captured by a camera mounted on a drone, and the high-altitude images are preprocessed to obtain preprocessed high-altitude images. The preprocessing includes at least grayscale conversion, image equalization, and noise reduction. Feature points are selected from the preprocessed high-altitude image to obtain a target feature point array; Optical flow calculation and secondary processing of displacement vector are performed based on the target feature point array to obtain the overall displacement vector of the current frame; The spin state of the drone is determined to obtain a first determination result; If the first judgment result indicates that the drone is in a non-spinning state, the drone is then subjected to an emergency stop judgment to obtain a second judgment result. If the second determination result indicates that the UAV is in a non-emergency stop state, the yaw correction is performed based on the overall displacement vector of the current frame and accumulated to obtain the current position of the UAV.

[0005] In one embodiment, the step of selecting feature points from the preprocessed high-altitude image to obtain a target feature point array includes: Using a preset coordinate point as a reference, candidate feature points of a preset number of rows and columns are selected in the preprocessed aerial image according to a preset step size. The preset coordinate point is the center point of the image. The preset step size is adjusted according to the resolution of the preprocessed aerial image. The preset number of rows is determined according to the height of the preprocessed aerial image and the preset step size. The preset number of columns is determined according to the height and width of the preprocessed aerial image. Feature descriptors are calculated for the candidate feature points to obtain the feature descriptor for each candidate feature point; Based on the feature descriptor, feature matching is performed to obtain matched feature point pairs; The target feature point array is obtained by filtering based on the geometric relationship between the matched feature point pairs, wherein the feature points in the target feature point array are uniformly distributed in the preprocessed high-altitude image.

[0006] In one embodiment, the step of performing optical flow calculation and secondary displacement vector processing based on the target feature point array to obtain the overall displacement vector of the current frame includes: The Lucas-Kanade optical flow algorithm is used to calculate the optical flow of the target feature point array to obtain the optical flow vector of each target feature point array from the previous frame to the current frame. Based on the direction and magnitude of the optical flow vector, calculate the displacement vector of each target feature point array; The displacement vectors of each of the target feature point arrays are processed twice to obtain the overall displacement vector of the current frame.

[0007] In one embodiment, the secondary processing of the displacement vectors of each of the target feature point arrays to obtain the overall displacement vector of the current frame includes: Extract the horizontal component of the displacement vector of each of the target feature point arrays, and determine the horizontal average component and the horizontal component standard deviation based on the horizontal component; The displacement vectors of each target feature point array are filtered based on the average level component and the standard deviation of the level component. Displacement vectors whose horizontal component is more than a preset multiple of the standard deviation of the level component are removed, and the displacement vectors after the first screening are obtained. Extract the vertical component of the displacement vector after the initial screening, and determine the average vertical component and the standard deviation of the vertical component based on the vertical component. Based on the average vertical component and the standard deviation of the vertical component, the displacement vectors after the initial screening are screened again, and displacement vectors whose vertical component is more than a preset multiple of the standard deviation of the vertical component are removed, resulting in displacement vectors after secondary screening. The average value of the displacement vector after the second filtering is calculated to obtain the overall displacement vector of the current frame.

[0008] In one embodiment, determining the spin state of the UAV to obtain a first determination result includes: The high-altitude images are divided into quadrants to obtain high-altitude images in four quadrants; Calculate the average displacement vector in the aerial images of the four quadrants respectively; Based on the horizontal and vertical component signs of the average displacement vector in the high-altitude images of the four quadrants, determine whether the UAV is in a spin state; If the horizontal and vertical component symbols of the average displacement vector in the high-altitude images of the four quadrants meet the preset conditions and the duration exceeds the preset time threshold, then the first judgment result is determined to be that the UAV is in a spin state. If the horizontal and vertical component symbols of the average displacement vector in the high-altitude images of the four quadrants do not meet the preset conditions and / or the duration does not exceed the preset time threshold, then the first judgment result is determined to be that the UAV is in a non-spinning state.

[0009] In one embodiment, the step of determining the emergency stop state of the drone and obtaining a second determination result when the first determination result indicates that the drone is in a non-spinning state includes: If the first judgment result indicates that the UAV is in a non-spinning state, the high-altitude image is divided into regions to obtain a left region and a right region. Obtain the displacement vector matrix, and extract and process elements of the left region and the right region based on the displacement vector matrix to obtain the average displacement vector of the left region and the average displacement vector of the right region. If the difference between the average displacement vector of the left region and the average displacement vector of the right region exceeds a preset threshold, then the second judgment result is determined to be that the drone is in an emergency stop state. If the difference between the average displacement vector of the left region and the average displacement vector of the right region does not exceed a preset threshold, then the second judgment result is determined to be that the drone is in a non-emergency stop state.

[0010] In one embodiment, when the second determination result indicates that the UAV is in a non-emergency stop state, the step of performing yaw correction based on the overall displacement vector of the current frame and accumulating the results to obtain the current position of the UAV includes: If the second judgment result indicates that the UAV is in a non-emergency stop state, the yaw angle is calculated based on the overall displacement vector of the current frame to obtain the yaw angle. The overall displacement vector of the current frame is corrected based on the yaw angle and the reference displacement vector to obtain the corrected overall displacement vector. The reference displacement vector is a fixed reference value obtained by optical flow calculation and displacement vector processing based on a preset feature point array. The current position of the UAV is determined by accumulating the corrected overall displacement vector.

[0011] In one embodiment, the step of calculating the yaw angle based on the overall displacement vector of the current frame to obtain the yaw angle includes: The vertical and horizontal heading angle offsets are determined based on the overall displacement vector of the current frame. The disturbance displacement vector is determined based on the vertical and horizontal heading angle offsets. The yaw angle is determined based on the disturbance displacement vector.

[0012] Furthermore, to achieve the above objectives, this application also proposes a vision-based unmanned aerial vehicle (UAV) positioning device, comprising: The preprocessing module is used to acquire high-altitude images through the camera mounted on the drone and preprocess the high-altitude images to obtain preprocessed high-altitude images. The preprocessing includes at least grayscale conversion, image equalization and noise reduction operations. The selection module is used to select feature points from the preprocessed high-altitude image to obtain a target feature point array, wherein the feature points in the target feature point array are uniformly distributed in the preprocessed high-altitude image. The filtering module is used to perform optical flow calculation and secondary processing of displacement vector based on the target feature point array to obtain the overall displacement vector of the current frame; The judgment module is used to judge the spin state of the UAV and obtain a first judgment result; The judgment module is further configured to determine the emergency stop state of the drone when the first judgment result indicates that the drone is in a non-spinning state, and obtain a second judgment result. The correction module is used to perform yaw correction and accumulate the results based on the overall displacement vector of the current frame when the second judgment result indicates that the UAV is in a non-emergency stop state, so as to obtain the current position of the UAV.

[0013] In addition, to achieve the above objectives, this application also proposes a vision-based unmanned aerial vehicle (UAV) positioning device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the method described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the method described above.

[0015] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps described above.

[0016] One or more technical solutions proposed in this application acquire high-altitude images using a camera mounted on a UAV, and preprocess the high-altitude images to obtain preprocessed high-altitude images. The preprocessing includes at least grayscale conversion, image equalization, and noise reduction. Feature points are selected from the preprocessed high-altitude images to obtain a target feature point array. Optical flow calculation and secondary displacement vector processing are performed based on the target feature point array to obtain the overall displacement vector of the current frame. The UAV's spin state is determined to obtain a first determination result. If the first determination result indicates that the UAV is in a non-spin state, an emergency stop determination is performed to obtain a second determination result. If the second determination result indicates that the UAV is not in an emergency stop state, yaw correction is performed based on the overall displacement vector of the current frame and accumulated to obtain the UAV's current position. By introducing spin and emergency stop determination mechanisms, errors are avoided, and errors caused by yaw are corrected based on the displacement vectors of feature points in different regions, further improving the UAV's positioning accuracy. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1This is a flowchart illustrating an embodiment of the vision-based UAV positioning method of this application. Figure 2 This is a schematic diagram of the average displacement direction of a UAV rotating clockwise (image rotating counterclockwise) according to an embodiment of the vision-based UAV positioning method of this application. Figure 3 This is a flowchart illustrating a second embodiment of the vision-based UAV positioning method of this application. Figure 4 This is a schematic diagram showing an image frame divided into left and right sides, provided for an embodiment of the vision-based UAV positioning method of this application; Figure 5 This is a schematic diagram of the displacement vectors of various feature points in a vision-based UAV positioning method according to an embodiment of this application, provided for a yaw-free test. Figure 6 This is a schematic diagram showing the different degrees of displacement vector shifts calculated from different feature points when yaw occurs, provided in an embodiment of the vision-based UAV positioning method of this application. Figure 7 This is a schematic diagram of the x-component of the displacement vector symmetrical about the X-axis in region A, provided in an embodiment of the vision-based UAV positioning method of this application. Figure 8 This is a schematic diagram showing an image frame divided into upper and lower sides, provided for an embodiment of the vision-based UAV positioning method of this application; Figure 9 This is a schematic diagram of the module structure of a vision-based UAV positioning device according to an embodiment of this application; Figure 10 This is a schematic diagram of the hardware operating environment involved in the vision-based drone positioning device in this application embodiment.

[0020] Explanation of icon numbers: 1001 - Processing device, 1002 - ROM, 1003 - Storage device, 1004 - RAM, 1005 - Bus, 1006 - I / O interface, 1007 - Input device, 1008 - Output device, 1009 - Communication device.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] The main solution of this application embodiment is as follows: High-altitude images are acquired using a camera mounted on a drone, and the high-altitude images are preprocessed to obtain preprocessed high-altitude images. The preprocessing includes at least grayscale conversion, image equalization, and noise reduction. Feature points are selected from the preprocessed high-altitude images to obtain a target feature point array. Optical flow calculation and secondary displacement vector processing are performed based on the target feature point array to obtain the overall displacement vector of the current frame. The drone's spin state is determined to obtain a first determination result. If the first determination result indicates that the drone is in a non-spin state, the drone's emergency stop state is determined to obtain a second determination result. If the second determination result indicates that the drone is in a non-emergency stop state, yaw correction is performed based on the overall displacement vector of the current frame and accumulated to obtain the drone's current position.

[0025] Existing technologies often employ image recognition to first identify people or ground vehicles, then perform key point detection using algorithms such as ORB on other areas, and finally use optical flow algorithms to track these key points and deduce the drone's own movement. This approach suffers from poor drone positioning accuracy.

[0026] This application provides a solution that avoids introducing errors by introducing a spin and emergency stop judgment mechanism, and then corrects the error caused by yaw based on the displacement vector of feature points in different regions, thereby further improving the positioning accuracy of the UAV.

[0027] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, such as a vision-based drone positioning device. The following description uses a vision-based drone positioning device as an example to illustrate this embodiment and the subsequent embodiments.

[0028] Based on this, embodiments of this application provide a vision-based UAV positioning method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the vision-based UAV positioning method of this application.

[0029] In this embodiment, the vision-based UAV positioning method includes steps S10 to S60: Step S10: Acquire high-altitude images using the camera mounted on the drone, and preprocess the high-altitude images to obtain preprocessed high-altitude images. The preprocessing includes at least grayscale conversion, image equalization, and noise reduction operations.

[0030] It should be noted that drones use cameras to capture stable high-altitude images to determine their own movement trajectory, and most importantly, to capture the overall movement of the image.

[0031] Understandably, when drones fly at high altitudes, the complex environment they encounter can result in high-altitude images containing significant noise and interference, which can negatively impact feature point selection and optical flow calculations. Therefore, preprocessing of the high-altitude images is necessary before feature point selection to improve image quality and clarity. Grayscale conversion converts color images to grayscale, reducing data volume and increasing processing speed; image equalization enhances contrast, making features more prominent; and noise reduction removes noise and interference, improving image clarity. These preprocessing operations result in a more stable high-altitude image, reducing noise interference in optical flow calculations and providing a solid foundation for subsequent feature point selection and optical flow calculations.

[0032] Step S20: Select feature points from the preprocessed high-altitude image to obtain a target feature point array.

[0033] It should be noted that feature point selection refers to pre-setting feature points for optical flow calculation in the entire image to form a target feature point array.

[0034] Understandably, a target feature point array should have the following characteristics: 1. The target feature points in the array are evenly distributed across the image, meaning they are uniformly distributed throughout the entire image to extract overall displacement information evenly. 2. The distance between target feature points in the array is appropriate. This means that while the distance between two target feature points should be sufficient to ensure that people on the ground and small moving units do not affect, or only affect, the optical flow calculation of one target feature point, the distance should not be too great to ignore too much ground feature information. Therefore, the spacing or number of target feature points needs to be balanced between these two mutually exclusive factors. 3. The target feature points in the array are fixed pixels. This means that the feature points calculated in each frame are all pixels at the target feature point's location. The displacement vector calculated in each optical flow instance measures the displacement of a fixed pixel. Therefore, the target feature points in the array cannot move with the pixels calculated in the current frame, nor will they be lost if the optical flow calculation fails to obtain a valid displacement vector in a particular frame.

[0035] It is worth noting that by setting the distribution of feature points reasonably, computational resources are saved while interference from ground movement information such as people or vehicles is greatly reduced.

[0036] In one feasible implementation, step S20 may include: selecting candidate feature points of a preset number of rows and columns in the preprocessed aerial image based on a preset coordinate point and according to a preset step size, wherein the preset coordinate point is the center point of the image, the preset step size is adjusted according to the resolution of the preprocessed aerial image, the preset number of rows is determined according to the height of the preprocessed aerial image and the preset step size, and the preset number of columns is determined according to the height and width of the preprocessed aerial image; calculating feature descriptors for the candidate feature points to obtain a feature descriptor for each candidate feature point; performing feature matching based on the feature descriptors to obtain matched feature point pairs; and filtering according to the geometric relationship between the matched feature point pairs to obtain a target feature point array, wherein the feature points in the target feature point array are uniformly distributed in the preprocessed aerial image.

[0037] It should be noted that the feature point selection rule starts from the preset coordinate point (X, Y) and is then calculated using... l The step size is set to (H / L-1) rows and (W / L-1) columns for candidate feature points, where W and H represent the width and height of the preprocessed aerial image, respectively. The preset coordinate point can be the center point of the image or any other location within the image, depending on the specific requirements. Step size l The selection of features needs to be adjusted according to the image resolution and the density requirements of feature points to ensure that the feature points are evenly distributed in the image.

[0038] Understandably, the feature descriptor calculation for candidate feature points can employ algorithms such as SIFT and SURF. By statistically analyzing the gradient and orientation information of pixels surrounding the candidate feature point, a feature descriptor uniquely identifying the feature point is obtained. Feature matching is then performed based on the similarity between feature descriptors, resulting in matched feature point pairs. Finally, based on the geometric relationships between the matched feature point pairs, mismatched feature point pairs are removed, yielding a target feature point array. These target feature point arrays are uniformly distributed in the preprocessed high-altitude image, saving computational power and reducing interference from ground-based mobile units. These will be used for subsequent optical flow calculations to achieve precise positioning of the UAV.

[0039] In practical implementation, when using SIFT (Scale Invariant Feature Transform) to calculate feature descriptors, the image is subjected to Gaussian blurring, and the gradient value of the image (G) is calculated. x and G y The gradient calculation formula is as follows: Where I(x,y) is the gray value of the image at position (x,y).

[0040] Calculate the gradient direction θ and magnitude m: θ=atan2(G y G x ) Then, the gradient directions around the gradient are statistically analyzed to form a histogram.

[0041] Select a 16×16 neighborhood around the key point, divide the region into 4×4 sub-regions, and calculate the gradient direction histogram (8 directions) for each sub-region. The final descriptor is a 4×4×8=128 dimensional vector.

[0042] It is worth noting that the K-nearest neighbor algorithm can be used to match feature descriptors to obtain initially matched feature point pairs; the Euclidean distance ratio between the initially matched feature point pairs is calculated, and feature point pairs with an Euclidean distance ratio less than a preset threshold are used as matched feature point pairs.

[0043] Step S30: Perform optical flow calculation and secondary processing of displacement vector based on the target feature point array to obtain the overall displacement vector of the current frame.

[0044] It should be noted that after setting up the target feature point array, optical flow calculation is performed on each pixel at a preset fixed point position. Because the target feature point array is not a high-quality feature point calculated through feature point detection, it is necessary to increase the number of layers, precision, and other parameters of the optical flow calculation function to ensure calculation accuracy.

[0045] Understandably, this implementation uses OpenCV's Lucas-Kanade optical flow calculation algorithm for individual pixels. This is an open-source algorithm used to observe the overall movement of an image. OpenCV's optical flow calculation function allows for direct parameter configuration to increase the number of calculation layers, thereby improving the accuracy of calculating the displacement of individual pixels.

[0046] In one feasible implementation, step S30 may include: using the Lucas-Kanade optical flow algorithm to perform optical flow calculation on the target feature point array to obtain the optical flow vector of each target feature point array from the previous frame to the current frame; calculating the displacement vector of each target feature point array according to the direction and magnitude of the optical flow vector; and performing secondary processing on the displacement vector of each target feature point array to obtain the overall displacement vector of the current frame.

[0047] It's important to note that optical flow is an algorithm for calculating the motion of objects in an image over time. In computer vision and image processing, optical flow is used to estimate the motion (i.e., velocity) of each pixel in an image, and it is commonly used to analyze dynamic information in video sequences. Its basic assumptions include: Brightness Constancy Assumption: This assumes that the brightness on the surface of an object remains constant across adjacent time frames. in, Indicates the position at time T ( Image brightness at ().

[0048] Small Motion Assumption: This assumption assumes that the motion of an object is relatively small over a short period of time, so as to make a linear approximation of the displacement.

[0049] It is worth noting that the derivation of the optical flow equation is based on the assumptions of constant brightness and small displacement. According to the linear approximation of the Taylor expansion, the following relationship is obtained: Where I(x,y,t) is the brightness value of the image at position (x,y) and time (t), and U and V are the horizontal and vertical optical flow components at position (x,y) in the image, respectively. These are the partial derivatives of image brightness I with respect to space and time, respectively.

[0050] The optical flow equation represents the spatial and temporal changes in the displacement of a pixel in an image. The goal is to estimate the optical flow vector for each pixel by solving this equation.

[0051] It is worth noting that optical flow calculation is a typical inverse problem; it is not a univariate equation but contains multiple unknowns (the horizontal and vertical optical flow components (u) and (v) for each pixel). Therefore, optical flow calculation usually requires additional assumptions or constraints to solve.

[0052] The Lucas-Kanade method is another widely used approach for optical flow calculation. It assumes that the optical flow is constant within local regions of an image, and that the optical flow of each pixel can be inferred from the brightness variations of its neighboring pixels. It uses gradient information from local image regions to estimate the optical flow. in, , , , , and (Ix,Iy,It) are the partial derivatives of the image brightness function I(x,y,t) with respect to (x,y,t).

[0053] The Lucas-Kanade algorithm estimates the horizontal and vertical optical flow components (u) and (v) for each pixel by solving this set of equations.

[0054] In one feasible implementation, the secondary processing of the displacement vectors of each of the target feature point arrays to obtain the overall displacement vector of the current frame includes: extracting the horizontal component of the displacement vector of each of the target feature point arrays, and determining an average horizontal component and a standard deviation of the horizontal component based on the horizontal component; filtering the displacement vectors of each of the target feature point arrays based on the average horizontal component and the standard deviation of the horizontal component, and removing displacement vectors whose horizontal component is more than a preset multiple of the standard deviation of the horizontal component, to obtain the displacement vector after initial filtering; extracting the vertical component of the displacement vector after initial filtering, and determining an average vertical component and a standard deviation of the vertical component based on the vertical component; filtering the displacement vector after initial filtering again based on the average vertical component and the standard deviation of the vertical component, and removing displacement vectors whose vertical component is more than a preset multiple of the standard deviation of the vertical component, to obtain the displacement vector after secondary filtering; and calculating the average value of the displacement vector after secondary filtering to obtain the overall displacement vector of the current frame.

[0055] It should be noted that in this embodiment, the displacement vectors calculated between each frame are stored in an array and then subjected to two standard deviation filtering processes.

[0056] First, extract the X component of each displacement vector, calculate the mean of the X components, and then calculate the standard deviation. Then, remove displacement vectors whose X component is more than A times the standard deviation from the mean, where A is an adjustable natural number.

[0057] Then, extract the Y-component of the remaining displacement vectors, calculate the mean of the Y-components, and then determine the standard deviation. Next, remove displacement vectors whose Y-components are more than A times the standard deviation from the mean, where A is an adjustable natural number.

[0058] It's worth mentioning that the average of the filtered displacement vectors is used to obtain the overall displacement vector of the current frame. Taking the initial frame as the origin (0,0), the average displacements calculated between each frame are accumulated to obtain the current coordinates.

[0059] Step S40: Determine the spin state of the UAV and obtain a first determination result.

[0060] It should be noted that spin state refers to the state in which a drone rotates around its own axis during flight. This state may be caused by external environmental factors (such as wind shear) or a malfunction in the drone's own control system. Spin state not only affects the drone's flight stability and trajectory accuracy but may also pose a threat to the drone's safety. Therefore, timely and accurate determination of the drone's spin state and taking appropriate control measures are crucial.

[0061] In one feasible implementation, step S40 may include: dividing the high-altitude image into quadrants to obtain high-altitude images in four quadrants; calculating the average displacement vector in the high-altitude images in the four quadrants respectively; determining whether the UAV is in a spin state based on the horizontal and vertical component signs of the average displacement vector in the high-altitude images in the four quadrants; if the horizontal and vertical component signs of the average displacement vector in the high-altitude images in the four quadrants meet preset conditions and the duration exceeds a preset time threshold, then the first determination result is that the UAV is in a spin state; if the horizontal and vertical component signs of the average displacement vector in the high-altitude images in the four quadrants do not meet preset conditions and / or the duration does not exceed the preset time threshold, then the first determination result is that the UAV is in a non-spin state.

[0062] It should be noted that if the wind direction is unstable while the drone is hovering, it may cause the drone to spin continuously without being able to stop itself. Therefore, it is necessary to identify this state and take appropriate measures.

[0063] It is understandable that when a drone hovers and spins, the direction of the average displacement vector in the high-altitude images of the four quadrants is fixed. Taking the drone rotating clockwise (i.e., the image rotating counterclockwise) as an example, the characteristics of the average displacement direction are as follows: the horizontal displacement is negative and the vertical displacement is positive in the first quadrant; the horizontal displacement is negative and the vertical displacement is negative in the second quadrant; the horizontal displacement is positive and the vertical displacement is negative in the third quadrant; and the horizontal displacement is positive and the vertical displacement is positive in the fourth quadrant. Figure 2 As shown, Figure 2 This is a schematic diagram of the average displacement direction of the drone rotating clockwise (the image rotates counterclockwise).

[0064] It is worth noting that when the UAV rotates counterclockwise (the image rotates clockwise), the component directions in corresponding directions of adjacent quadrants are opposite. Therefore, the horizontal and vertical component signs of the average displacement vector in the high-altitude images of the four quadrants can be used to determine the UAV's rotation direction. If the following preset conditions are met simultaneously—a positive horizontal displacement and a negative vertical displacement in the first quadrant, a positive horizontal displacement and a positive vertical displacement in the second quadrant, a negative horizontal displacement and a positive vertical displacement in the third quadrant, and a negative horizontal displacement and a positive vertical displacement in the fourth quadrant—and the duration exceeds a preset time threshold, then the UAV's rotation direction is determined to be counterclockwise, indicating that it has undergone spin. The preset time threshold can be 10 seconds. By determining the UAV's rotation direction, we can further understand its flight status and provide important information for subsequent control decisions.

[0065] In practical implementation, when determining the UAV's spin state, since spin states include both clockwise and counterclockwise rotations, the corresponding preset conditions can be that the horizontal and vertical components of the average displacement vector in the high-altitude images of the four quadrants correspond to the directional characteristics of clockwise and counterclockwise rotation, respectively. For example, during clockwise rotation, the preset conditions could be that the horizontal displacement in the first quadrant is negative and the vertical displacement is positive, the horizontal displacement in the second quadrant is negative and the vertical displacement is negative, the horizontal displacement in the third quadrant is positive and the vertical displacement is negative, and the horizontal displacement in the fourth quadrant is positive and the vertical displacement is positive. Conversely, during counterclockwise rotation, the preset conditions are the opposite. These preset conditions are derived from the changing patterns of the displacement vector in the images during UAV spin. When the UAV is in a spin state, due to the influence of rotation, the average displacement vector in the four quadrants will exhibit specific directions. Therefore, by detecting these directional characteristics, it is possible to effectively determine whether the UAV is in a spin state.

[0066] The preset time threshold is set to ensure the accuracy of the judgment results. Because brief fluctuations in external environmental factors or the drone's own control system may cause temporary changes in the displacement vector, such changes are insufficient to indicate that the drone is in a spinning state. Therefore, a time threshold needs to be set; only when the duration of the displacement vector change exceeds this threshold is the drone considered to be in a spinning state.

[0067] By accurately judging the spin state of the drone, potential flight safety issues can be detected in a timely manner, and corresponding control measures can be taken to ensure the flight stability and safety of the drone.

[0068] Furthermore, the sign of the X and Y components of the average displacement vector in each quadrant is detected. If the preset condition is met for more than 10 seconds, the drone is determined to have entered a spin state, and self-resolvement is initiated. When the drone is in an area with turbulent airflow, a spin state may occur that the flight controller cannot resolve on its own. If this state is not resolved in time, the drone's spin will increase. After this state occurs, manually manipulating the joystick to move the drone forward or backward a short distance will resolve the state. Therefore, after identifying that the drone has been in a self-selected state for a long time using the above method, sending a command to move the drone forward or backward by the same distance will resolve the state.

[0069] Step S50: If the first judgment result indicates that the UAV is in a non-spinning state, perform an emergency stop judgment on the UAV to obtain a second judgment result.

[0070] It should be noted that when a drone makes an emergency stop, it will generate an excessively large pitch angle. If the maximum compensation angle of the gimbal is less than the pitch angle during the emergency stop, perspective interference will be introduced into the image captured by the camera. The characteristics of perspective interference can be used to determine the emergency stop state.

[0071] In one feasible implementation, step S50 may include: if the first determination result indicates that the UAV is in a non-spinning state, dividing the high-altitude image into regions to obtain a left region and a right region; obtaining a displacement vector matrix, and extracting and processing elements of the left region and the right region based on the displacement vector matrix to obtain the average displacement vector of the left region and the average displacement vector of the right region; if the difference between the average displacement vector of the left region and the average displacement vector of the right region exceeds a preset threshold, determining the second determination result as the UAV being in an emergency stop state; if the difference between the average displacement vector of the left region and the average displacement vector of the right region does not exceed the preset threshold, determining the second determination result as the UAV being in a non-emergency stop state.

[0072] It should be noted that in this embodiment, optical flow calculation is performed only once per frame, and the result of that optical flow calculation, i.e., the displacement vector matrix, is directly extracted in subsequent processing. Elements of different regions can then be directly extracted from the displacement vector matrix for further processing, effectively improving processing efficiency.

[0073] It is understandable that when the camera is tilted upwards due to the drone's sudden stop (i.e., the camera is not perpendicular to the ground), the side of the camera that is higher records a larger area due to perspective principles. This results in the average displacement vector of the preset feature points on that side, calculated by optical flow, being much larger than the average displacement vector on the other side. Dividing the aerial image into a left region (A) and a right region (B), the relationship between the average displacement vectors of regions A and B is as follows: or The method for calculating the average displacement of a region is as follows: when the difference between the average displacements of region A and region B exceeds a certain threshold, i.e.: This indicates that the drone has made an emergency stop.

[0074] In practical implementation, the high-altitude image can be divided into an upper region (C) and a lower region (D). When the difference in average displacement between region C and region D exceeds a certain threshold, i.e.: This indicates that the drone has made an emergency stop.

[0075] Step S60: If the second judgment result indicates that the UAV is in a non-emergency stop state, perform yaw correction based on the overall displacement vector of the current frame and accumulate it to obtain the current position of the UAV.

[0076] It should be noted that yaw correction refers to correcting the interference factors introduced by the rotation of the UAV relative to the ground after the UAV rotates, so that the overall displacement vector calculated in each frame is more accurate, thereby ensuring the position accuracy of the UAV.

[0077] In one feasible implementation, step S60 may further include: calculating the angle of the UAV's spin motion (excluding horizontal movement) based on the overall displacement vector of the current frame; and correcting the calculated overall displacement vector based on the calculated angle. Through yaw correction, the overall displacement vector calculated in each frame can be made closer to the actual displacement vector of the UAV, thereby improving the positioning accuracy of the UAV.

[0078] In practice, the accuracy of yaw correction can be further improved by combining information such as the speed and altitude of the drone.

[0079] It is worth noting that if the second judgment result indicates that the drone is in an emergency stop state, the displacement vector of the current frame is set to 0 to reduce the calculation error caused by the lens pitch due to the emergency stop.

[0080] This embodiment provides a vision-based UAV positioning method. The method involves acquiring high-altitude images using a camera mounted on the UAV, and preprocessing these images to obtain a preprocessed high-altitude image. The preprocessing includes at least grayscale conversion, image equalization, and noise reduction. Feature points are selected from the preprocessed high-altitude image to obtain a target feature point array. Optical flow calculation and secondary displacement vector processing are performed based on the target feature point array to obtain the overall displacement vector of the current frame. The UAV's spin state is determined to obtain a first determination result. If the first determination result indicates that the UAV is in a non-spin state, an emergency stop determination is performed to obtain a second determination result. If the second determination result indicates that the UAV is not in an emergency stop state, yaw correction is performed based on the overall displacement vector of the current frame and accumulated to obtain the UAV's current position. By introducing spin and emergency stop determination mechanisms, errors are avoided, and errors caused by yaw are corrected based on the displacement vectors of feature points in different regions, further improving the accuracy of UAV positioning.

[0081] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 Step S60 includes steps S601 to S603: Step S601: If the second judgment result indicates that the UAV is in a non-emergency stop state, calculate the yaw angle based on the overall displacement vector of the current frame to obtain the yaw angle.

[0082] It should be noted that the yaw angle is the drone's spin angle, that is, the angle by which the drone yaws and rotates left and right. By calculating the yaw angle, we can understand the drone's rotation during flight, providing crucial data for subsequent positioning corrections. The yaw angle can be calculated based on the overall displacement vector of the current frame, combined with information such as the drone's flight attitude and velocity through comprehensive analysis.

[0083] In practice, yaw angle can be calculated using either geometric or vector methods. Geometric methods primarily calculate the yaw angle by measuring the angle between the UAV's current position and the predetermined flight path; while vector methods calculate the yaw angle by measuring the angle between the vector of the UAV's current flight direction and the vector of the predetermined flight path. Each method has its advantages and disadvantages, and the appropriate method can be chosen based on the specific circumstances.

[0084] In one feasible implementation, step S601 may include: determining the vertical heading angle offset and the horizontal heading angle offset based on the overall displacement vector of the current frame; determining the interference displacement vector based on the vertical heading angle offset and the horizontal heading angle offset; and determining the yaw angle based on the interference displacement vector.

[0085] It should be noted that during operation, the UAV may yaw actively or passively. Yaw introduces interference displacement vectors into the optical flow calculation of the target feature point array, therefore, it is necessary to perform yaw correction on the calculated average displacement vector and accumulate it.

[0086] Understandably, displacement calculation methods essentially detect the displacement of points on an image. The displacement vector calculated by the camera based on the pixels in the image is the resultant displacement vector, which is the displacement component affected by both translation and rotation caused by yaw. What we need is the translation displacement vector (which is also the actual displacement vector of the camera), because the displacement vector calculated by yaw is an error term or interference term. Therefore, reducing the displacement vector caused by yaw aims to reduce calculation errors and thus improve calculation accuracy.

[0087] In the specific implementation, the image captured by the camera is divided into two equal parts, namely the left region (A) and the right region (B), as follows: Figure 4 As shown, Figure 4 This is a diagram showing how an image can be divided into left and right sides.

[0088] If there is no change in yaw angle, the displacement vectors calculated from the two parts should remain parallel and their magnitudes should not differ significantly. That is, when there is no yaw, the displacement vectors of each feature point should be parallel, such as... Figure 5 As shown, Figure 5 This is a schematic diagram of the displacement vectors of various feature points when there is no yaw.

[0089] If yaw occurs, an additional interference displacement vector will be introduced into the calculation of the displacement vector between each frame, causing the calculated displacement vector to shift, such as... Figure 6 As shown, Figure 6 A schematic diagram showing the different degrees of displacement of the displacement vectors calculated for different feature points when yaw occurs.

[0090] The displacement vector calculated when yaw occurs is: The actual displacement vector can then be obtained: In the left region (region A), when the UAV only yaws, the x-components of the displacement vector calculated from the target feature point array symmetrical about the X-axis are the same in magnitude but opposite in direction, such as... Figure 7 As shown, Figure 7 This is a schematic diagram of the x-component of the displacement vector symmetric about the x-axis in region A.

[0091] Therefore, averaging all displacement vectors in region A yields an average displacement of (0, y). Similarly, averaging all displacements in region B yields an average displacement of (0, -y).

[0092] Based on the above algorithm, the calculated average displacement of regions A and B can be obtained. Given that the actual displacements of regions A and B are the same, the following can be calculated: Therefore, the heading angle offset in the Y-axis direction is: If the image is 2L long and 2W wide (i.e., region A is 2W long and 2W wide), then the displacement vector Y component occurs at a distance r = 0.5 * L from the rotation center.

[0093] Similarly, divide the image into two equal parts, namely region C and region D, as follows: Figure 8 As shown, Figure 8 This is a diagram showing the image divided into upper and lower sections.

[0094] It can be calculated that: If the image is 2L long and 2W wide (i.e., region C is 2L long and W wide), then the X component of the displacement vector occurs at a distance r = 0.5 * W from the rotation center.

[0095] Since the X and Y components calculated above occur at different distances from the center (r), the disturbance displacement vector introduced by yaw at a distance of r = 0.5 * L from the rotation center can be obtained as ( If y), then the angle of deflection can be calculated as: Step S602: Correct the overall displacement vector of the current frame according to the yaw angle and the reference displacement vector to obtain the corrected overall displacement vector, wherein the reference displacement vector is a fixed reference value obtained by optical flow calculation and displacement vector processing based on a preset feature point array.

[0096] It should be noted that when a drone yaws (rotates relative to the ground), the yaw angle needs to be calculated because this means that the drone's coordinate system has shifted from the ground's coordinate system, and the calculation error caused by this angle needs to be corrected. This correction essentially involves converting the drone's displacement vector into a displacement vector in the ground coordinate system.

[0097] Understandably, the reference displacement vector is a fixed reference value obtained by optical flow calculation and displacement vector processing based on a preset feature point array, and it serves as a benchmark throughout the positioning process. By comparing the overall displacement vector of the current frame with the reference displacement vector and combining it with the calculated yaw angle, errors caused by yaw can be corrected more accurately.

[0098] In the specific implementation, based on the reference displacement vector and The average displacement vector calculated during yaw can be corrected, and the correction value is: Where N is the modulus of (X, Y), = , The angle of deflection.

[0099] By calculating and then summing the corrected overall displacement vector, we can obtain the continuous position information of the UAV during flight, thereby achieving precise positioning of the UAV. The angles are also calculated cumulatively, and need to be inverted when used. This is because the calculated angle is offset relative to the original direction. The angle needs to be corrected back in the opposite direction. Once the drone veers off course, The effect of angles is always present, so they need to be accumulated.

[0100] Optionally, if it is determined that yaw correction is needed, the corrected overall displacement vector is calculated based on the yaw angle, the UAV's current flight speed, altitude, and other information, combined with a preset yaw correction model or algorithm. The corrected overall displacement vector can more accurately reflect the UAV's actual position on the predetermined flight path.

[0101] Step S603: Accumulate the corrected overall displacement vector to determine the current position of the UAV.

[0102] It should be noted that the UAV's current position is updated based on the corrected global displacement vector. After obtaining the corrected global displacement vector, it is applied to the UAV's position update algorithm to calculate the UAV's current position in three-dimensional space. This position information can be used for various aspects of the UAV, such as navigation, path planning, and obstacle avoidance, and is the foundation for the UAV's autonomous flight and mission completion.

[0103] In practical implementation, the drone's current position can be gradually updated through accumulation, based on its initial position and the corrected overall displacement vector. The drone's initial position can be obtained through positioning devices such as GPS, while the corrected overall displacement vector reflects the drone's actual movement during flight. By continuously updating the drone's current position, real-time tracking and monitoring of its flight trajectory can be achieved, providing strong support for autonomous flight and remote control. Furthermore, after determining the drone's current position, path planning and obstacle avoidance decisions can be performed by combining factors such as the drone's mission requirements and environmental information, thereby achieving more intelligent and autonomous drone flight control.

[0104] In this embodiment, by processing preset feature points in different areas separately, errors caused by yaw can be corrected, thereby effectively improving the accuracy of UAV point positioning.

[0105] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the vision-based UAV positioning method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0106] This application also provides a vision-based drone positioning device, please refer to... Figure 9 The device includes: The preprocessing module 10 is used to acquire high-altitude images through the camera mounted on the UAV and preprocess the high-altitude images to obtain preprocessed high-altitude images. The preprocessing includes at least grayscale conversion, image equalization and noise reduction operations.

[0107] The selection module 20 is used to select feature points from the preprocessed high-altitude image to obtain a target feature point array.

[0108] The filtering module 30 is used to perform optical flow calculation and secondary processing of displacement vector based on the target feature point array to obtain the overall displacement vector of the current frame.

[0109] The judgment module 40 is used to judge the spin state of the UAV and obtain a first judgment result.

[0110] The judgment module 40 is further configured to determine the emergency stop state of the drone when the first judgment result indicates that the drone is in a non-spinning state, and obtain a second judgment result.

[0111] The correction module 50 is used to perform yaw correction and accumulate the total displacement vector of the current frame when the second judgment result indicates that the UAV is in a non-emergency stop state, so as to obtain the current position of the UAV.

[0112] The vision-based UAV positioning device provided in this application, employing the vision-based UAV positioning method described in the above embodiments, can solve the technical problem of poor UAV positioning accuracy in the prior art. Compared with the prior art, the beneficial effects of the vision-based UAV positioning device provided in this application are the same as those of the vision-based UAV positioning method provided in the above embodiments, and other technical features in the vision-based UAV positioning device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0113] This application provides a vision-based drone positioning device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the vision-based drone positioning method in Embodiment 1 above.

[0114] The following is for reference. Figure 10 The diagram illustrates a structural schematic of a vision-based drone positioning device suitable for implementing embodiments of this application. The vision-based drone positioning device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 10 The vision-based drone positioning device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0115] like Figure 10As shown, a vision-based UAV positioning device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in ROM 1002 (Read Only Memory) or a program loaded from storage device 1003 into RAM 1004 (Random Access Memory). RAM 1004 also stores various programs and data required for the operation of the vision-based UAV positioning device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. I / O interface 1006 (input / output interface) is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, LCDs (Liquid Crystal Displays), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the vision-based UAV positioning device to communicate wirelessly or wiredly with other devices to exchange data. While the figures show vision-based UAV positioning devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0116] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0117] The vision-based UAV positioning device provided in this application, employing the vision-based UAV positioning method described in the above embodiments, can solve the technical problem of poor UAV positioning accuracy in the prior art. Compared with the prior art, the beneficial effects of the vision-based UAV positioning device provided in this application are the same as those of the vision-based UAV positioning method provided in the above embodiments, and other technical features in this vision-based UAV positioning device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0118] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0119] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0120] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the vision-based UAV positioning method in the above embodiments.

[0121] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory or Flash Memory), optical fibers, CD-ROM (CD-Read Only Memory), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0122] The aforementioned computer-readable storage medium may be included in a vision-based drone positioning device; or it may exist independently and not assembled into a vision-based drone positioning device.

[0123] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by a vision-based UAV positioning device, the vision-based UAV positioning device performs the following actions: acquires high-altitude images using a camera mounted on the UAV, and preprocesses the high-altitude images to obtain preprocessed high-altitude images, wherein the preprocessing includes at least grayscale conversion, image equalization, and noise reduction operations; selects feature points from the preprocessed high-altitude images to obtain a target feature point array; performs optical flow calculation and secondary displacement vector processing based on the target feature point array to obtain the overall displacement vector of the current frame; determines the spin state of the UAV to obtain a first determination result; if the first determination result indicates that the UAV is in a non-spin state, determines the UAV's emergency stop state to obtain a second determination result; if the second determination result indicates that the UAV is in a non-emergency stop state, performs yaw correction based on the overall displacement vector of the current frame and accumulates the results to obtain the UAV's current position.

[0124] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including LAN (Local Area Network) or WAN (Wide Area Network)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0125] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0126] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0127] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described vision-based UAV positioning method, thereby solving the technical problem of poor UAV positioning accuracy in the prior art. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the vision-based UAV positioning method provided in the above embodiments, and will not be repeated here.

[0128] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the vision-based UAV positioning method described above.

[0129] The computer program product provided in this application can solve the technical problem of poor positioning accuracy of existing UAVs. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the vision-based UAV positioning method provided in the above embodiments, and will not be repeated here.

[0130] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A vision-based unmanned aerial vehicle positioning method, characterized in that, The method comprises: collecting high-altitude images by a camera carried by a UAV, and preprocessing the high-altitude images to obtain preprocessed high-altitude images, wherein the preprocessing at least includes grayscale conversion, image equalization and noise reduction operation; performing feature point selection on the preprocessed high-altitude images to obtain a target feature point array; performing optical flow calculation and displacement vector secondary processing based on the target feature point array to obtain an overall displacement vector of the current frame; judging the self-rotation state of the UAV to obtain a first judgment result; in the case that the first judgment result is that the UAV is in a non-self-rotation state, judging the emergency stop state of the UAV to obtain a second judgment result; in the case that the second judgment result is that the UAV is in a non-emergency stop state, performing yaw correction and accumulation based on the overall displacement vector of the current frame to obtain the current position of the UAV.

2. The method of claim 1, wherein, The feature point selection on the preprocessed high-altitude images to obtain a target feature point array comprises: taking a preset coordinate point as a reference, selecting a preset number of rows and a preset number of columns of candidate feature points in the preprocessed high-altitude images according to a preset step length, wherein the preset coordinate point is the center point of the image, the preset step length is adjusted according to the resolution of the preprocessed high-altitude images, the preset number of rows is determined according to the height of the preprocessed high-altitude images and the preset step length, and the preset number of columns is determined according to the height and width of the preprocessed high-altitude images; performing feature descriptor calculation on the candidate feature points to obtain a feature descriptor of each candidate feature point; performing feature matching based on the feature descriptors to obtain matched feature point pairs; screening according to the geometric relationship between the matched feature point pairs to obtain a target feature point array, wherein the feature points in the target feature point array are uniformly distributed in the preprocessed high-altitude images.

3. The method of claim 1, wherein, The optical flow calculation and displacement vector secondary processing based on the target feature point array to obtain an overall displacement vector of the current frame comprises: performing optical flow calculation on the target feature point array using the Lucas-Kanade optical flow algorithm to obtain the optical flow vector of each target feature point array from the previous frame to the current frame; calculating the displacement vector of each target feature point array according to the direction and size of the optical flow vector; performing secondary processing on the displacement vectors of the target feature point arrays to obtain an overall displacement vector of the current frame.

4. The method of claim 3, wherein, The secondary processing on the displacement vectors of the target feature point arrays to obtain an overall displacement vector of the current frame comprises: extracting the horizontal components of the displacement vectors of the target feature point arrays, and determining the average horizontal component and the horizontal component standard deviation based on the horizontal components; screening the displacement vectors of the target feature point arrays based on the average horizontal component and the horizontal component standard deviation, and removing the displacement vectors whose horizontal components are more than a preset multiple of the horizontal component standard deviation to obtain the displacement vectors after the first screening; extracting the vertical components of the displacement vectors after the first screening, and determining the average vertical component and the vertical component standard deviation based on the vertical components; Secondly, the displacement vectors after the primary screening are screened again based on the average vertical component and the vertical component standard deviation, and the displacement vectors with a vertical component distance greater than a preset multiple of the vertical component standard deviation from the average vertical component are removed, to obtain displacement vectors after secondary screening. Thirdly, an average value is calculated based on the displacement vectors after the secondary screening, to obtain an overall displacement vector of the current frame.

5. The method of claim 1, wherein, The self-spin state judgment on the unmanned aerial vehicle to obtain a first judgment result comprises: The high-altitude image is divided into four quadrants to obtain high-altitude images of the four quadrants; Average displacement vectors in the high-altitude images of the four quadrants are calculated respectively; Whether the unmanned aerial vehicle is in a self-spin state is judged according to the signs of the horizontal direction components and the vertical direction components of the average displacement vectors in the high-altitude images of the four quadrants; If the signs of the horizontal direction components and the vertical direction components of the average displacement vectors in the high-altitude images of the four quadrants satisfy a preset condition and the duration exceeds a preset time threshold, it is determined that the first judgment result is that the unmanned aerial vehicle is in a self-spin state; If the signs of the horizontal direction components and the vertical direction components of the average displacement vectors in the high-altitude images of the four quadrants do not satisfy the preset condition and / or the duration does not exceed the preset time threshold, it is determined that the first judgment result is that the unmanned aerial vehicle is in a non-self-spin state.

6. The method of claim 1, wherein, The emergency stop state judgment on the unmanned aerial vehicle in the case that the first judgment result is that the unmanned aerial vehicle is in a non-self-spin state to obtain a second judgment result comprises: In the case that the first judgment result is that the unmanned aerial vehicle is in a non-self-spin state, the high-altitude image is divided into a left region and a right region; A displacement vector matrix is obtained, and elements in the left region and the right region are extracted and processed based on the displacement vector matrix to obtain average displacement vectors of the left region and the right region; If the difference between the average displacement vectors of the left region and the right region exceeds a preset threshold, it is determined that the second judgment result is that the unmanned aerial vehicle is in an emergency stop state; If the difference between the average displacement vectors of the left region and the right region does not exceed the preset threshold, it is determined that the second judgment result is that the unmanned aerial vehicle is in a non-emergency stop state.

7. The method of claim 1, wherein, In the case that the second judgment result is that the unmanned aerial vehicle is in a non-emergency stop state, the overall displacement vector of the current frame is used for yaw correction and accumulation to obtain a current position of the unmanned aerial vehicle, which comprises: In the case that the second judgment result is that the unmanned aerial vehicle is in a non-emergency stop state, a yaw angle is calculated based on the overall displacement vector of the current frame; The overall displacement vector of the current frame is corrected according to the yaw angle and a reference displacement vector, to obtain a corrected overall displacement vector, wherein the reference displacement vector is a fixed reference value obtained by optical flow calculation and displacement vector processing on a preset feature point array; The corrected overall displacement vector is used for accumulation to determine the current position of the unmanned aerial vehicle.

8. The method of claim 7, wherein, The yaw angle calculation based on the overall displacement vector of the current frame to obtain a yaw angle comprises: determine a yaw angle offset in a vertical direction and a yaw angle offset in a horizontal direction based on the overall displacement vector of the current frame; determine an interference displacement vector based on the yaw angle offset in the vertical direction and the yaw angle offset in the horizontal direction; determine a yaw angle based on the interference displacement vector.

9. A vision-based unmanned aerial vehicle positioning device, comprising: Comprise: a preprocessing module configured to collect high-altitude images by a camera carried by a UAV, and to preprocess the high-altitude images to obtain preprocessed high-altitude images, wherein the preprocessing at least includes grayscale conversion, image equalization, and noise reduction operations; a selection module configured to select feature points from the preprocessed high-altitude images to obtain a target feature point array; a screening module configured to perform optical flow calculation and displacement vector secondary processing based on the target feature point array to obtain an overall displacement vector of a current frame; a judgment module configured to judge a self-rotation state of the UAV to obtain a first judgment result; the judgment module is further configured to, if the first judgment result indicates that the UAV is in a non-self-rotation state, judge an emergency stop state of the UAV to obtain a second judgment result; a correction module configured to, if the second judgment result indicates that the UAV is in a non-emergency stop state, perform yaw correction and accumulation based on the overall displacement vector of the current frame to obtain a current position of the UAV.

10. A storage medium, characterized by The storage medium has a visual-based UAV positioning program stored thereon, and the visual-based UAV positioning program, when executed by the processor, implements the method of any one of claims 1 to 8.

Images