Unmanned aerial vehicle positioning method and apparatus, unmanned aerial vehicle, and storage medium

Abstract

This application relates to a method, apparatus, drone, computer-readable storage medium, and computer program product for drone positioning. The method includes: real-time acquisition of scene images of the scene where the drone is located; determining the optical flow field between two adjacent frames of the scene, and the rigid optical flow field of the scene caused by the drone's movement, based on two adjacent frames of the scene images; and determining the drone's position coordinates through the optical flow field and the rigid optical flow field. This method enables drone positioning, ensuring the drone can operate normally even under signal interference or obstruction, is not limited by lighting conditions, and has a wide range of applications. Furthermore, this method enables autonomous drone positioning, eliminating the need for interaction between the drone and ground control facilities, resulting in higher real-time positioning performance.

Description

Technical Field

[0001] This application relates to the field of automatic control technology, and in particular to a method, apparatus, drone, computer-readable storage medium, and computer program product for locating unmanned aerial vehicles (UAVs). Background Technology

[0002] With the continuous development of the drone industry, the number and importance of civilian drones in indoor and outdoor environments are gradually increasing, such as express delivery, aerial photography for geographic mapping, temporary network coverage in disaster areas, and reconnaissance within nuclear power plants. Drones must fly autonomously, either fully or partially, to complete their designated missions. To support autonomous flight, ground control facilities need to continuously locate and monitor the drone's position and send this information to the drone's navigation controller to provide autonomous navigation capabilities.

[0003] Currently, drone positioning commonly uses satellite inertial navigation technology. However, satellite inertial navigation is generally suitable for open outdoor environments. In indoor environments, signal interference or obstructions can lead to decreased positioning accuracy, preventing drones from functioning properly. Therefore, there is an urgent need for a drone positioning method that can overcome the inaccuracies of existing methods under signal interference or obstruction conditions, ensuring the normal operation of drones. Summary of the Invention

[0004] Therefore, it is necessary to provide a drone positioning method, device, drone, computer-readable storage medium, and computer program product that can accurately locate a drone under signal interference or obstruction conditions, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a method for locating a drone. The method includes:

[0006] Real-time acquisition of scene images of the area where the drone is located;

[0007] Based on two adjacent frames of the scene image, determine the optical flow field of the scene between the two adjacent frames of the scene image, as well as the rigid optical flow field of the scene caused by the movement of the UAV;

[0008] The position coordinates of the UAV are determined by the optical flow field and the rigid optical flow field.

[0009] In one embodiment, determining the optical flow field of the scene between two adjacent frames based on two adjacent scene images includes:

[0010] Extract the common features of the scene in two adjacent frames of the scene image;

[0011] The optical flow field of the scene is obtained based on the common features of two adjacent frames of the scene image.

[0012] In one embodiment, obtaining the optical flow field of the scene based on common features of two adjacent frames of the scene images includes:

[0013] Feature enhancement is performed on the common features of two adjacent frames of the scene images to obtain optimized features;

[0014] Feature matching is performed on the optimized features of two adjacent frames of the scene image to obtain the matching result;

[0015] Based on the matching results, the estimated optical flow of each pixel in the scene is determined, and the optical flow field is obtained.

[0016] In one embodiment, extracting common features of the scene in two adjacent frames of the scene images includes:

[0017] The common features of the scene in each frame of the scene image are extracted using a feature extraction model; wherein, the training method of the feature extraction model includes:

[0018] Obtain a training dataset; the training dataset includes consecutive daytime images and consecutive nighttime images of the same scene;

[0019] By using the feature extraction model to be trained, features are extracted from consecutive daytime images and consecutive nighttime images of the same scene to obtain common daytime features, private daytime features, common nighttime features, and private nighttime features.

[0020] Based on the common daytime features, the private daytime features, the common nighttime features, and the private nighttime features, daytime image reconstruction and nighttime image reconstruction are performed to obtain daytime reconstructed images and nighttime reconstructed images;

[0021] Based on the training dataset, the daytime reconstructed image, and the nighttime reconstructed image, a loss function value is calculated, and the parameters of the feature extraction model to be trained are updated with the goal of minimizing the loss function value, until the training termination condition is met, thus obtaining the feature extraction model that has been trained.

[0022] In one embodiment, obtaining the training dataset includes:

[0023] A day-to-night image transformation network was trained using unpaired day and night images of the same scene;

[0024] The trained day-night image conversion network is used to generate consecutive night images corresponding to consecutive day images to obtain a training dataset.

[0025] In one embodiment, the real-time acquisition of scene images of the scene where the drone is located includes: real-time acquisition of scene images of the scene where the drone is located, and obtaining the depth field of each frame of the scene image;

[0026] The determination of the rigid optical flow field of the scene caused by the movement of the drone based on two adjacent frames of the scene image includes:

[0027] Based on the depth field, preset stiffness field, and camera pose of two adjacent frames of the scene image, determine the rigid optical flow expression of each pixel in the scene image;

[0028] The rigid optical flow field of the scene is obtained by using the rigid optical flow expression of each pixel in the scene image.

[0029] In one embodiment, determining the position coordinates of the UAV using the optical flow field and the rigid optical flow field includes:

[0030] Modeling is performed based on the relationship between the optical flow field and the rigid optical flow field to determine the pose transformation matrix of the camera in the scene;

[0031] The camera pose of the scene is determined based on the pose transformation matrix;

[0032] The position coordinates of the UAV are determined based on the camera's orientation.

[0033] Secondly, this application also provides a drone positioning device. The device includes:

[0034] The data acquisition module is used to acquire scene images of the drone's location in real time.

[0035] The optical flow field determination module is used to determine the optical flow field between two adjacent frames in the scene, and the rigid optical flow field of the scene caused by the movement of the UAV, based on two adjacent frames of the scene image.

[0036] A positioning module is used to determine the position coordinates of the UAV through the rigid optical flow field and the optical flow field.

[0037] Thirdly, this application also provides a drone. The drone includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0038] Real-time acquisition of scene images of the area where the drone is located;

[0039] Based on two adjacent frames of the scene image, determine the optical flow field of the scene between the two adjacent frames of the scene image, as well as the rigid optical flow field of the scene caused by the movement of the UAV;

[0040] The position coordinates of the UAV are determined by the optical flow field and the rigid optical flow field.

[0041] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0042] Real-time acquisition of scene images of the area where the drone is located;

[0043] Based on two adjacent frames of the scene image, determine the optical flow field of the scene between the two adjacent frames of the scene image, as well as the rigid optical flow field of the scene caused by the movement of the UAV;

[0044] The position coordinates of the UAV are determined by the optical flow field and the rigid optical flow field.

[0045] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0046] Real-time acquisition of scene images of the area where the drone is located;

[0047] Based on two adjacent frames of the scene image, determine the optical flow field of the scene between the two adjacent frames of the scene image, as well as the rigid optical flow field of the scene caused by the movement of the UAV;

[0048] The position coordinates of the UAV are determined by the optical flow field and the rigid optical flow field.

[0049] The aforementioned UAV positioning method, device, UAV, computer-readable storage medium, and computer program product acquire scene images of the scene where the UAV is located in real time; based on two adjacent scene images, determine the optical flow field between the two adjacent frames in the scene, as well as the rigid optical flow field of the scene caused by the movement of the UAV; determine the position coordinates of the UAV through the optical flow field and the rigid optical flow field. This yields the optical flow field composed of the estimated optical flow values ​​of each pixel between two adjacent scene images, and the rigid optical flow field composed of the rigid optical flow values ​​of each pixel between two adjacent scene images. Since the rigid optical flow value is caused by the movement of the UAV and is related to the position coordinates of the UAV, the position coordinates of the UAV can be obtained based on the relationship between the optical flow field and the rigid optical flow value, thereby achieving UAV positioning and ensuring that the UAV can work normally under signal interference or obstruction. Furthermore, by achieving autonomous UAV positioning, the interaction process between the UAV and ground control facilities is eliminated, resulting in higher real-time positioning performance. Attached Figure Description

[0050] Figure 1This is an application environment diagram of the UAV positioning method in one embodiment;

[0051] Figure 2 This is a flowchart illustrating a drone positioning method in one embodiment;

[0052] Figure 3 This is a schematic diagram illustrating the process of determining the optical flow field of a scene between two adjacent frames based on two adjacent scene images in one embodiment.

[0053] Figure 4 This is a schematic diagram of the process for determining the estimated optical flow in one embodiment;

[0054] Figure 5 This is a flowchart illustrating the training method of a feature extraction model in one embodiment;

[0055] Figure 6 This is a schematic diagram illustrating the process of generating the training dataset in one embodiment;

[0056] Figure 7 This is a schematic diagram of part of the training process of the feature extraction model in one embodiment;

[0057] Figure 8 This is a structural block diagram of a drone positioning device in one embodiment;

[0058] Figure 9 This is a diagram of the internal structure of a drone in one embodiment. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0060] Currently, UAV positioning commonly employs technologies such as satellite inertial navigation positioning, motion capture positioning, UWB (Ultra Wideband) positioning, and SLAM (Simultaneous Localization and Mapping) positioning. Satellite inertial navigation positioning is generally suitable for open outdoor environments; its accuracy decreases in the presence of signal interference or obstructions. Motion capture positioning and UWB positioning require the pre-deployment of various devices in the UAV's flight area, thus making them unusable in unknown areas. In SLAM positioning, lidar-based positioning loses data during high-speed movement and suffers from slow processing speed and low output frequency. Multi-sensor fusion-based positioning requires additional sensors, inevitably increasing the UAV's weight and reducing its range. Furthermore, processing data from multiple sensors consumes more power. Therefore, this application provides a UAV positioning method capable of accurate positioning under conditions of signal interference or obstruction, both day and night.

[0061] The UAV positioning method provided in this application embodiment can be applied to UAVs, such as... Figure 1 As shown, the drone includes an image acquisition device 102 and a processor 104 connected to each other. The image acquisition device 102 acquires scene images of the scene where the drone is located in real time and transmits them to the processor 104. The processor 104 determines the optical flow field of the scene between two adjacent scene images, as well as the rigid optical flow field of the scene caused by the movement of the drone, based on two adjacent scene images. The position coordinates of the drone are determined by the optical flow field and the rigid optical flow field. The image acquisition device 102 can be an RGB-D (Red Green Blue-Depth) camera, or other types of depth cameras that can acquire the depth field of each frame of the acquired image. For ease of explanation, the image acquisition device 102 will be described as an RGB-D camera in the following description.

[0062] In another embodiment, the processor 104 communicates with the server via a network. The image acquisition device 102 acquires scene images of the scene where the drone is located in real time and transmits them to the processor 104. The processor 104 transmits the images to the server via the network. Based on two adjacent scene images, the server determines the optical flow field of the scene between the two adjacent scene images, as well as the rigid optical flow field of the scene caused by the drone's movement. The position coordinates of the drone are determined by the optical flow field and the rigid optical flow field. The server can be a standalone server located on the ground or a server cluster consisting of multiple servers. The server can also be a cloud server.

[0063] In one embodiment, such as Figure 2 As shown, a method for locating unmanned aerial vehicles (UAVs) is provided, which can be applied to... Figure 1Taking the processor in the example, the following steps are included:

[0064] Step 200: Real-time acquisition of scene images of the scene where the drone is located.

[0065] The RGB-D camera can capture scene images of the drone's location and the depth field of each frame of the scene image. The scene can be an indoor scene or an outdoor scene. The scene images can be scene images with sufficient light, such as daytime scene images, or scene images with insufficient light, such as nighttime scene images.

[0066] Step 300: Based on two adjacent scene images, determine the optical flow field of the scene between the two adjacent scene images, as well as the rigid optical flow field of the scene caused by the movement of the drone.

[0067] The optical flow field includes the estimated optical flow of each pixel in the scene, the rigid optical flow field includes the rigid flow (i.e., rigid optical flow) of each pixel in the scene, and the depth field includes the depth of each pixel in the scene. The optical flow field can be obtained by using a trained optical flow generation network to determine the estimated optical flow of each pixel between the previous frame and the current frame of the scene image.

[0068] When determining rigid flow, the camera pose transformation matrix can be used as an unknown variable. The rigid optical flow expression can be obtained by using the camera pose transformation matrix and the depth field of each frame of scene image. The rigid optical flow expression of each pixel constitutes the rigid optical flow field.

[0069] Step 400: Determine the position coordinates of the UAV using optical flow field and rigid optical flow field.

[0070] Since the rigid optical flow field and the optical flow field between two adjacent frames should theoretically be equal, the camera's pose transformation matrix can be solved by modeling the optical flow field and the rigid optical flow field. From this matrix, the camera's pose can be obtained, and thus its position coordinates. Because the RGB-D camera is mounted on the drone's body and moves with the drone without relative motion, the camera's position is the same as the drone's position.

[0071] The aforementioned UAV positioning method acquires scene images of the scene where the UAV is located in real time; based on two adjacent scene images, it determines the optical flow field between the two adjacent frames in the scene, as well as the rigid optical flow field of the scene caused by the UAV's movement; and determines the UAV's position coordinates through the optical flow field and the rigid optical flow field. This yields the optical flow field composed of the estimated optical flow values ​​of each pixel between two adjacent scene images, and the rigid optical flow field composed of the rigid optical flow values ​​of each pixel between two adjacent scene images. Since the rigid optical flow value is caused by the UAV's movement and is related to its position coordinates, the UAV's position coordinates can be obtained based on the relationship between the optical flow field and the rigid optical flow value, thus achieving UAV positioning and ensuring the UAV can operate normally under signal interference or obstruction. Furthermore, by achieving autonomous UAV positioning, the interaction process between the UAV and ground control facilities is eliminated, resulting in higher real-time positioning and a higher level of UAV intelligence. Moreover, the positioning method in this embodiment can not only meet the positioning needs during the day when lighting conditions are good, but also meet the positioning needs at night when lighting conditions are poor, making it more widely applicable.

[0072] In one embodiment, such as Figure 3 As shown, in step 300, the optical flow field of the scene between two adjacent frames is determined based on two adjacent scene images, including: step 310, extracting the common features of the scene in the two adjacent scene images; step 320, obtaining the optical flow field of the scene based on the common features of the two adjacent scene images.

[0073] For daytime and nighttime images of the same scene, although the lighting is completely different, the optical flow information should be consistent. Therefore, the basic information of daytime and nighttime images of a scene should be similar. Based on this, an image can be divided into two parts: common features and private features. Common features are also invariant features, that is, features that remain unchanged when the scene image changes.

[0074] In this embodiment, in order to ensure the consistency of optical flow in different lighting environments, the interference of private features is removed, and the optical flow field is determined by common features. The optical flow field obtained is more reliable, and at the same time, the UAV positioning method can meet the positioning requirements in various lighting environments.

[0075] In one embodiment, step 320 includes: performing feature enhancement on the common features of two adjacent scene images to obtain optimized features; performing feature matching on the optimized features of two adjacent scene images to obtain matching results; and determining the estimated optical flow of each pixel in the scene based on the matching results to obtain the optical flow field.

[0076] In this embodiment, an optical flow field is generated by a global matching optical flow estimation method. By performing feature enhancement, higher quality optimized features can be obtained. Therefore, matching is performed based on high-quality distinguishable features to obtain a more accurate estimated optical flow.

[0077] Specifically, refer to Figure 4 For two adjacent scene images I1 and I2, their common features F1 and F2 can be extracted relatively independently from a weight-sharing convolutional neural network. The Transformer model, based on a multi-head attention mechanism, can model the relationship between the two sets using an attention mechanism, thus improving the interdependence of F1 and F2. Since F1 and F2 are just two sets of features without spatial location, fixed two-dimensional sine and cosine positional encodings can be added to the features first. Adding positional information allows the matching process to consider the spatial distance between features while considering feature similarity, which helps improve performance. After adding positional information, six stacked self-attention, cross-attention, and feedforward networks are executed to improve the quality of the initial features. Specifically, for self-attention, the query, key, and value are the same feature; for cross-attention, the query is different, but the key and value are the same, to introduce their interdependence, thereby improving the quality of the common features.

[0078] It should also be noted that a problem with the standard Transformer architecture is the quadratic computational complexity caused by pairwise attention operations. To improve efficiency, the moving local window attention method of the Swin Transformer is specifically adopted. However, unlike the Swin Transformer which uses a fixed window size, this embodiment first determines the number of local windows, then segments the common features into a corresponding number of feature windows, and finally sets the sliding window size to be the same as the feature window size. For example, if K*K local windows are used, then the feature with an input size of H*W is segmented, and the size of each small feature window is... Self-attention and cross-attention are performed independently within each local window. In this embodiment, the window partitions are moved for every two consecutive local windows. By introducing cross-window connections, the features can be divided into 2*2 windows.

[0079] In optical flow estimation algorithms, feature matching can be performed by comparing the similarity of optimized features between two adjacent scene images to determine the estimated optical flow of successfully matched pixels between the two frames. The specific matching process is not limited. However, this matching method is ineffective for pixels that are occluded or exceed boundaries in the scene, resulting in unsuccessful matches. To address this, when a match fails, the high structural similarity between the optical flow field and the image itself can be leveraged. By calculating the self-similarity of features, the high-quality optical flow prediction from matched pixels can be propagated to unmatched pixels, thus obtaining the estimated optical flow for occluded and / or boundary-exceeding pixels. This operation can be implemented using a self-attention layer, which will not be elaborated further.

[0080] In another embodiment, step 300, determining the optical flow field of the scene between two adjacent frames based on two adjacent scene images, may include: determining the optical flow field of the scene between two adjacent frames based on the two adjacent scene images and the optical flow estimation model.

[0081] The optical flow estimation model can include an optical flow estimation sub-model and a feature extraction model. The feature extraction model extracts common features from each frame of scene images, while the optical flow estimation sub-model obtains the estimated optical flow of each pixel in the scene based on the common features in two adjacent frames. The construction method of the optical flow estimation sub-model can refer to the above embodiment. Furthermore, the optical flow estimation model can also be performed in a supervised manner. For example, the EPE (End-Point-Error) loss function can be used to constrain the above process, defined as the Euclidean distance between the estimated optical flow of each predicted pixel and the true value. The loss function value is defined as L... EPE :

[0082] L EPE =||V i -V gt ||2

[0083] Among them, V i V represents the estimated optical flow. gt Represents the true optical flow (given value), expressed in terms of the loss function value L. EPE With minimization as the objective, adjust the parameters of the optical flow estimation model until L... EPE If the requirements are met, a trained optical flow estimation model is obtained, and the estimated optical flow obtained through this optical flow estimation model is more reliable.

[0084] In one embodiment, such as Figure 5 As shown, step 310 includes: extracting common features of the scene in each frame of the scene image using a feature extraction model. The training method for the feature extraction model includes steps 311-314.

[0085] Step 311: Obtain the training dataset; the training dataset includes consecutive daytime images and consecutive nighttime images of the same scene.

[0086] Specifically, obtaining the training dataset may include: training a day-to-night image conversion network using unpaired day and night images of the same scene; and using the trained day-to-night image conversion network to generate consecutive night images corresponding to consecutive day images, in order to obtain the training dataset.

[0087] Existing localization algorithms do not consider low-light environments such as nighttime, and lack nighttime training datasets. It is understandable that paired daytime and nighttime images of the same scene are practically unavailable. Therefore, in this embodiment, a trained day-night image conversion network is obtained to generate continuous nighttime images of the same scene, which are then used to train a feature extraction model to improve nighttime localization capabilities.

[0088] Specifically, refer to Figure 6 First, using the Shift dataset to provide unpaired day and night images, a CycleGAN network (i.e., day-night image conversion network) is trained to perform day-night image conversion. The trained CycleGAN conversion network generates corresponding continuous night images for continuous day images provided by the Shift dataset, thus obtaining the training dataset.

[0089] Step 312: Using the feature extraction model to be trained, extract features from consecutive daytime images and consecutive nighttime images of the same scene to obtain common daytime features, private daytime features, common nighttime features, and private nighttime features.

[0090] Scene images consist of two parts: common features and private features, such as... Figure 7 As shown, given two consecutive daytime images and two consecutive nighttime images, a shared-weighted common feature extractor can be used to extract common features for both daytime and nighttime use, which are then used to estimate optical flow generation. Private features for daytime and nighttime are extracted using separate private feature extractors to assist the common feature extractor. The private and common features of the images are complementary and entirely different. Therefore, direct orthogonal loss and Gram orthogonal loss are also used to ensure that the private and invariant features are completely different.

[0091] For direct orthogonality loss, since the private and common features are relatively large, a convolutional layer with a 1×1 kernel is first used to reduce complexity. Then, the simplified features are resized into one-dimensional vectors. Finally, the inner product of the private feature vector and the invariant feature vector is calculated, which is the orthogonality loss, defined as L. f :

[0092]

[0093] Where V is the operation that converts multidimensional features into one-dimensional features, v i and v p v represents private features and shared features respectively. i d Indicates common features during the day, v i p Indicates common characteristics at night, v p d Indicates daytime private features, v p n This indicates a private feature at night.

[0094] For Gram orthogonal loss, the Gram matrix is ​​widely used to identify the style of features. However, private and public features have different styles. Therefore, the Gram matrix η of private and invariant features is calculated first. p and η i Then, it is resized into a one-dimensional vector, and finally, the orthogonality loss between these vectors is calculated. The loss is defined as L. g :

[0095]

[0096] Here, V is the operation that transforms multidimensional features into features of type 1. This represents the Gram matrix representing common features during the day. Represents the Gram matrix of private features during the day. This represents the Gram matrix representing common features at night. This represents the Gram matrix representing the private features at night.

[0097] The final orthogonal loss is defined as L ortho :L ortho =L f +L g .

[0098] Step 313: Reconstruct the daytime image and the nighttime image based on the common features of daytime, the private features of daytime, the common features of nighttime, and the private features of nighttime to obtain the reconstructed daytime image and the reconstructed nighttime image.

[0099] Daytime reconstructed images are obtained by using common daytime features and private daytime features; nighttime reconstructed images are obtained by using common nighttime features and private nighttime features.

[0100] Step 314: Calculate the loss function value based on the training dataset, daytime reconstructed images, and nighttime reconstructed images. Update the parameters of the feature extraction model to be trained with the goal of minimizing the loss function value until the training termination condition is met, and obtain the trained feature extraction model.

[0101] After image reconstruction, the reconstruction loss function needs to be used to calculate the loss function value, which can be defined as L. recons :

[0102]

[0103] Among them, I d Represents a daytime image, I n Indicates nighttime image, I d Indicates the reconstructed image during the day, I n Represents a nighttime reconstructed image; x∈[1,N], N is I d and I n The number of pixels.

[0104] In the aforementioned network, domain transfer existing between daytime and low-light conditions is fully considered. A domain transfer module is designed to separate common and private features in daytime and low-light images, while using common features for optical flow estimation. In the feature matching part, the fixed window size in the Swin Transformer is improved by using a fixed number of windows to balance performance and quality. For occluded and out-of-bounds pixels, the self-similarity of features is calculated to complete the propagation of optical flow and generate the final optical flow.

[0105] In another embodiment, the feature extraction model may include a shared feature extraction sub-model, a private feature extraction sub-model, and an image reconstruction sub-model. Based on the training dataset, the shared feature extraction sub-model, and the private feature extraction sub-model, shared features during the day, private features during the day, shared features at night, and private features at night for the same scene are obtained. Based on the shared features during the day, private features during the day, shared features at night, private features at night, and the image reconstruction sub-model, daytime image reconstruction and nighttime image reconstruction are performed to obtain daytime reconstructed images and nighttime reconstructed images. Finally, based on the training dataset, daytime reconstructed images, and nighttime reconstructed images, the loss function value is calculated, and the parameters of the feature extraction model to be trained are updated with the goal of minimizing the loss function value, until the training termination condition is met, and the trained feature extraction model is obtained.

[0106] In one embodiment, real-time acquisition of scene images of the scene where the drone is located includes real-time acquisition of scene images of the scene where the drone is located and obtaining the depth field of each frame of scene image. In step 300, determining the rigid optical flow field of the scene caused by the movement of the drone based on two adjacent frames of scene images includes: determining the rigid optical flow expression of each pixel in the scene image according to the depth field, preset stiffness field and camera attitude of two adjacent frames of scene images; obtaining the rigid optical flow field of the scene through the rigid optical flow expression of each pixel in the scene image.

[0107] To obtain the expression for rigid optical flow, an additional variable—stiffness W (W = 0 or 1)—needs to be introduced. Optical flow can then be viewed as a combination of rigid flow related to camera movement and scene structure, and unconstrained flow describing the motion of general objects. Rigid flow, where W = 1, is generated entirely by camera and scene movement; unconstrained flow, where W = 0, includes not only general motion but also the flow generated by the deformation of objects themselves. Therefore, to accurately estimate the drone's position, this embodiment assumes all objects in the scene are rigid and uses rigid flow (W = 1) for subsequent calculations, greatly reducing errors caused by environmental factors.

[0108] When acquiring the scene image, the depth field of each frame is also acquired, allowing the depth of each pixel to be determined. Then, a rigid flow r is obtained by applying a rigid transformation T to the scene with defined depth and stiffness, assuming π. t (θ j Given the camera pose at time t, the result will be compared with θ. j The pixel coordinates of the associated 3D point coordinates projected onto the camera's imaging plane can be expressed by the following formula:

[0109]

[0110] Then rigid flow can be expressed as: r t (j)=π t (θ j )-π t-1 (θ j )

[0111] Where, r t (j) represents the rigid flow generated by pixel j at time t, K represents the camera intrinsic parameter matrix, Ti represents the rigid transformation of the camera at time i, and θ j Represents the depth of pixel j, [x j ,y j ,1] T This represents the coordinates of pixel j, where 1 indicates that the z-coordinate of the imaging plane is restricted to 1.

[0112] In one embodiment, determining the position coordinates of the UAV using an optical flow field and a rigid optical flow field includes: modeling the relationship between the optical flow field and the rigid optical flow field to determine the pose transformation matrix of the camera in the scene; determining the camera pose associated with each pixel in the scene based on the pose transformation matrix; and determining the position coordinates of the UAV based on the camera pose.

[0113] The rigid flow between each two frames is generated by the actual camera movement and is an expression with camera attitude. The estimated optical flow between each two frames is obtained through the above model. Each pixel has a certain relationship between the rigid flow and the estimated flow between each two frames. Therefore, after modeling, the camera attitude can be determined by each pixel, thereby determining the position of the UAV.

[0114] In this embodiment, UAV positioning is achieved jointly based on the generated optical flow and the RGB-D camera mounted on the UAV. After acquiring the scene image, the relationship between the optical flow field and the rigid optical flow field between two frames can be modeled based on the above embodiment. Taking one pixel as an example, its rigid flow r t Modeling is performed between (j) and the estimated flow X:

[0115]

[0116] Here, the probability density function ρ(·||·) represents the probability of obtaining a rigid flow under the estimated flow conditions, which can be calculated using a Gaussian distribution. In the above equation, only T is an unknown variable, and it can ultimately be modeled as:

[0117]

[0118] Therefore, the camera's pose transformation matrix (i.e., rigid transformation T) can be calculated using the above formula. The camera's attitude can then be determined based on the pose transformation matrix, and the camera's position coordinates, i.e., the UAV's position coordinates, can be determined based on the camera's attitude, thus achieving UAV localization. The method for determining the camera attitude and position coordinates based on the pose transformation matrix is ​​not limited.

[0119] The aforementioned UAV localization method generates paired daytime and low-light datasets based on the Shift dataset and the CycleGAN algorithm for training the optical flow estimation network. In the optical flow estimation network, since the optical flow generated in daytime and low-light environments should be identical under the same scene, and considering the domain transfer present in daytime and low-light environments, a domain transfer module is designed to separate invariant and private features in daytime and low-light images, using the invariant features for optical flow estimation. In the feature matching part, the fixed window size approach in the Swin Transformer is improved by using a fixed number of windows to balance performance and quality. For occluded and boundary-extended pixels, the self-similarity of features is calculated to propagate the optical flow and generate the final optical flow. Based on the generated optical flow, by modeling the relationship between the UAV system's position and the estimated optical flow, the current position of the UAV system can be estimated in real time, achieving localization. This localization method can be applied to indoor and outdoor environments, is not limited by lighting conditions, and has a wide range of applications.

[0120] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0121] Based on the same inventive concept, this application also provides a drone positioning device for implementing the drone positioning method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more drone positioning device embodiments provided below can be found in the limitations of the drone positioning method described above, and will not be repeated here.

[0122] In one embodiment, such as Figure 8 As shown, a UAV positioning device is provided, comprising: a data acquisition module 10, an optical flow field determination module 20, and a positioning module 30, wherein:

[0123] The data acquisition module 10 is used to acquire scene images of the scene where the drone is located in real time.

[0124] The optical flow field determination module 20 is used to determine the optical flow field between two adjacent frames in the scene, as well as the rigid optical flow field of the scene caused by the movement of the drone, based on two adjacent frames of scene images.

[0125] The positioning module 30 is used to determine the position coordinates of the UAV through a rigid optical flow field and an optical flow field.

[0126] In one embodiment, the optical flow field determination module 20 is further configured to extract common features of the scene in two adjacent scene images; and obtain the optical flow field of the scene based on the common features of the two adjacent scene images.

[0127] In one embodiment, the optical flow field determination module 20 is further configured to enhance the common features of two adjacent scene images to obtain optimized features; perform feature matching on the optimized features of two adjacent scene images to obtain matching results; and determine the estimated optical flow of each pixel in the scene based on the matching results to obtain the optical flow field.

[0128] In one embodiment, the optical flow field determination module 20 is further configured to extract common features of the scene in each frame of scene image through a feature extraction model.

[0129] In one embodiment, the optical flow field determination module 20 is further configured to acquire a training dataset; the training dataset includes consecutive daytime images and consecutive nighttime images of the same scene; through the feature extraction model to be trained, features are extracted from the consecutive daytime images and consecutive nighttime images of the same scene to obtain common daytime features, private daytime features, common nighttime features, and private nighttime features; based on the common daytime features, private daytime features, common nighttime features, and private nighttime features, daytime image reconstruction and nighttime image reconstruction are performed to obtain reconstructed daytime images and reconstructed nighttime images; based on the training dataset, reconstructed daytime images, and reconstructed nighttime images, the loss function value is calculated, and the parameters of the feature extraction model to be trained are updated with the goal of minimizing the loss function value, until the training termination condition is met, and the trained feature extraction model is obtained.

[0130] In one embodiment, the optical flow field determination module 20 is further configured to train a day-night image conversion network using unpaired day and night images; and to generate continuous night images corresponding to continuous day images using the trained day-night image conversion network to obtain a training dataset.

[0131] In one embodiment, the data acquisition module 10 is further configured to acquire scene images of the scene where the drone is located in real time and obtain the depth field of each frame of scene image; the optical flow field determination module 20 is configured to determine the rigid optical flow expression of each pixel in the scene image based on the depth field, preset stiffness field and camera attitude of two adjacent frames of scene images; and obtain the rigid optical flow field of the scene through the rigid optical flow expression of each pixel in the scene image.

[0132] In one embodiment, the positioning module 30 is further configured to model the relationship between the optical flow field and the rigid optical flow field to determine the pose transformation matrix of the camera in the scene; determine the camera pose of the scene based on the pose transformation matrix; and determine the position coordinates of the UAV based on the camera pose.

[0133] Each module in the aforementioned UAV positioning device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the UAV's processor in hardware form or independent of it, or stored in the UAV's memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0134] In one embodiment, a drone is provided, the internal structure of which can be shown as follows: Figure 9 As shown, the UAV includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The UAV's processor provides computational and control capabilities. The UAV's memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The UAV's database stores data that the processor needs to process, such as scene images of each frame and their depth fields, estimated optical flow of each pixel in each scene image, rigid flow, etc. The UAV's I / O interfaces are used for exchanging information between the processor and external devices. The UAV's communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a UAV positioning method.

[0135] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the drone to which the present application is applied. A specific drone may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0136] In one embodiment, a drone is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0137] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0138] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0139] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0140] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0141] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for locating unmanned aerial vehicles (UAVs), characterized in that, The method includes: Real-time acquisition of scene images of the drone's location; Based on two adjacent frames of the scene image, determine the optical flow field of the scene between the two adjacent frames of the scene image, as well as the rigid optical flow field of the scene caused by the movement of the UAV; The position coordinates of the UAV are determined by the optical flow field and the rigid optical flow field; The step of determining the optical flow field of the scene between two adjacent frames based on the scene images includes: Extract the common features of the scene in two adjacent frames of the scene image; The optical flow field of the scene is obtained based on the common features of two adjacent frames of the scene image; The step of extracting common features of the scene in two adjacent frames of the scene image includes: The common features of the scene in each frame of the scene image are extracted using a feature extraction model; wherein, the training method of the feature extraction model includes: Obtain a training dataset; the training dataset includes consecutive daytime images and consecutive nighttime images of the same scene; By using the feature extraction model to be trained, features are extracted from consecutive daytime images and consecutive nighttime images of the same scene to obtain common daytime features, private daytime features, common nighttime features, and private nighttime features. Based on the common daytime features, the private daytime features, the common nighttime features, and the private nighttime features, daytime image reconstruction and nighttime image reconstruction are performed to obtain daytime reconstructed images and nighttime reconstructed images; Based on the training dataset, the daytime reconstructed image, and the nighttime reconstructed image, a loss function value is calculated, and the parameters of the feature extraction model to be trained are updated with the goal of minimizing the loss function value, until the training termination condition is met, thus obtaining the feature extraction model that has been trained.

2. The method according to claim 1, characterized in that, The step of obtaining the optical flow field of the scene based on the common features of two adjacent frames of the scene image includes: Feature enhancement is performed on the common features of two adjacent frames of the scene images to obtain optimized features; Feature matching is performed on the optimized features of two adjacent frames of the scene image to obtain the matching result; Based on the matching results, the estimated optical flow of each pixel in the scene is determined, and the optical flow field is obtained.

3. The method according to claim 1, characterized in that, The acquisition of the training dataset includes: A day-to-night image transformation network was trained using unpaired day and night images of the same scene; The trained day-night image conversion network is used to generate consecutive night images corresponding to consecutive day images to obtain a training dataset.

4. The method according to any one of claims 1 to 3, characterized in that: The real-time acquisition of scene images of the scene where the drone is located includes: real-time acquisition of scene images of the scene where the drone is located, and obtaining the depth field of each frame of the scene image. The determination of the rigid optical flow field of the scene caused by the movement of the drone based on two adjacent frames of the scene image includes: Based on the depth field, preset stiffness field, and camera pose of two adjacent frames of the scene image, determine the rigid optical flow expression of each pixel in the scene image; The rigid optical flow field of the scene is obtained by using the rigid optical flow expression of each pixel in the scene image.

5. The method according to claim 4, characterized in that, Determining the position coordinates of the UAV through the optical flow field and the rigid optical flow field includes: Modeling is performed based on the relationship between the optical flow field and the rigid optical flow field to determine the pose transformation matrix of the camera in the scene; The camera pose of the scene is determined based on the pose transformation matrix; The position coordinates of the UAV are determined based on the camera's orientation.

6. A drone positioning device, characterized in that, The device includes: The data acquisition module is used to acquire scene images of the drone's location in real time. The optical flow field determination module is used to determine the optical flow field between two adjacent frames in the scene, and the rigid optical flow field of the scene caused by the movement of the UAV, based on two adjacent frames of the scene image. The positioning module is used to determine the position coordinates of the UAV through the rigid optical flow field and the optical flow field; The step of determining the optical flow field of the scene between two adjacent frames based on the scene images includes: Extract the common features of the scene in two adjacent frames of the scene image; The optical flow field of the scene is obtained based on the common features of two adjacent frames of the scene image; The step of extracting common features of the scene in two adjacent frames of the scene image includes: The common features of the scene in each frame of the scene image are extracted using a feature extraction model; wherein, the training method of the feature extraction model includes: Obtain a training dataset; the training dataset includes consecutive daytime images and consecutive nighttime images of the same scene; By using the feature extraction model to be trained, features are extracted from consecutive daytime images and consecutive nighttime images of the same scene to obtain common daytime features, private daytime features, common nighttime features, and private nighttime features. Based on the common daytime features, the private daytime features, the common nighttime features, and the private nighttime features, daytime image reconstruction and nighttime image reconstruction are performed to obtain daytime reconstructed images and nighttime reconstructed images; Based on the training dataset, the daytime reconstructed image, and the nighttime reconstructed image, a loss function value is calculated, and the parameters of the feature extraction model to be trained are updated with the goal of minimizing the loss function value, until the training termination condition is met, thus obtaining the feature extraction model that has been trained.

7. A drone, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images