A multi-view positioning method, device, equipment and storage medium for unmanned aerial vehicle

Abstract

The present invention discloses a multi-perspective positioning method, apparatus, equipment, and storage medium for unmanned aerial vehicle (UAV). The method comprises receiving multiple first images captured by a multi-perspective image acquisition device; the multiple first images include image information of the UAV's surrounding environment at the first moment from different perspectives; and extracting the full-range features of each first image to obtain multiple sets of feature point data. The method uses multiple monocular cameras to perform real-time odometer calculations for the UAV's position and reconstructs a three-dimensional map model of the images captured by the UAV. This method avoids the problem of being unable to calculate the true three-dimensional environmental scale when using a single monocular camera, the problem of limited image distance when using a depth camera, and the high time complexity of the matching algorithm when directly performing odometer calculations with ordinary binocular cameras.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle (UAV) positioning and navigation technology, and in particular to a multi-perspective positioning method, device, equipment and storage medium for UAV that improves the computational efficiency of a traditional SLAM algorithm and can quickly calculate the real-time position and positioning information of the UAV. Background Art

[0002] The positioning and navigation of drones often rely on a combined navigation system consisting of GPS and inertial navigation systems. However, this positioning and navigation method will not be effective in indoor environments or other satellite-denied environments. In addition to this method, the use of lidar and visual cameras as simultaneous positioning and mapping (SLAM) perception sensors has also become a research hotspot for drone positioning and navigation in recent years. Lidar can obtain accurate environmental point cloud information to complete positioning and mapping functions, and visual cameras can also complete positioning and mapping functions through multi-view geometry methods based on image feature points. Currently, drone positioning and mapping usually adopts lidar-based and visual camera-based methods. The main problems with these two methods are:

[0003] When using drones equipped with lidar for positioning and mapping, multi-line lidar can improve accuracy but incurs heavy loads and high power consumption. Single-line lidar offers lower loads and power consumption, but severely limits accuracy. When using drones equipped with visual cameras for positioning and mapping, a single camera has a limited field of view and cannot provide the multi-perspective information necessary for map reconstruction. While multiple cameras can provide multi-perspective images, the simultaneous use of these images in visual odometry calculations significantly increases computational resource consumption.

[0004] Therefore, when it comes to drone positioning and navigation operations, due to load and power consumption considerations, drones often rely on cameras for visual positioning and mapping. To ensure they can capture wide-field scene images and multi-perspective images for visual 3D reconstruction, drones typically carry multiple cameras. This significantly improves 3D scene reconstruction in offline mode, as computing resources are not a concern. However, for real-time SLAM positioning and mapping, multi-perspective images pose a significant challenge to the drone's onboard computing platform.

[0005] The ORB-SLAM2 algorithm, which uses features from accelerated segment test (FAST) and rotation robust binary independent descriptors (BRIEF), is a mainstream visual SLAM technology solution. The algorithm supports monocular, binocular and RGBD depth camera modes. The algorithm framework is as follows: Figure 6 As shown in the figure, it mainly includes tracking, local mapping, loop closure detection and mapping modules: the tracking module extracts ORB features from the image, estimates the positioning pose based on the previous frame image, and then tracks the reconstructed local map to optimize the pose; the local mapping module mainly completes the local map construction operation to complete the key frame insertion, map point screening and positioning pose optimization using BA local bundle adjustment; the loop closure detection module mainly includes loop closure detection and loop closure correction operations to complete loop fusion and mapping optimization; the mapping module mainly completes the 3D map construction operation based on the map points obtained by the previous modules.

[0006] The ORB-SLAM2 algorithm completes the overall algorithm process calculation based on feature points, but it is prone to frame loss when the camera is only rotating or moving at high speed. The feature-based method leads to sparse point clouds in the reconstructed map, making it difficult to restore the structural information of the real physical environment. It is also necessary to ensure low-speed camera movement during initialization. At the same time, good positioning and mapping effects can only be achieved in three-dimensional physical environments with rich alignment features and geometric textures. Summary of the Invention

[0007] In view of the above problems, the present invention provides a multi-perspective positioning method, device, equipment and storage medium for unmanned aerial vehicle (UAV) for overcoming the above problems or at least partially solving the above problems.

[0008] The present invention provides the following solutions:

[0009] A multi-view positioning method for an unmanned aerial vehicle, comprising:

[0010] Receiving a plurality of first images captured by a multi-view image acquisition device; the plurality of first images include image information of the drone's surrounding environment at a first moment under different view angles;

[0011] Extracting full-range features of each of the first images to obtain multiple sets of feature point data;

[0012] Determine a three-dimensional map reference coordinate system using a binocular structure formed by two monocular cameras included in the multi-view image acquisition device in combination with multiple sets of feature point data;

[0013] Determining overlapping pixel blocks of a plurality of sets of feature point data, and dividing the overlapping pixel blocks to obtain feature-rich blocks in the overlapping area;

[0014] receiving a plurality of second images captured by the multi-view image acquisition device; the plurality of second images including image information of the drone's surrounding environment at a second moment under different view angles; the second moment being a moment subsequent to the first moment;

[0015] Acquire feature points of the feature-rich block, and search for feature points matching the feature points of the feature-rich block in a plurality of second images to obtain matching feature point pairs;

[0016] Constructing a geometric reprojection error using the matching feature point pairs and performing beam adjustment optimization to calculate a relative pose transformation matrix between the UAV at the first moment and the UAV at the second moment in the three-dimensional map reference coordinate system;

[0017] Performing feature point matching on the non-overlapping pixel area at the first moment, and constructing a reprojection residual based on the relative posture transformation matrix as a projection transformation matrix;

[0018] The reprojection residual is used as the initial value of the beam adjustment optimization to perform least squares iterative optimization to obtain the target transformed pose of the UAV in the three-dimensional map reference coordinate system.

[0019] Preferably, the multi-view image acquisition device includes a plurality of monocular cameras that are installed in a surrounding manner and have equal angular spacing and equal distance spacing.

[0020] Preferably, the external parameters between each two cameras in the multiple monocular cameras are calibrated before the visual odometry positioning calculation, and the overlapping range of the images taken by the multiple monocular cameras is obtained based on the external parameter transformation.

[0021] Preferably, the monocular cameras include three, and the overlapping pixel area of ​​the images taken by two of the monocular cameras is not less than one-fifth of the total pixel area of ​​the image taken by a single monocular camera.

[0022] Preferably, the method of determining a three-dimensional map reference coordinate system by utilizing a binocular structure formed by two monocular cameras included in the multi-view image acquisition device in combination with multiple sets of feature point data comprises:

[0023] Performing feature matching on the plurality of sets of feature point data to obtain matching point pairs between any two images;

[0024] According to a binocular structure formed by a plurality of monocular cameras in pairs, a binocular stereo vision technology is used to calculate the depth of the matching feature point pairs to obtain three-dimensional map points;

[0025] An initial three-dimensional map is created using the three-dimensional map points, and the current UAV posture is assigned to a unit matrix, which is set as the origin of the reference coordinate system in the three-dimensional map to determine the three-dimensional map reference coordinate system.

[0026] Preferably, the step of obtaining the feature points of the feature-rich block includes:

[0027] The plurality of sets of feature point data are marked, and the overlapping pixel blocks are divided into quadtrees to obtain quadtree node blocks, so as to obtain feature points of the overlapping feature-rich blocks.

[0028] Preferably, the method of constructing a geometric reprojection error using the matching feature point pairs and performing beam adjustment optimization to calculate the relative pose transformation matrix between the UAV at the first moment and the UAV at the second moment in the three-dimensional map reference coordinate system comprises:

[0029] The matching feature point pairs are used to construct geometric reprojection errors and perform beam adjustment optimization to solve the rotation transformation matrix and translation transformation vector of the drone at adjacent moments to obtain the relative posture transformation matrix.

[0030] A multi-view positioning device for an unmanned aerial vehicle, comprising:

[0031] A first image receiving unit is configured to receive a plurality of first images captured by a multi-view image acquisition device; the plurality of first images include image information of the drone's surrounding environment at a first moment under different view angles;

[0032] a feature point data extraction unit, configured to extract full-range features of each of the first images to obtain multiple sets of feature point data;

[0033] A coordinate system determining unit, configured to determine a three-dimensional map reference coordinate system by utilizing a binocular structure formed by two monocular cameras included in the multi-view image acquisition device in combination with multiple sets of feature point data;

[0034] a feature-rich block determining unit, configured to determine overlapping pixel blocks of a plurality of sets of feature point data, and to divide the overlapping pixel blocks to obtain feature-rich blocks in the overlapping area;

[0035] A second image receiving unit is configured to receive a plurality of second images captured by the multi-view image acquisition device; the plurality of second images include image information of the drone's surrounding environment at a second moment under different view angles; the second moment is a moment subsequent to the first moment;

[0036] a matching feature point pair acquisition unit, configured to acquire feature points of the feature-rich block, and search for feature points matching the feature points of the feature-rich block in a plurality of second images to obtain matching feature point pairs;

[0037] a relative pose transformation matrix acquisition unit, configured to construct a geometric reprojection error using the matching feature point pairs and perform beam adjustment optimization to calculate the relative pose transformation matrix between the UAV at the first moment and the UAV at the second moment in the three-dimensional map reference coordinate system;

[0038] a reprojection residual construction unit, configured to perform feature point matching on the non-overlapping pixel region at the first moment, and construct a reprojection residual based on the relative posture transformation matrix as a projection transformation matrix;

[0039] The target transformation pose acquisition unit is used to perform least squares iterative optimization using the reprojection residual as the initial value of the beam adjustment optimization to obtain the target transformation pose of the UAV in the three-dimensional map reference coordinate system.

[0040] A multi-view positioning device for an unmanned aerial vehicle, comprising a processor and a memory:

[0041] The memory is used to store program code and transmit the program code to the processor;

[0042] The processor is used to execute the above-mentioned drone multi-view positioning method according to the instructions in the program code.

[0043] A computer-readable storage medium is used to store program code, and the program code is used to execute the above-mentioned multi-view positioning method for unmanned aerial vehicle.

[0044] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0045] The embodiments of the present application provide a multi-perspective positioning method, apparatus, device, and storage medium for drones. These methods use multiple monocular cameras to perform real-time odometer calculations of the drone's position and orientation, and reconstruct a three-dimensional map model of the drone's images. This avoids the inability to calculate the true three-dimensional environmental scale when using a single monocular camera, the limited image distance when using a depth camera, and the high time complexity of the matching algorithm when directly calculating odometers with ordinary binocular cameras.

[0046] In addition, under the preferred embodiment, the method provided in the embodiment of the present application takes into account the overlapping pixel blocks of multiple monocular cameras and performs a two-step pose solution to obtain accurate drone positioning information. In the first step, the drone's preliminary relative pose transformation matrix is ​​quickly solved based on small-scale feature matching of overlapping pixels. In the second step, feature points and the preliminary transformation matrix are matched in non-overlapping areas, and more residual models are constructed to optimize reprojection errors, ultimately obtaining an accurate positioning solution. This method enables drones to locate and navigate indoors, outdoors, and in other locations, solving the problem of difficult positioning when there is no GPS signal.

[0047] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be derived from these drawings without inventive effort.

[0049] Figure 1 This is a flow chart of a multi-view positioning method for a drone provided by an embodiment of the present invention;

[0050] Figure 2 1 is a schematic diagram of the layout of a three-view monocular camera provided by an embodiment of the present invention;

[0051] Figure 3 This is a diagram of a three-view visual odometry positioning framework based on overlapping feature-rich blocks provided by an embodiment of the present invention;

[0052] Figure 4 1 is a schematic diagram of a multi-view positioning device for a drone provided by an embodiment of the present invention;

[0053] Figure 5 This is a schematic structural diagram of a multi-view positioning device for a drone provided by an embodiment of the present invention;

[0054] Figure 6 This is a diagram of the ORB-SLAM2 algorithm framework provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of the present invention.

[0056] See also Figure 1 , is a multi-view positioning method for a drone provided by an embodiment of the present invention, such as Figure 1 As shown, the method may include:

[0057] S101: Receive multiple first images captured by a multi-view image acquisition device; the multiple first images include image information of the first moment of the drone's surrounding environment under different viewpoints; the multi-view image acquisition device provided in the embodiment of the present application may include multiple cameras, and the embodiment of the present application may provide that the multi-view image acquisition device includes multiple monocular cameras 1 installed in a surrounding manner and having equal angle spacing and equal distance spacing.

[0058] Specifically, the extrinsic parameters between each two cameras in the multiple monocular cameras 1 are calibrated before the visual odometry positioning calculation, and the overlapping range of the images taken by the multiple monocular cameras 1 is obtained based on the extrinsic parameter transformation.

[0059] Furthermore, the monocular cameras 1 include three, and the overlapping pixel area of ​​the images taken by two of the monocular cameras 1 is not less than one-fifth of the total pixel area of ​​the image taken by a single monocular camera 1.

[0060] S102: extracting full-range features of each of the first images to obtain multiple sets of feature point data;

[0061] S103: Determine a three-dimensional map reference coordinate system using a binocular structure formed by two monocular cameras included in the multi-view image acquisition device in combination with the multiple sets of feature point data. In this embodiment of the present application, a three-dimensional map reference coordinate system can be determined using a binocular structure formed by two monocular cameras in combination with the multiple sets of feature point data. In a specific implementation, this embodiment of the present application can provide feature matching for the multiple sets of feature point data to obtain matching point pairs between two images.

[0062] According to the binocular structure formed by the plurality of monocular cameras, the depth of the feature points is calculated by using binocular stereo vision technology to obtain three-dimensional map points.

[0063] An initial three-dimensional map is created using the three-dimensional map points, and the current UAV posture is assigned to a unit matrix, which is set as the origin of the reference coordinate system in the three-dimensional map to determine the three-dimensional map reference coordinate system.

[0064] S104: determining overlapping pixel blocks of a plurality of sets of feature point data, and dividing the overlapping pixel blocks to obtain feature-rich blocks in the overlapping area;

[0065] S105: Receive multiple second images captured by a multi-view image acquisition device; the multiple second images include image information of the drone's surrounding environment at a second moment under different viewpoints; the second moment is a moment subsequent to the first moment;

[0066] S106: Obtain the rich feature block feature points of the rich feature block, and search for feature points matching the rich feature block feature points in multiple second images to obtain matching feature point pairs; specifically, mark multiple groups of the feature point data, and perform quadtree division on the overlapping pixel blocks to obtain quadtree node blocks, so as to obtain the overlapping rich feature block feature points.

[0067] S107: Use the matching feature point pairs to construct a geometric reprojection error and perform beam adjustment optimization to solve the relative pose transformation matrix between the drone at the first moment and the drone at the second moment in the three-dimensional map reference coordinate system; specifically, use the matching feature point pairs to construct a geometric reprojection error and perform beam adjustment optimization to solve the rotation transformation matrix and translation transformation vector of the drone at adjacent moments to solve the relative pose transformation matrix.

[0068] S108: performing feature point matching on the non-overlapping pixel area at the first moment, and constructing a reprojection residual based on the relative posture transformation matrix as a projection transformation matrix;

[0069] S109: Using the reprojection residual as an initial value for beam adjustment optimization to perform least squares iterative optimization, and obtaining a target transformed pose of the UAV in the three-dimensional map reference coordinate system.

[0070] The embodiment of the present application provides a multi-perspective positioning method for drones. The drone adopts a large-field-of-view, multi-perspective image source, uses a multi-camera mode as a sensing device, and pre-solves the pixel overlapping blocks of multiple cameras based on the camera installation position and calibration technology. The feature points are quickly searched and matched based on the feature-rich pixel blocks in the overlapping area of ​​the multi-perspective image, which can effectively reduce the high computing resource consumption brought by traditional visual SLAM multi-camera cameras.

[0071] At the same time, the three-dimensional depth of the image points is restored by stereo vision technology for the matching feature points of the multi-view images taken at each moment, and the matching feature points of the multi-view images at adjacent moments are optimized by cluster adjustment through geometric reprojection errors, which further improves the computational efficiency of the traditional SLAM algorithm and can quickly solve the real-time position and positioning information of the drone.

[0072] The following describes in detail the method provided in the embodiment of the present application by taking the setting of three monocular cameras as an example.

[0073] like Figure 2 As shown, by surrounding and fixing three monocular cameras to the drone, a three-perspective visual image acquisition device is formed in an installation manner with equal angle intervals and equal distance intervals. Whenever the visual image acquisition device captures a new three-perspective image, the three-perspective image is processed by visual odometry calculation using the method provided in the embodiment of the present application to quickly calculate a robust visual pose, thereby helping the drone to complete the visual positioning and mapping function.

[0074] The required hardware equipment consists of three fixed monocular cameras in a circle, which are installed in a circle with equal angles and equal distances. The extrinsic parameters between each of the three cameras are calibrated before the visual odometry positioning calculation, and based on the extrinsic parameter transformation, the overlapping range of the images taken by the three monocular cameras is obtained. Specifically, the surrounding installation position of the fixed cameras should ensure equal angles and equal distances, and the overlapping pixel area of ​​the images taken by each camera must be guaranteed to be no less than one-fifth of the total number of pixels of the image taken by a single camera. The camera parameter matrices of the three-view visual image acquisition equipment are calibrated before the visual odometry positioning calculation, and the pixel overlapping range of the images taken by the three monocular cameras is solved based on the extrinsic parameter transformation. The camera extrinsic parameters and overlapping range will be input as system parameters into the subsequent visual odometry positioning solution.

[0075] The visual odometry positioning calculation can be divided into the visual initialization stage and the visual tracking stage, such as Figure 3 As shown, the following steps are included:

[0076] Acquire three-view monocular image data, which contains image information of the drone's surrounding environment from different perspectives;

[0077] Perform full-range feature extraction on the input three-view images to obtain three sets of feature point data;

[0078] Based on the pixel overlapping range of the three-view visual image acquisition device, the feature points belonging to the overlapping range in the three groups of feature points obtained are marked, and the overlapping pixel blocks are divided into quadtrees to obtain quadtree node blocks with more feature points, that is, overlapping feature-rich block feature points are obtained; it is determined whether the three-view image data currently input is the first input. If the three-view image data is currently input for the first time, the processing process enters the initialization stage. If it is not the first input image data, the processing process enters the visual odometer tracking stage. It can be understood that when the first image is obtained in the embodiment of the present application, it can be considered to be the initialization stage. Of course, the initialization stage can also be implemented by separately obtaining three monocular images.

[0079] The first three-view image input begins the visual initialization phase. This phase first extracts full-range features from the three monocular images and partitions the overlapping region into a quadtree to obtain feature points for the feature-rich blocks in the overlapping region. Based on the binocular architecture formed by the two cameras, this phase further utilizes binocular stereo vision technology to recover the 3D depth information of the feature points, generating an initial sparse 3D map. The pose of the current image frame is then assigned to the unit matrix, which is then set as the origin of the 3D map reference coordinate system.

[0080] The initialization phase is mainly used to obtain the initial 3D map point data and the 3D map reference coordinate system. The specific steps are as follows:

[0081] Perform feature matching on the three sets of feature point data obtained by calculation to obtain matching point pairs between each two images;

[0082] Based on the binocular structure formed by three monocular cameras, binocular stereo vision technology is used to calculate the depth of the matching feature points to obtain the three-dimensional map points;

[0083] Based on the obtained 3D map points, an initial 3D map is created, the current UAV pose is assigned to the unit matrix, and it is set as the origin of the reference coordinate system in the 3D map.

[0084] The three-view images that are not input for the first time will enter the visual tracking stage for processing. In this stage, the feature points of the feature-rich blocks in the overlapping area marked during the initialization stage are first matched. Based on the matched feature points, the n-point perspective (PnP: Perspective-n-Points) beam adjustment optimization is performed to preliminarily solve the rotation transformation matrix R′ and translation transformation vector t′ of the drone at adjacent moments to form a preliminary pose transformation matrix T′.

[0085] Then, based on T′, the image feature points of the non-overlapping area in the three-view image at the previous moment are projected into the image at the current moment, and the feature matching operation is completed within a limited search range to obtain more matching feature points to form the reprojection error equation. Finally, T′ is used as the initial value to participate in the bundle adjustment least squares optimization, and the precise relative position change T of the drone at adjacent moments is iteratively optimized. The visual odometry tracking stage is used during the drone's flight, and the real-time positioning information of the current drone is continuously obtained by solving the input three-view images.

[0086] The specific steps are as follows:

[0087] For the feature points of the feature-rich blocks in the pixel overlapping area of ​​the three-view image at the previous moment, find matching feature points in the three-view image taken at the current moment;

[0088] Based on the obtained matching feature point pairs, the geometric reprojection error is constructed, and the beam adjustment optimization is performed to solve the relative pose transformation matrix T′ between the UAV at the previous moment and the UAV at the current moment;

[0089] Feature point matching is performed on the non-overlapping pixel area at the previous moment, and the reprojection residual is constructed based on the T′ calculated in the previous step as the projection transformation matrix. At the same time, the least squares iterative optimization is performed as the initial value of the PnP beam adjustment optimization to obtain the accurate UAV transformation pose T.

[0090] In summary, the multi-perspective positioning method for drones provided in this application uses multiple monocular cameras to complete the real-time solution of the drone's posture odometer and reconstruct a three-dimensional map model of the images taken by the drone, avoiding the problem of being unable to calculate the true three-dimensional environment scale when using a monocular camera alone, the problem of limited image shooting distance when using a depth camera, and the high time complexity of the matching algorithm when ordinary binocular cameras are directly used for odometer solution.

[0091] At the same time, the method provided in the embodiments of this application takes into account the overlapping pixel blocks of multiple monocular cameras and performs a two-step pose solution to obtain accurate drone positioning information. The first step of the solution quickly solves the drone's preliminary relative pose transformation matrix based on small-scale feature matching of overlapping pixels. The second step uses non-overlapping regions to match feature points and the preliminary transformation matrix, constructing more residual models to optimize reprojection errors, ultimately obtaining an accurate positioning solution. This method enables drone positioning and navigation indoors, outdoors, and in other locations, solving the problem of positioning difficulties in the absence of GPS signals.

[0092] See also Figure 4 , the embodiment of the present application can also provide a multi-view positioning device for a drone, such as Figure 4 As shown, the device may include:

[0093] The first image receiving unit 401 is configured to receive a plurality of first images captured by a multi-view image acquisition device; the plurality of first images include image information of the drone's surrounding environment at a first moment under different view angles;

[0094] A feature point data extraction unit 402 is configured to extract full-range features of each of the first images to obtain multiple sets of feature point data;

[0095] A coordinate system determining unit 403 is configured to determine a three-dimensional map reference coordinate system by utilizing a binocular structure formed by two monocular cameras included in the multi-view image acquisition device in combination with multiple sets of feature point data;

[0096] A feature-rich block determining unit 404 is configured to determine overlapping pixel blocks of a plurality of sets of feature point data, and to divide the overlapping pixel blocks to obtain feature-rich blocks in the overlapping area;

[0097] The second image receiving unit 405 is configured to receive a plurality of second images captured by the multi-view image acquisition device; the plurality of second images include image information of the drone's surrounding environment at a second moment under different view angles; the second moment is the next moment after the first moment;

[0098] A matching feature point pair acquisition unit 406 is configured to acquire feature points of the feature-rich block, and search for feature points matching the feature points of the feature-rich block in a plurality of second images to obtain matching feature point pairs;

[0099] a relative pose transformation matrix acquisition unit 407, configured to construct a geometric reprojection error using the matching feature point pairs and perform beam adjustment optimization to calculate a relative pose transformation matrix between the UAV at the first moment and the UAV at the second moment in the three-dimensional map reference coordinate system;

[0100] a reprojection residual construction unit 408, configured to perform feature point matching on the non-overlapping pixel region at the first moment, and construct a reprojection residual based on the relative posture transformation matrix as a projection transformation matrix;

[0101] The target transformation pose acquisition unit 409 is configured to perform least squares iterative optimization using the reprojection residual as an initial value for beam adjustment optimization to obtain the target transformation pose of the UAV in the three-dimensional map reference coordinate system.

[0102] The present application also provides a multi-view positioning device for a drone, the device comprising a processor and a memory:

[0103] The memory is used to store program code and transmit the program code to the processor;

[0104] The processor is used to execute the steps of the above-mentioned drone multi-view positioning method according to the instructions in the program code.

[0105] like Figure 5 As shown, an embodiment of the present application provides a multi-view positioning device for a drone, which may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, the memory 11, and the communication interface 12 all communicate with each other via the communication bus 13.

[0106] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field programmable gate array, or other programmable logic devices.

[0107] The processor 10 can call the program stored in the memory 11. Specifically, the processor 10 can execute the operations in the embodiment of the multi-view positioning method for a drone.

[0108] The memory 11 is used to store one or more programs. The program may include program code, and the program code includes computer operating instructions. In the embodiment of the present application, the memory 11 stores at least a program for implementing the following functions:

[0109] Receiving a plurality of first images captured by a multi-view image acquisition device; the plurality of first images include image information of the drone's surrounding environment at a first moment under different view angles;

[0110] Extracting full-range features of each of the first images to obtain multiple sets of feature point data;

[0111] Determine a three-dimensional map reference coordinate system using a binocular structure formed by two monocular cameras included in the multi-view image acquisition device in combination with multiple sets of feature point data;

[0112] Determining overlapping pixel blocks of a plurality of sets of feature point data, and dividing the overlapping pixel blocks to obtain feature-rich blocks in the overlapping area;

[0113] Receiving a plurality of second images captured by a multi-view image acquisition device; the plurality of second images include image information of the drone's surrounding environment at a second moment under different view angles; the second moment being a moment subsequent to the first moment;

[0114] Acquire feature points of the feature-rich block, and search for feature points matching the feature points of the feature-rich block in a plurality of second images to obtain matching feature point pairs;

[0115] Constructing a geometric reprojection error using the matching feature point pairs and performing beam adjustment optimization to calculate a relative pose transformation matrix between the UAV at the first moment and the UAV at the second moment in the three-dimensional map reference coordinate system;

[0116] Performing feature point matching on the non-overlapping pixel area at the first moment, and constructing a reprojection residual based on the relative posture transformation matrix as a projection transformation matrix;

[0117] The reprojection residual is used as the initial value of the beam adjustment optimization to perform least squares iterative optimization to obtain the target transformed pose of the UAV in the three-dimensional map reference coordinate system.

[0118] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required for at least one function (such as a file creation function, a data reading and writing function), etc.; the data storage area can store data created during use, such as initialization data, etc.

[0119] In addition, the memory 11 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0120] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0121] Of course, it needs to be explained that Figure 5 The structure shown does not constitute a limitation on the multi-view positioning device for drones in the embodiment of the present application. In actual applications, the multi-view positioning device for drones may include Figure 5 More or fewer components than shown, or combinations of certain components.

[0122] An embodiment of the present application may also provide a computer-readable storage medium, which is used to store program code, and the program code is used to execute the steps of the above-mentioned drone multi-perspective positioning method.

[0123] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0124] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present application.

[0125] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0126] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A multi-view positioning method for unmanned aerial vehicles, characterized in that: The method comprises: Receiving a plurality of first images captured by a multi-view image acquisition device; the plurality of first images include image information of the drone's surrounding environment at a first moment under different view angles; Extracting full-range features of each of the first images to obtain multiple sets of feature point data; Determine a three-dimensional map reference coordinate system using a binocular structure formed by two monocular cameras included in the multi-view image acquisition device in combination with multiple sets of feature point data; Determining overlapping pixel blocks of a plurality of sets of feature point data, and dividing the overlapping pixel blocks to obtain feature-rich blocks in the overlapping area; receiving a plurality of second images captured by the multi-view image acquisition device; the plurality of second images including image information of the drone's surrounding environment at a second moment under different view angles; the second moment being a moment subsequent to the first moment; Acquire feature points of the feature-rich block, and search for feature points matching the feature points of the feature-rich block in a plurality of second images to obtain matching feature point pairs; Constructing a geometric reprojection error using the matching feature point pairs and performing beam adjustment optimization to calculate a relative pose transformation matrix between the UAV at the first moment and the UAV at the second moment in the three-dimensional map reference coordinate system; Performing feature point matching on the non-overlapping pixel area at the first moment, and constructing a reprojection residual based on the relative posture transformation matrix as a projection transformation matrix; The reprojection residual is used as the initial value of the beam adjustment optimization to perform least squares iterative optimization to obtain the target transformed pose of the UAV in the three-dimensional map reference coordinate system.

2. The multi-view positioning method for unmanned aerial vehicles according to claim 1, characterized in that: The multi-view image acquisition device includes a plurality of monocular cameras that are installed in a surrounding manner and have equal angular spacing and equal distance spacing.

3. The multi-view positioning method for unmanned aerial vehicle according to claim 2, characterized in that: The external parameters between each of the multiple monocular cameras are calibrated before the visual odometry positioning calculation, and the overlapping range of the images taken by the multiple monocular cameras is obtained based on the external parameter transformation.

4. The multi-view positioning method for unmanned aerial vehicle according to claim 3, characterized in that: There are three monocular cameras, and the overlapping pixel areas of the images taken by two of the monocular cameras are not less than one-fifth of the total pixels of the image taken by a single monocular camera.

5. The multi-view positioning method for unmanned aerial vehicle according to claim 1, characterized in that: The method of determining a three-dimensional map reference coordinate system by utilizing a binocular structure formed by two monocular cameras included in the multi-view image acquisition device in combination with multiple sets of feature point data comprises: Performing feature matching on the plurality of sets of feature point data to obtain matching point pairs between any two images; According to a binocular structure formed by a plurality of monocular cameras in pairs, a binocular stereo vision technology is used to calculate the depth of the matching feature point pairs to obtain three-dimensional map points; An initial three-dimensional map is created using the three-dimensional map points, and the current UAV posture is assigned to a unit matrix, which is set as the origin of the reference coordinate system in the three-dimensional map to determine the three-dimensional map reference coordinate system.

6. The multi-view positioning method for unmanned aerial vehicle according to claim 1, characterized in that: The obtaining of the feature-rich block feature points of the feature-rich block includes: The plurality of sets of feature point data are marked, and the overlapping pixel blocks are divided into quadtrees to obtain quadtree node blocks, so as to obtain feature points of overlapping feature-rich blocks.

7. The multi-view positioning method for unmanned aerial vehicle according to claim 1, characterized in that: The method of constructing a geometric reprojection error by using the matching feature point pairs and performing beam adjustment optimization to obtain a relative pose transformation matrix between the UAV at the first moment and the UAV at the second moment in the three-dimensional map reference coordinate system comprises: The matching feature point pairs are used to construct geometric reprojection errors and perform beam adjustment optimization to solve the rotation transformation matrix and translation transformation vector of the drone at adjacent moments to obtain the relative posture transformation matrix.

8. A multi-view positioning device for an unmanned aerial vehicle, characterized in that: The device comprises: A first image receiving unit is configured to receive a plurality of first images captured by a multi-view image acquisition device; the plurality of first images include image information of the drone's surrounding environment at a first moment under different view angles; a feature point data extraction unit, configured to extract full-range features of each of the first images to obtain multiple sets of feature point data; A coordinate system determining unit, configured to determine a three-dimensional map reference coordinate system by utilizing a binocular structure formed by two monocular cameras included in the multi-view image acquisition device in combination with multiple sets of feature point data; a feature-rich block determining unit, configured to determine overlapping pixel blocks of a plurality of sets of feature point data, and to divide the overlapping pixel blocks to obtain feature-rich blocks in the overlapping area; A second image receiving unit is configured to receive a plurality of second images captured by the multi-view image acquisition device; the plurality of second images include image information of the drone's surrounding environment at a second moment under different view angles; the second moment is a moment subsequent to the first moment; a matching feature point pair acquisition unit, configured to acquire feature points of the feature-rich block, and search for feature points matching the feature points of the feature-rich block in a plurality of second images to obtain matching feature point pairs; a relative pose transformation matrix acquisition unit, configured to construct a geometric reprojection error using the matching feature point pairs and perform beam adjustment optimization to calculate the relative pose transformation matrix between the UAV at the first moment and the UAV at the second moment in the three-dimensional map reference coordinate system; a reprojection residual construction unit, configured to perform feature point matching on the non-overlapping pixel region at the first moment, and construct a reprojection residual based on the relative posture transformation matrix as a projection transformation matrix; The target transformation pose acquisition unit is used to perform least squares iterative optimization using the reprojection residual as the initial value of the beam adjustment optimization to obtain the target transformation pose of the UAV in the three-dimensional map reference coordinate system.

9. A multi-view positioning device for drones, characterized in that: The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the drone multi-view positioning method according to any one of claims 1 to 7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store program code, and the program code is used to execute the multi-perspective positioning method for a drone according to any one of claims 1 to 7.

Images