Unmanned aerial vehicle positioning method, electronic device, and storage medium

Abstract

The application discloses a UAV positioning method, an electronic device and a storage medium. The UAV positioning method comprises the following steps: acquiring an initial pose of a UAV in a relative coordinate system when the UAV collects a current frame image; determining a current scaling scale between the relative coordinate system and a 3D coordinate system according to a target object in the current frame image; and performing conversion processing on the initial pose according to an acquired target pose transformation relationship between the relative coordinate system and the 3D coordinate system and the current scaling scale, so as to obtain a target pose of the UAV in the 3D coordinate system. The above scheme can improve the accuracy of the determined target pose, thereby improving the accuracy of UAV positioning.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a UAV positioning method, electronic device, and storage medium. Background Technology

[0002] In recent years, the technology of real-time fusion of images and videos acquired by drones with 3D scenes has been widely used in fields such as digital twins and video twins. However, the fusion process heavily relies on the accurate real-time positioning of the drone. If the quality of the drone's real-time positioning (3D geographic information scene) is poor, the fusion effect between the acquired video and the 3D geographic information scene will be poor.

[0003] Therefore, there is an urgent need for an effective method for locating drones. Summary of the Invention

[0004] This application provides at least one drone positioning method, electronic device, and storage medium, which can improve the accuracy of drone positioning.

[0005] This application provides a UAV positioning method, which includes: acquiring the initial pose of the UAV in a relative coordinate system when acquiring the current frame image; determining the current scaling scale between the relative coordinate system and the 3D coordinate system based on the target object in the current frame image; and performing transformation processing on the initial pose based on the acquired target pose transformation relationship between the relative coordinate system and the 3D coordinate system and the current scaling scale to obtain the target pose of the UAV in the 3D coordinate system.

[0006] This application provides a UAV positioning device, including: an acquisition module, a determination module, and a conversion module; the acquisition module is used to acquire the initial pose of the UAV in the relative coordinate system when acquiring the current frame image; the determination module is used to determine the current scaling scale between the relative coordinate system and the 3D coordinate system based on the target object in the current frame image; the conversion module is used to convert the initial pose according to the acquired target pose transformation relationship between the relative coordinate system and the 3D coordinate system and the current scaling scale to obtain the target pose of the UAV in the 3D coordinate system.

[0007] This application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-described UAV positioning method.

[0008] This application provides a computer-readable storage medium storing program instructions thereon, which, when executed by a processor, implement the above-described UAV positioning method.

[0009] The above scheme obtains the initial pose of the UAV in the relative coordinate system when acquiring the current frame image; determines the current scaling scale between the relative coordinate system and the 3D coordinate system based on the target object in the current frame image; transforms the initial pose according to the obtained target pose transformation relationship between the relative coordinate system and the 3D coordinate system and the current scaling scale to obtain the target pose of the UAV in the 3D coordinate system. Thus, an accurate initial pose is obtained from the current frame image. The current scaling scale and the target pose transformation relationship represent information in different dimensions between the relative coordinate system and the 3D coordinate system. Furthermore, the initial pose is transformed in real time based on the comprehensive information in different dimensions between the relative coordinate system and the 3D coordinate system to improve the accuracy of the target pose in the 3D coordinate system obtained by transformation.

[0010] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0012] Figure 1 This is a first flowchart illustrating an exemplary embodiment of the UAV positioning method of this application; Figure 2 This is a second flowchart illustrating an exemplary embodiment of the UAV positioning method of this application; Figure 3 This is a schematic diagram of the third process of an exemplary embodiment of the UAV positioning method of this application; Figure 4 This is a schematic diagram of the structure of an embodiment of the drone positioning device of this application; Figure 5 This is a schematic diagram of the structure of an embodiment of the electronic device of this application; Figure 6 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are only for explaining this application and not for limiting it. Furthermore, it should be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, not all structures. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0014] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. The term "and / or" is merely a description of the association of related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, "many" in this document means two or more. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of elements, for example, including at least one of A, B, and C, which can mean including any one or more elements selected from the set consisting of A, B, and C. Additionally, the term "several" in this document means one or more.

[0016] This application provides several drone positioning methods and devices. The application scenarios for these drone positioning methods include, but are not limited to, drone positioning scenarios, drone-denied environments, and the fusion of images acquired by drones with 3D environmental information. For example, the application scenarios of this application can be drone inspection and video fusion within a digital twin scenario, specifically: coastline inspection, industrial inspection, disaster monitoring, etc. The executing entity of the drone positioning method can be a drone positioning device, such as a drone or a cloud device communicating with the drone. For instance, the drone positioning device can be located in a terminal device, server, or other processing device. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, vehicle-mounted device, etc. In some possible implementations, the drone positioning method can be implemented by a processor calling computer-readable instructions stored in memory.

[0017] Please see Figure 1 , Figure 1 This is a first flowchart illustrating an exemplary embodiment of the UAV positioning method of this application. Specifically, the UAV positioning method may include S11, S12, and S13: S11: Obtain the initial pose of the UAV in the relative coordinate system when acquiring the current frame image.

[0018] A drone can be a device capable of image acquisition. The current frame image is an image captured by the drone of its driving environment at the current moment. The current frame image can be an image acquired in real time by the drone. The drone includes an image acquisition device used to acquire images of the driving environment in which the drone is located. The relative coordinate system represents the coordinate system in which the relative map is located when the drone constructs a relative map based on the current frame image and a preset map construction algorithm. The relative coordinate system is the coordinate system in which the relative map is located. Specifically, the preset map construction algorithm can be visual SLAM, namely Simultaneous Localization and Mapping (SLAM) based on camera vision. Visual SLAM is a core technology that enables robots to locate and construct environmental maps in real time in unknown environments using image sensors. The initial pose represents the pose of the drone in the relative coordinate system or in the relative map when it acquires the current frame image. The initial pose includes the initial attitude angle related to the attitude angle and the initial position information related to the position information.

[0019] In some application scenarios, S11 above can be: the UAV collects images of the driving environment in real time to obtain the current frame image; the current frame image is input into a preset map building algorithm to obtain the relative map output by the preset map building algorithm and the relative pose of the UAV when collecting the current frame image in the relative map, and the relative pose is used as the above initial pose.

[0020] In some application scenarios, prior to S11 above, the UAV positioning method may further include the following steps: determining whether the UAV is in a denied environment; in response to the UAV being in a denied environment, performing the step described above of the UAV acquiring images of the driving environment in real time to obtain the current frame image. A denied environment indicates that the signal quality of the GPS signal of the UAV at its current driving position does not meet the signal quality requirements.

[0021] S12: Determine the current scaling scale between the relative coordinate system and the 3D coordinate system based on the target object in the current frame image.

[0022] The target object can be a calibrated landmark in the drone's operating environment. For example, a landmark can be a checkerboard pattern or other objects with known true size parameters. The true size parameter can be the actual physical distance between at least two preset feature points in the target object. The 3D coordinate system can refer to the world coordinate system. Specifically, the true size parameter can be the actual physical distance between at least two preset feature points in the target object in the 3D coordinate system. The current zoom scale represents the zoom scale between the relative coordinate system of the relative map determined based on the current frame image and the 3D coordinate system. For example, the 3D coordinate system can refer to the three-dimensional coordinate system of a constructed oblique photogrammetry model. An oblique photogrammetry model is a three-dimensional model generated through multi-angle photogrammetry, often referred to as a "real-world 3D model." This model is generated by images captured by multi-angle cameras carried by a drone or aerial equipment, and can include cameras with vertical viewing angles and four oblique viewing angles (front, back, left, and right) to ensure that the acquired data contains three-dimensional elements. For a detailed explanation of the oblique photogrammetry model in the technical field, please refer to the relevant explanations; they will not be elaborated upon here.

[0023] Specifically, S12 can be: obtaining the current scaling scale based on the detection information of the target object in the current frame image.

[0024] S13: Based on the target pose transformation relationship between the relative coordinate system and the 3D coordinate system and the current scaling scale, the initial pose is transformed to obtain the target pose of the UAV in the 3D coordinate system.

[0025] Transformation processing can involve converting poses between two coordinate systems using the pose transformation relationship between the relative coordinate system and the 3D coordinate system, as well as scaling. Target pose represents the pose of the UAV in the 3D coordinate system or in the oblique photogrammetry model when acquiring the current frame image. Target pose includes target attitude angles related to attitude angles and target position information related to position information.

[0026] It should be noted that the coordinate information obtained in the initial pose usually cannot directly reflect the position information of the UAV in a three-dimensional coordinate system. Therefore, this application uses a coordinate transformation method to transform the real-time coordinate information obtained by the UAV into a three-dimensional coordinate system, thus obtaining the three-dimensional coordinates.

[0027] The above scheme obtains the initial pose of the UAV in the relative coordinate system when acquiring the current frame image. Based on the target object in the current frame image, the current scaling scale between the relative coordinate system and the 3D coordinate system is determined. The initial pose is transformed based on the obtained target pose transformation relationship between the relative and 3D coordinate systems and the current scaling scale to obtain the target pose of the UAV in the 3D coordinate system. Thus, an accurate initial pose is obtained from the current frame image. The current scaling scale and the target pose transformation relationship represent information in different dimensions between the relative and 3D coordinate systems. Furthermore, the initial pose is comprehensively transformed in real time based on the information in these different dimensions to improve the accuracy of the target pose in the obtained 3D coordinate system.

[0028] In some embodiments, S12 may include the following steps: In response to the current frame image not containing a target object, using a preset scaling scale as the current scaling scale; In response to the current frame image containing a target object, obtaining the current scaling scale based on at least two edge feature points of the target object in the current frame image.

[0029] The preset scaling scale can be either the historical scaling scale corresponding to historical frame images or the default scaling scale. The historical frame images were acquired earlier than the current frame image. The historical frame images are the previous frame images acquired by the UAV for the current frame image. Historical frame images are also used to construct relative maps. The historical frame images are input into a preset map construction algorithm to obtain the relative map output by the algorithm. The method for determining the relative pose of the UAV when acquiring historical frame images in the relative map is the same as the method for determining the relative pose when acquiring the current frame image, and will not be elaborated here. The preset map construction algorithm constructs the relative map and relative pose by continuously optimizing the output real-time frame images using continuous real-time frame images. The default scaling scale can be a preset scaling scale, dynamically set according to the accuracy of UAV positioning, or it can be the scaling scale between the relative coordinate system and the 3D coordinate system of the initialized map obtained by the preset map construction algorithm during the initialization mapping process.

[0030] If the target object is not present in the current frame image, the preset scaling scale is directly used as the current scaling scale. If the target object is present in the current frame image, the current scaling scale is constructed in real time based on at least two edge feature points of the target object in the current frame image.

[0031] In some embodiments, the step of obtaining the current scaling scale based on at least two edge feature points of the target object in the current frame image includes: obtaining the true distance between each edge feature point and the image distance between each edge feature point in the current frame image. The ratio between the true distance and the image distance is used as the current scaling scale.

[0032] Image distance represents the pixel distance between corresponding edge feature points in the current frame image. Ground truth distance and image distance are the distances between at least two identical locations of the target object, respectively.

[0033] In some application scenarios, obtaining the true distance between edge feature points can be achieved by retrieving the size parameters of the corresponding target object from the database and using the distance between the corresponding edge feature points in the target object's size parameters as the image distance. Alternatively, the image distance can be obtained by using the corresponding edge feature points in the current frame image as candidate edge feature points and using the pixel distance between each candidate edge feature point in the current frame image as the image distance. For example, feature points on a marker are detected in the current frame image, and the true physical distance of these feature points is known. For instance, if the target object is a chessboard, the distance between two points on the edge of one square on the chessboard is known; this is the true distance corresponding to the marker target feature point. Within the map constructed by visual SLAM, the 3D coordinates corresponding to the two feature points are found based on the feature point's index, and the Euclidean distance between the two 3D coordinates is calculated. This distance is then used as the image distance between the aforementioned feature points.

[0034] Please see Figure 2 , Figure 2 This is a second flowchart illustrating an exemplary embodiment of the drone positioning method of this application.

[0035] In some embodiments, prior to S13, the UAV localization method further includes the following steps: S21: Acquiring an initial mapping image and a pre-calibration image captured by the UAV. S22: Obtaining the first pose of the UAV in the relative coordinate system when acquiring the pre-calibration image based on the pre-calibration image, the initial mapping image, and the initial mapping pose of the UAV in the relative coordinate system when acquiring the initial mapping image. S23: Determining the target pose transformation relationship based on the second pose and the first pose of the UAV in the 3D coordinate system when acquiring the pre-calibration image.

[0036] The initial mapping image is a real-time frame image acquired by the UAV during the initial mapping process using a preset map building algorithm. For example, the initial mapping image is the first frame image acquired by the UAV when initial mapping is successful. The pre-calibration image represents the image acquired by the UAV from the driving environment including the target object. The image content of the pre-calibration image includes the target object and the pose of the UAV in the 3D coordinate system when acquiring the pre-calibration image is pre-calibrated. The acquisition time of the initial mapping image is earlier than the acquisition time of the current frame image. The initial mapping image can be obtained by: selecting one of the real-time frame images acquired by the UAV during the initial mapping process as the initial mapping image; or, performing image matching between the real-time frame images acquired by the UAV during the initial mapping process and the pre-calibration image, and using the real-time frame image with successful image matching as the initial mapping image.

[0037] The initial mapping pose represents the pose of the UAV in a relative coordinate system when acquiring the initial mapping image. The initial mapping pose can be obtained by inputting the initial mapping image into a preset map building algorithm, obtaining the relative map output by the algorithm, and the pose of the UAV in the coordinate system corresponding to the relative map when acquiring the initial mapping image. This pose is then used as the initial mapping pose. The first pose represents the pose of the UAV in a relative coordinate system when acquiring the pre-calibrated image. The second pose represents the pose of the UAV in a 3D coordinate system when acquiring the pre-calibrated image; these are pre-calibrated parameters and can be directly obtained. The target pose transformation relationship represents the transformation relationship between the first pose and the second pose. The target pose transformation relationship includes a first target pose transformation relationship regarding attitude angles and a second target pose transformation relationship regarding position information.

[0038] The steps described above for obtaining the first pose of the UAV in the relative coordinate system when acquiring the pre-calibration image based on the pre-calibration image, the initial mapping image, and the initial mapping pose in the relative coordinate system when the UAV acquires the initial mapping image can be as follows: determine the pose transformation parameters between the pre-calibration image and the initial mapping image based on the pre-calibration image and the initial mapping image; obtain the first pose of the UAV in the relative coordinate system when acquiring the pre-calibration image based on the pose transformation parameters between the pre-calibration image and the initial mapping image and the initial mapping pose.

[0039] Please see Figure 3 , Figure 3 This is a schematic diagram of the third process of an exemplary embodiment of the UAV positioning method of this application.

[0040] In some embodiments, the step of acquiring the initial mapping image collected by the UAV may include the following steps: S31: In response to the UAV receiving a first operation command, acquire the first acquired image collected by the UAV while it is operating according to the first operation command. S32: Perform image matching between the first acquired image and the pre-calibrated image to obtain an image matching result. S33: In response to the image matching result meeting a preset condition, use the first acquired image as the initial mapping image.

[0041] The first operation command represents the operation command generated in response to the first operation. The first operation is the setting of the UAV's travel route and image acquisition frequency during the initialization mapping process. The first acquired image is the image acquired by the UAV while traveling along the travel route corresponding to the first operation command.

[0042] Image matching results characterize the feature point matching degree between each feature point in the first acquired image and each feature point in the pre-calibrated image.

[0043] The preset condition can be that the image matching degree meets the matching requirements, specifically whether the feature point matching degree in the image matching result is greater than the matching degree threshold.

[0044] In some application scenarios, if the image matching result meets preset conditions, the first acquired image is used as the initial mapping image. Specifically, if the feature point matching degree in the image matching result is greater than the matching degree threshold, the first acquired image is used as the initial mapping image.

[0045] In other application scenarios, in response to the image matching result not meeting preset conditions, candidate images acquired by the UAV during its flight according to the first operation command are reacquired. Based on the image matching result between the candidate images and the pre-calibrated images, it is determined whether to use the candidate images as the initial mapping images. It can be understood that if the image matching result does not meet the preset conditions, candidate images acquired by the UAV during its flight according to the first operation command are reacquired until the initial mapping image is determined.

[0046] For example, an accurate image (i.e., a pre-calibrated image) is obtained offline along the UAV's flight path, and the pose parameters in the coordinate system of the oblique photogrammetry 3D model are calculated. This second pose is then used as the second pose in the 3D coordinate system when the pre-calibrated image is displayed. Specifically, the second pose can be constructed by: using a 2D-3D PnP solution method to obtain the initial parameters of the UAV pose; and using a nonlinear optimization method to optimize the initial parameters to obtain the optimal pose parameters. This second pose, in the 3D coordinate system, is used as the pre-calibrated image. In other application scenarios, before constructing the second pose, the above UAV localization method further includes: calibrating the parameters of the image acquisition device in the UAV to obtain a parameter-calibrated image acquisition device; and using the parameter-calibrated image acquisition device to perform subsequent image acquisition in order to obtain the first acquired image. Specifically, the parameters to be calibrated include the intrinsic parameter matrix and distortion coefficients of the image acquisition device. The intrinsic parameter matrix can be represented as... ,in, Focal length The main point is the distortion coefficient, which can be expressed as... The specific method for parameter calibration can use Zhang Zhengyou's checkerboard method, which will not be elaborated here. This step requires the UAV to collect checkerboard images from different angles while stationary, and then use Zhang Zhengyou's calibration method to calculate the lens's intrinsic parameters and distortion parameters for subsequent calculations.

[0047] Before initiating real-time UAV localization, the visual SLAM system for monocular visual odometry needs to be initialized by inputting video or a continuous sequence of images (i.e., completing the initial mapping process described above). After successful initial mapping, subsequent motion estimation steps begin to output the pose of each image frame in the relative coordinate system. Specifically, during the initial mapping process, the input to the preset map building algorithm is: an image sequence and camera intrinsic parameters. The image sequence includes two consecutive frames or at least two monocular images, where at least two frames must have sufficient disparity. The camera intrinsic parameters include the focal length, principal point coordinates, and distortion coefficients obtained from the parameter calibration described above. The output of the preset map building algorithm is: an initial map (i.e., the relative map described above), initial keyframe poses, and feature point tracking relationships. The initial map is a set of 3D map points (MapPoints) with coordinates at a relative scale. The initial keyframe poses include the pose of the first image frame, which is usually set to the origin of the relative coordinate system of the map constructed by SLAM, and the pose of the second image frame, given by R (rotation matrix) and t (translation vector). The first image frame is the first image captured after successful initial mapping, i.e., the initial mapping image mentioned above. R represents the directional change of the second image frame relative to the first image frame (relative coordinate system) in the initial map or relative map, used to describe the rotation of the camera pose. t represents the positional change of the second image frame relative to the first image frame in the initial map or relative map, used to give the coordinates of the camera optical center in the first frame's coordinate system. Feature point tracking relationships represent the association between matched feature points and their corresponding map points, used for subsequent tracking and mapping.

[0048] In other application scenarios, as the drone moves, target detection is performed on the real-time captured current frame image. When the target detection result includes a target object, the aforementioned step of obtaining the current scaling scale based on at least two edge feature points of the target object in the current frame image is triggered to continuously update the scaling scale. This application takes into account that scale drift may occur during the operation of visual SLAM. Scale restoration upon detecting a marker is a correction method that improves positioning accuracy to some extent.

[0049] By using image matching methods (such as ORB features), we find the feature points that match the initial mapping image and the pre-calibration image. By removing the mismatched feature point pairs, we can obtain the pose transformation parameters between the initial mapping image and the pre-calibration image (the pose transformation parameters can be expressed as R and t, which can be obtained by solving the fundamental matrix to decompose R with respect to the pose angle transformation and t with respect to the position information transformation).

[0050] The process of obtaining the first pose of the UAV in the relative coordinate system when acquiring the precalibrated image based on the pose transformation parameters between the precalibrated image and the initial mapping image and the initial mapping pose can be referred to the following formulas (1), (2), (3) and (4).

[0051] quat quat_qcw(qcw.w(), qcw.x(), qcw.y(), qcw.z()) Formula (1); matrix_wcs2vcs = quat_qcw.matrix() Formula (2); qcw_calib = R * matrix_wcs2vcs Formula (3); tcw_calib = R * tcw+ t / medianDepth formula (4); In formula (1), quat_qcw represents the quaternion of the attitude angle in the second pose, specifically qcw.w(), qcw.x(), qcw.y(), and qcw.z(). In formula (2), matrix_wcs2vcs represents the rotation matrix corresponding to the attitude angle in the first pose obtained by transforming the quaternion of the attitude angle in the second pose. quat_qcw.matrix() represents the transformation process performed on the quaternion of the attitude angle in the second pose. * represents multiplication. R represents the transformation parameter of the pose transformation parameters with respect to the attitude angle transformation; it can be understood that R represents the direction change of the current frame image relative to the initial mapping image (relative coordinate system) in the relative map constructed based on the current frame image, used to describe the rotation part of the camera pose. `t` represents the transformation parameter related to position information transformation in the pose transformation parameters; it can be understood as representing the position change of the current frame image relative to the initial mapping image (relative coordinate system) in the relative map constructed based on the current frame image, used to give the coordinates of the camera optical center in the relative coordinate system when acquiring the current frame image. `tcw` represents the position information in the second pose. `qcw_calib` represents the pose angle in the first pose, which can be in the form of a quaternion or a rotation matrix. `tcw_calib` represents the position information in the first pose, and its specific form is not limited. ` / ` represents a semicolon, with `t` as the numerator and `medianDepth` as the denominator. `medianDepth` represents the median depth of the map built by visual SLAM. Median depth refers to the median depth value of all map points observed in the initial mapping image in the camera coordinate system. For example, extract all valid map points from the observation list of the initial mapping image, transform each map point to the camera coordinate system using the initial mapping pose (transformation matrix from the world to the camera); take the components of each map point (i.e., the distance along the camera optical axis); sort the values ​​of all valid components, and take the value at the middle position of the sort as the median depth.

[0052] The process of determining the target pose transformation relationship based on the second pose and the first pose in the 3D coordinate system when the UAV acquires the pre-calibrated image can be referred to the following formulas (5), (6), and (7).

[0053] Calculate the first frame (reference origin) after successful drone initialization. (The scale factor value of the real-time frame can be used to calibrate the image.) matrix_wcs2vcs_calib = quat_qcw_calib.matrix() formula (5); R_ori = A*r_calib formula (6); t_ori = A*(t_calib - scale * tcw_calib) formula (7); Where `matrix_wcs2vcs_calib` represents the rotation matrix corresponding to the attitude angles in the first pose, which can be specifically represented as `quat_qcw_calib(qcw_calib.w(),qcw_calib.x(), qcw_calib.y(),qcw_calib.z())`. `quat_qcw_calib.matrix()` represents the transformation processing of the quaternions of the attitude angles in the second pose. `A` represents `matrix_wcs2vcs_calib.inverse()`, which is the inverse of the rotation matrix corresponding to the attitude angles in the first pose. `r_calib` represents the attitude angles in the second pose. `t_calib` represents the position information in the second pose. `R_ori` represents the first target pose transformation relationship. `t_ori` represents the second target pose transformation relationship. `scale` represents the current scaling scale. `*` represents multiplication.

[0054] In some embodiments, the initial pose includes an initial attitude angle and initial position information, and the target pose transformation relationship represents a first target transformation relationship with respect to the attitude angle and a second target transformation relationship with respect to the position information. The target pose includes a target attitude angle and target position information. S13 described above may include the following steps: determining the target attitude angle based on the initial attitude angle and the first target transformation relationship; and determining the target position information based on the initial attitude angle, initial position information, current scaling scale, and the second target transformation relationship.

[0055] In some application scenarios, the steps described above for determining the target attitude angle based on the initial attitude angle and the first target transformation relationship can be: obtaining the target attitude angle based on the quaternions or rotation matrix in the initial attitude angle and the first target transformation relationship. The quaternions and rotation matrix in the initial attitude angle can be converted to each other.

[0056] In some embodiments, the step of determining the target position information based on the initial attitude angle, initial position information, current scaling scale, and second target transformation relationship may include the following steps: obtaining a first transformation value based on the initial attitude angle and the second target pose transformation relationship; obtaining a second transformation value based on the initial position information and the current scaling scale; and using the sum of the first transformation value and the second transformation value as the target position information.

[0057] The first transformation value represents the intermediate parameter value obtained based on the rotation matrix in the initial attitude angle and the relationship between the second target pose transformation. The second transformation value represents the intermediate parameter value obtained based on the initial position information and the current scaling scale.

[0058] For example, the process of transforming the initial pose based on the obtained target pose transformation relationship between the relative coordinate system and the 3D coordinate system and the current scaling scale to obtain the target pose of the UAV in the 3D coordinate system can be referred to the following formulas (8), (9), (10) and (11): Quat quat_qcw(qcw.w(), qcw.x(), qcw.y(), qcw.z()) formula (8); matrix_wcs2vcs = quat_qcw.matrix() Formula (9); r_3d = matrix_wcs2vcs.inverse() * R_ori Formula (10); t_3d = matrix_wcs2vcs * t_ori + scale * tcw formula (11); In formula (8), Quat quat_qcw represents the quaternion of the attitude angle in the initial pose. In formulas (9) and (10), matrix_wcs2vcs represents the rotation matrix of the attitude angle in the initial pose. * represents multiplication. R_ori represents the first target transformation relation, which is in matrix form. r_3d represents the relevant parameters of the attitude angle in the target pose, and its attitude angle, quaternion and rotation matrix can be converted to each other. t_3d represents the relevant parameters of the position information in the target pose. matrix_wcs2vcs * t_ori represents the first transformation value obtained according to the initial attitude angle and the second target pose transformation relation, which is the product between the rotation matrix of the initial attitude angle and the second target pose transformation relation. scale * tcw represents the second transformation value obtained according to the initial position information and the current scaling scale, which is the product between the initial position information and the current scaling scale.

[0059] It can be assumed that, based on the target pose transformation relationship obtained from the aforementioned steps and the current scaling scale, the real-time pose (initial pose) of the UAV in the relative coordinate system is transformed into the pose parameters (i.e., target pose) in the coordinate system of the oblique photogrammetry model.

[0060] In some embodiments, after S13 above, the UAV localization method further includes: projecting the current frame image onto an oblique photogrammetry model based on the target pose to obtain the projected edge of the current frame image; and determining the quality detection result of the target pose based on the comparison result between the texture information of the oblique photogrammetry model and the projected edge.

[0061] For example, based on the target pose of the UAV, the current frame image (image data) acquired corresponding to the target pose can be projected onto the oblique photogrammetry model to obtain the projected edge of the current frame image and the surface texture of the oblique photogrammetry model. The texture information of the oblique photogrammetry model is compared with the projected edge to obtain a comparison result; the quality detection result of whether the UAV's target pose is accurate is determined based on the comparison result. The comparison method may include, but is not limited to, matching the projected edge features with the texture features; if they match, the target pose is accurate; if they do not match, the target pose is inaccurate. Alternatively, the real-time projected edge and the texture of the 3D model can be displayed, and the alignment of the real-time projected edge and the texture of the 3D model can be determined by manual inspection. If they are not aligned, a comparison result sent by the user will be received, and the quality detection result of whether the target pose is accurate will be determined accordingly.

[0062] For example, the comparison result characterizes the alignment between the texture information of the oblique photogrammetry model and the projected edge. In response to the alignment degree in the comparison result being less than the alignment degree threshold, the quality detection result is determined to be that the accuracy of the target pose is lower than the accuracy threshold; in response to the alignment degree in the comparison result being greater than or equal to the alignment degree threshold, the quality detection result is determined to be that the accuracy of the target pose is greater than or equal to the accuracy threshold.

[0063] In some embodiments, after determining the quality detection result of the target pose based on the comparison result between the texture information of the oblique photogrammetry model and the projected edge, the UAV localization method further includes: receiving a second operation command in response to the accuracy of the quality detection result representing the target pose being lower than an accuracy threshold; acquiring a second acquired image captured by the UAV while it is operating according to the second operation command, and a candidate pose in a relative coordinate system when the second acquired image was acquired; updating the current scaling scale based on the second acquired image to obtain the target scaling scale; and transforming the candidate pose according to the target pose transformation relationship and the target scaling scale to obtain the final pose of the UAV.

[0064] The second operation command represents the operation command generated in response to the second operation. The second operation is the setting of the UAV's travel path and image acquisition frequency after initial mapping. The second acquired image is an image acquired by the UAV while traveling along the travel path corresponding to the second operation command. The travel path corresponding to the second operation command contains the location of at least one target object. The second acquired image includes the target object. The candidate pose represents the UAV's pose in a relative coordinate system when acquiring the second acquired image. The final pose represents the UAV's pose in a 3D coordinate system when acquiring the second acquired image.

[0065] The method for obtaining candidate poses is the same as that for obtaining initial poses. Specifically, the second acquired image is input into a preset map building algorithm to obtain a relative map output by the preset map building algorithm and the pose of the UAV in the coordinate system of the relative map when acquiring the second acquired image.

[0066] The target scaling scale represents the scaling ratio between the relative coordinate system of the relative map determined based on the second acquired image and the 3D coordinate system. If the accuracy of the quality detection result representing the target pose is lower than the accuracy threshold, it indicates that the previously constructed current scaling scale has a large deviation, requiring a re-updating of the scaling scale and a re-transformation to obtain the pose in the 3D coordinate system. Specifically, the target scaling scale can be constructed as follows: based on at least two edge feature points of the target object in the second acquired image, the target scaling scale is obtained, including: acquiring the true distance between each edge feature point and the image distance between the corresponding edge feature points in the second acquired image. The ratio between the true distance and the image distance is used as the target scaling scale.

[0067] The steps described above for transforming candidate poses based on the target pose transformation relationship and target scaling scale to obtain the final pose of the UAV may include the following: determining the attitude angles in the final pose based on the attitude angles in the candidate poses and the first target transformation relationship; determining the position information in the final pose based on the attitude angles in the candidate poses, the position information in the candidate poses, the target scaling scale, and the second target transformation relationship, including: obtaining a new first transformation value based on the attitude angles in the candidate poses and the second target pose transformation relationship; obtaining a new second transformation value based on the position information in the candidate poses and the target scaling scale; and using the sum of the new first transformation value and the new second transformation value as the position information in the final pose. It is understood that the specific processing logic for transforming candidate poses based on the target pose transformation relationship and target scaling scale to obtain the final pose of the UAV is the same as the specific processing logic of S13 described above, and will not be repeated here.

[0068] Understandably, this application does not rely on GPS signals or other sensor data, enabling the UAV to perform positioning in indoor, outdoor, and other location environments. This solves the positioning difficulty problem when there are no GPS signals or other sensor data, and improves the accuracy of UAV pose estimation. Furthermore, it does not require inertial sensor data to recover the trajectory scale; instead, this application recovers the scale using the actual distance to the target object and the image distance, thereby improving the accuracy of UAV positioning.

[0069] The above scheme obtains the initial pose of the UAV in the relative coordinate system when acquiring the current frame image; determines the current scaling scale between the relative coordinate system and the 3D coordinate system based on the target object in the current frame image; transforms the initial pose according to the obtained target pose transformation relationship between the relative coordinate system and the 3D coordinate system and the current scaling scale to obtain the target pose of the UAV in the 3D coordinate system. Thus, an accurate initial pose is obtained from the current frame image. The current scaling scale and the target pose transformation relationship represent information in different dimensions between the relative coordinate system and the 3D coordinate system. Furthermore, the initial pose is transformed in real time based on the comprehensive information in different dimensions between the relative coordinate system and the 3D coordinate system to improve the accuracy of the target pose in the 3D coordinate system obtained by transformation.

[0070] Please see Figure 4 , Figure 4 This is a schematic diagram of an embodiment of the UAV positioning device of this application. The UAV positioning device 40 includes an acquisition module 41, a determination module 42, and a conversion module 43; the acquisition module 41 is used to acquire the initial pose of the UAV in the relative coordinate system when acquiring the current frame image; the determination module 42 is used to determine the current scaling scale between the relative coordinate system and the 3D coordinate system based on the target object in the current frame image; the conversion module 43 is used to convert the initial pose according to the acquired target pose transformation relationship between the relative coordinate system and the 3D coordinate system and the current scaling scale to obtain the target pose of the UAV in the 3D coordinate system.

[0071] Please refer to the UAV positioning method for the functions performed by each module; they will not be repeated here.

[0072] The above scheme obtains the initial pose of the UAV in the relative coordinate system when acquiring the current frame image; determines the current scaling scale between the relative coordinate system and the 3D coordinate system based on the target object in the current frame image; transforms the initial pose according to the obtained target pose transformation relationship between the relative coordinate system and the 3D coordinate system and the current scaling scale to obtain the target pose of the UAV in the 3D coordinate system. Thus, an accurate initial pose is obtained from the current frame image. The current scaling scale and the target pose transformation relationship represent information in different dimensions between the relative coordinate system and the 3D coordinate system. Furthermore, the initial pose is transformed in real time based on the comprehensive information in different dimensions between the relative coordinate system and the 3D coordinate system to improve the accuracy of the target pose in the 3D coordinate system obtained by transformation.

[0073] Please see Figure 5 , Figure 5This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 50 includes a memory 51 and a processor 52. The processor 52 is used to execute program instructions stored in the memory 51 to implement the steps in the above-described UAV positioning method embodiment. In a specific implementation scenario, the electronic device 50 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 50 may also include mobile devices such as laptops and tablets, which are not limited here.

[0074] Specifically, processor 52 controls itself and memory 51 to implement the steps in the above-described UAV positioning method embodiments. Processor 52 can also be referred to as a CPU (Central Processing Unit). Processor 52 may be an integrated circuit chip with signal processing capabilities. Processor 52 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 52 can be implemented using integrated circuit chips.

[0075] The above scheme obtains the initial pose of the UAV in the relative coordinate system when acquiring the current frame image; determines the current scaling scale between the relative coordinate system and the 3D coordinate system based on the target object in the current frame image; transforms the initial pose according to the obtained target pose transformation relationship between the relative coordinate system and the 3D coordinate system and the current scaling scale to obtain the target pose of the UAV in the 3D coordinate system. Thus, an accurate initial pose is obtained from the current frame image. The current scaling scale and the target pose transformation relationship represent information in different dimensions between the relative coordinate system and the 3D coordinate system. Furthermore, the initial pose is transformed in real time based on the comprehensive information in different dimensions between the relative coordinate system and the 3D coordinate system to improve the accuracy of the target pose in the 3D coordinate system obtained by transformation.

[0076] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 60 stores program instructions 601 thereon, which, when executed by a processor, implement the steps in any of the above embodiments of the UAV positioning method.

[0077] The above scheme obtains the initial pose of the UAV in the relative coordinate system when acquiring the current frame image; determines the current scaling scale between the relative coordinate system and the 3D coordinate system based on the target object in the current frame image; transforms the initial pose according to the obtained target pose transformation relationship between the relative coordinate system and the 3D coordinate system and the current scaling scale to obtain the target pose of the UAV in the 3D coordinate system. Thus, an accurate initial pose is obtained from the current frame image. The current scaling scale and the target pose transformation relationship represent information in different dimensions between the relative coordinate system and the 3D coordinate system. Furthermore, the initial pose is transformed in real time based on the comprehensive information in different dimensions between the relative coordinate system and the 3D coordinate system to improve the accuracy of the target pose in the 3D coordinate system obtained by transformation.

[0078] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0079] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0080] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0081] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0082] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. A method for locating unmanned aerial vehicles (UAVs), characterized in that, The UAV positioning method includes: Obtain the initial pose of the UAV in the relative coordinate system when it acquires the current frame image; Based on the target object in the current frame image, determine the current scaling scale between the relative coordinate system and the 3D coordinate system; The initial pose is transformed based on the target pose transformation relationship between the relative coordinate system and the 3D coordinate system and the current scaling scale to obtain the target pose of the UAV in the 3D coordinate system.

2. The UAV positioning method according to claim 1, characterized in that, The initial pose includes initial attitude angles and initial position information. The target pose transformation relationship represents a first target transformation relationship with respect to attitude angles and a second target transformation relationship with respect to position information. The target pose includes target attitude angles and target position information. The step of transforming the initial pose based on the acquired target pose transformation relationship between the relative coordinate system and the 3D coordinate system and the current scaling scale to obtain the target pose of the UAV in the 3D coordinate system includes: The target attitude angle is determined based on the initial attitude angle and the first target transformation relationship; The target position information is determined based on the initial attitude angle, the initial position information, the current scaling scale, and the second target transformation relationship.

3. The UAV positioning method according to claim 2, characterized in that, The step of determining the target position information based on the initial attitude angle, the initial position information, the current scaling scale, and the second target transformation relationship includes: Based on the initial attitude angle and the second target pose transformation relationship, the first transformation value is obtained; The second transformation value is obtained based on the initial position information and the current scaling scale; The sum of the first transformation value and the second transformation value is used as the target location information.

4. The UAV positioning method according to claim 1, characterized in that, The step of determining the current scaling scale between the relative coordinate system and the 3D coordinate system based on the target object in the current frame image includes: In response to the fact that the current frame image does not contain the target object, the preset scaling scale is used as the current scaling scale; In response to the presence of a target object in the current frame image, the current scaling scale is obtained based on at least two edge feature points of the target object in the current frame image.

5. The UAV positioning method according to claim 4, characterized in that, The step of obtaining the current scaling scale based on at least two edge feature points of the target object in the current frame image includes: Obtain the true distance between each edge feature point and the image distance between each edge feature point in the current frame image; The ratio between the actual distance and the image distance is used as the current scaling scale.

6. The UAV positioning method according to claim 1, characterized in that, Before the step of transforming the initial pose according to the acquired target pose transformation relationship between the relative coordinate system and the 3D coordinate system and the current scaling scale to obtain the target pose of the UAV in the 3D coordinate system, the UAV localization method further includes: The initial mapping image and the pre-calibration image collected by the UAV are acquired, wherein the acquisition time of the initial mapping image is earlier than the acquisition time of the current frame image; Based on the pre-calibrated image, the initial mapping image, and the initial mapping pose of the UAV in the relative coordinate system when acquiring the initial mapping image, the first pose of the UAV in the relative coordinate system when acquiring the pre-calibrated image is obtained. Based on the second pose and the first pose of the UAV in the 3D coordinate system when acquiring the pre-calibrated image, the target pose transformation relationship is determined.

7. The UAV positioning method according to claim 6, characterized in that, The step of acquiring the initial mapping image collected by the UAV includes: In response to the drone receiving a first operation command, the drone acquires a first image captured while it is operating according to the first operation command. Image matching is performed between the first acquired image and the pre-calibrated image to obtain the image matching result; In response to the image matching result satisfying a preset condition, the first acquired image is used as the initial mapping image.

8. The UAV positioning method according to claim 1, characterized in that, After the step of transforming the initial pose according to the acquired target pose transformation relationship between the relative coordinate system and the 3D coordinate system and the current scaling scale to obtain the target pose of the UAV in the 3D coordinate system, the UAV localization method further includes: The current frame image is projected onto the oblique photography model according to the target pose to obtain the projection edge of the current frame image; The quality detection result of the target pose is determined based on the comparison between the texture information of the oblique photogrammetry model and the projected edge.

9. The UAV positioning method according to claim 8, characterized in that, After the step of determining the quality detection result of the target pose based on the comparison result between the texture information of the oblique photography model and the projected edge, the UAV localization method further includes: In response to the quality detection result indicating that the accuracy of the target pose is lower than the accuracy threshold, a second operation command is received; The second image acquired by the UAV during its flight according to the second operation command is obtained, along with the candidate pose in the relative coordinate system at the time of acquiring the second image. The second image includes the target object. The current scaling scale is updated based on the second acquired image to obtain the target scaling scale; The candidate poses are transformed according to the target pose transformation relationship and the target scaling scale to obtain the final pose of the UAV.

10. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the UAV positioning method as described in any one of claims 1-9.

11. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they are used to implement the UAV positioning method as described in any one of claims 1-9.

Images