Positioning method, device, storage medium and unmanned aerial vehicle

Abstract

The present specification discloses a positioning method, device, storage medium and unmanned aerial vehicle. In the method, a key frame is determined from an image collected by the unmanned aerial vehicle, and the key frame is subjected to image segmentation to obtain a plurality of sub-images. For each sub-image, actual depth information of the sub-image is determined according to a matching relationship between a current image collected by the unmanned aerial vehicle and the key frame, and initial depth information of the sub-image determined by a single-point TOF sensor located on the unmanned aerial vehicle. The current positioning information of the unmanned aerial vehicle is determined according to the actual depth information of each sub-image. In the method, the actual depth information of each sub-image in the key frame obtained by the single-point TOF sensor is different, which can describe the ground undulation information of the image collection area of the unmanned aerial vehicle. Therefore, the accuracy of determining the distance between the ground area and the unmanned aerial vehicle can be improved at low cost, and the positioning accuracy of the unmanned aerial vehicle can be improved.

Description

Technical Field

[0001] This specification relates to the field of autonomous driving, and in particular to a positioning method, device, storage medium, and unmanned aerial vehicle. Background Technology

[0002] In the field of autonomous driving, unmanned devices can determine their own pose using visual inertial odometry. The accuracy of this pose determination is closely related to the accuracy of the depth information.

[0003] Taking a drone as an example, when a drone is flying at high altitude, the ground can be assumed to be a plane at the same height. Then, the altitude information between the drone and the ground is obtained by ultrasonic sensors installed on the drone, and this altitude information is used as depth information. Finally, the drone's pose is determined based on visual images continuously captured by cameras installed on the drone and the depth information of each visual image.

[0004] However, assuming the ground to be a flat plane at the same height ignores the undulations of the ground, resulting in inaccurate distances between the drone and ground targets. This reduces the accuracy of depth information in each visual image, thereby reducing the drone's positioning accuracy. Summary of the Invention

[0005] This specification provides a positioning method, device, storage medium, and drone to partially solve the problems existing in the prior art.

[0006] The embodiments in this specification adopt the following technical solutions:

[0007] This specification provides a positioning method, including:

[0008] Keyframes are determined based on the image sequences captured by the drone;

[0009] The keyframes are segmented to obtain multiple sub-images;

[0010] For each sub-image, the actual depth information corresponding to the sub-image is determined based on the matching relationship between the current image captured by the UAV and the key frame, as well as the initial depth information corresponding to the sub-image determined by the single-point TOF sensor located on the UAV.

[0011] The current positioning information of the UAV is determined based on the actual depth information corresponding to each sub-image in the keyframe.

[0012] Optionally, based on the matching relationship between the current image captured by the UAV and the keyframe, and the initial depth information corresponding to the sub-image determined by the single-point TOF sensor located on the UAV, the actual depth information corresponding to the sub-image is determined, specifically including:

[0013] The current image captured by the UAV and the initial depth information corresponding to the sub-image determined by the single-point TOF sensor on the UAV when the UAV captures the key frame are obtained, and used as the initial depth information corresponding to the sub-image determined under the key frame;

[0014] Based on the matching relationship between the acquired current image and the keyframe, the initial depth information corresponding to the sub-image determined under the keyframe is updated to obtain the actual depth information corresponding to the sub-image.

[0015] Optionally, based on the matching relationship between the acquired current image and the keyframe, the initial depth information corresponding to the sub-image determined under the keyframe is updated, specifically including:

[0016] Determine the matching degree between the currently acquired image and the keyframe;

[0017] If the matching degree is greater than a preset threshold, the current image will not be re-determined as a key frame, and the initial depth information corresponding to the sub-image determined under the key frame will be updated according to the depth information observed by the single-point TOF sensor when the UAV collects the current image.

[0018] If the matching degree is not greater than a preset threshold, the current image is redefined as a keyframe, and for each sub-image in the current image that has been redefined as a keyframe, the initial depth information corresponding to the sub-image determined under the current image is updated.

[0019] Optionally, the initial depth information corresponding to the sub-image is determined, specifically including:

[0020] If the current image is the keyframe, determine the historical keyframe, which refers to the keyframe determined before the UAV acquired the current image;

[0021] For each sub-image in the current image, determine whether the sub-image has a spatial mapping relationship with the historical keyframe;

[0022] If the sub-image has a spatial mapping relationship with the historical keyframe, the initial depth information corresponding to the sub-image is determined based on the specified sub-image in the historical keyframe that has a spatial mapping relationship with the sub-image and the depth information corresponding to the specified sub-image.

[0023] Optionally, the method further includes:

[0024] If the sub-image does not have a spatial mapping relationship with the historical keyframe, the depth information observed by the single-point TOF sensor when the UAV acquires the current image is determined by the single-point TOF sensor located on the UAV, and used as the initial depth information corresponding to the sub-image.

[0025] Optionally, based on the depth information observed by the single-point TOF sensor when the UAV acquires the current image, the initial depth information corresponding to the sub-image determined in the keyframe is updated, specifically including:

[0026] If the current image is not the keyframe, determine the spatial mapping relationship between the keyframe and the current image based on the pose of the UAV when acquiring the keyframe and the pose of the UAV when acquiring the current image.

[0027] Based on the spatial mapping relationship and the target depth information observed by the single-point TOF sensor when the UAV acquires the current image, a sub-image corresponding to the region where the UAV observes the target depth information in the keyframe is determined as the target sub-image, and the depth information transformed from the target depth information to the target sub-image is determined as the updated depth information;

[0028] The initial depth information corresponding to the target sub-image is updated based on the updated depth information.

[0029] Optionally, the initial depth information corresponding to the target sub-image is updated according to the updated depth information, specifically including:

[0030] Determine the reliability of the initial depth information corresponding to the target sub-image and the reliability of the target depth information observed by the single-point TOF sensor;

[0031] Based on the credibility of the initial depth information and the credibility of the target depth information, a credibility coefficient between the initial depth information and the target depth information corresponding to the target sub-image is determined;

[0032] The initial depth information corresponding to the target sub-image is updated based on the initial depth information, the updated depth information, and the confidence coefficient.

[0033] Optionally, the current positioning information of the UAV is determined based on the actual depth information corresponding to each sub-image in the keyframe, specifically including:

[0034] Determine the projected pixel point of the UAV vertically projected onto the key frame at the current moment, and determine the sub-image associated with the projected pixel point in the key frame as the positioning sub-image;

[0035] Based on the actual depth information corresponding to each sub-image in the keyframe, determine the actual depth information corresponding to the center pixel of the positioning sub-image;

[0036] The actual depth information corresponding to the projected pixel is determined based on the actual depth information corresponding to the center pixel of the positioning sub-image.

[0037] The current positioning information of the drone is determined based on the actual depth information corresponding to the projected pixels.

[0038] This specification provides a positioning device, comprising:

[0039] The keyframe determination module is used to determine keyframes based on the image sequence captured by the UAV.

[0040] The image segmentation module is used to segment the keyframe to obtain multiple sub-images;

[0041] The depth information determination module is used to determine the actual depth information of each sub-image based on the matching relationship between the current image captured by the UAV and the key frame, and the initial depth information of the sub-image determined by the single-point TOF sensor located on the UAV.

[0042] The positioning module is used to determine the current positioning information of the UAV based on the actual depth information corresponding to each sub-image in the keyframe.

[0043] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned positioning method.

[0044] This specification provides a drone, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described positioning method.

[0045] The above-described at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects:

[0046] In this embodiment, keyframes are determined from images captured by the UAV. Then, the keyframes are segmented to obtain multiple sub-images. For each sub-image, the actual depth information is determined based on the matching relationship between the current image captured by the UAV and the keyframes, as well as the initial depth information corresponding to that sub-image determined by a single-point TOF sensor located on the UAV. Based on the actual depth information of each sub-image, the current positioning information of the UAV is determined. In this method, each sub-image in the keyframes acquired by the single-point TOF sensor corresponds to different actual depth information, which can describe the ground undulation information of the UAV image acquisition area. Therefore, at a low cost, the accuracy of determining the distance between the ground area and the UAV can be improved, and the positioning accuracy of the UAV can be improved. Attached Figure Description

[0047] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0048] Figure 1 This is a schematic flowchart of the positioning method provided in the embodiments of this specification;

[0049] Figure 2 This is a schematic diagram of keyframe segmentation provided for an embodiment of this specification;

[0050] Figure 3 A schematic diagram illustrating the determination of a designated sub-image as provided in the embodiments of this specification;

[0051] Figure 4 A schematic diagram illustrating the determination of a target sub-image as provided in the embodiments of this specification;

[0052] Figure 5 This is a schematic diagram of the positioning device structure provided in the embodiments of this specification;

[0053] Figure 6 This is a structural schematic diagram of the UAV provided in the embodiments of this specification. Detailed Implementation

[0054] The positioning method provided in this specification aims to acquire depth information corresponding to each sub-image in a keyframe image within a visual inertial odometry system using a depth sensor. Then, based on the current image acquired by the unmanned device and the keyframe image, the current positioning information of the unmanned device is determined. The keyframe image is equivalent to a grid map.

[0055] Visual inertial odometry (VIO) systems can include: image acquisition devices (e.g., cameras), IMUs, and visual odometry algorithms. The principle of visual odometry is as follows: After the image acquisition device on the unmanned device acquires an image sequence, it performs feature point matching on adjacent frames. Based on the matching results, the pose change of the image acquisition device when acquiring adjacent frames is determined, thus determining the pose transformation of the unmanned device when acquiring adjacent frames. In other words, when the currently acquired image is present in two adjacent frames, the position and orientation of the unmanned device when acquiring the current frame can be determined.

[0056] After identifying matching feature points in two adjacent image frames, an IMU (Integrated Measurement Unit) can predict the trajectory of the unmanned aerial vehicle (UAV) during the acquisition of those two frames. However, when the UAV is a drone, the IMU cannot accurately predict its trajectory. Therefore, a depth sensor (such as a depth camera or Time-of-Flight (TOF) sensor) can be used to obtain the distance between the UAV and the target point during the acquisition of two adjacent image frames—that is, the image depth information. Based on the image depth corresponding to two adjacent image frames, the IMU is assisted in predicting the UAV's trajectory during the acquisition of those frames. The UAV's localization result can then be determined based on its trajectory.

[0057] Therefore, the image depth information collected by unmanned equipment will affect the visual positioning effect of the unmanned equipment. The more accurate the image depth, the more accurate the visual positioning result of the unmanned equipment.

[0058] In this specification, a raster map can be constructed based on keyframe images acquired by an unmanned device and the areas covered by those images. Each raster stores image depth information and its corresponding confidence level; a keyframe image is equivalent to a raster map. The Kalman filter algorithm can be used to update the depth information and confidence level of each raster in the keyframe image and predict the depth information and confidence level of each raster in a new keyframe image. The process of updating the depth information and confidence level of each raster in the keyframe image is the process of constructing the raster map; the process of predicting the depth information and confidence level of each raster in a new keyframe image is the process of constructing multiple raster maps.

[0059] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0060] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0061] Figure 1 A schematic flowchart of the positioning method provided in the embodiments of this specification includes:

[0062] S100: Determine keyframes based on the image sequence captured by the drone.

[0063] In the embodiments of this specification, image acquisition devices, Time-of-Flight (ToF) sensors, etc., can be installed on unmanned equipment, and the image acquisition direction of the image acquisition device is consistent with the direction of the beam emitted by the ToF sensor. The unmanned equipment can include drones and unmanned vehicles. Unmanned equipment can be used in the logistics and delivery field, including both instant delivery (such as food delivery and general delivery) and other non-instant delivery fields. The image acquisition device can be a monocular camera. The ToF sensor can include: an area array ToF sensor and a single-point ToF sensor. An area array ToF sensor can emit multiple beams simultaneously, while a single-point ToF sensor can only emit one beam. Furthermore, the point illuminated by the beam emitted by the ToF sensor is called the target point.

[0064] At the same power, while an area-array ToF sensor can acquire different depth information simultaneously by emitting multiple light beams, compared to a single-point ToF sensor, an area-array ToF sensor has several drawbacks. The propagation distance of multiple beams is shorter than that of a single beam, and the intensity of multiple beams is weaker. Furthermore, multiple beams are more susceptible to the influence of outdoor light than a single beam. Additionally, area-array ToF sensors are heavier than single-point ToF sensors, increasing the power consumption of unmanned equipment, and are also more expensive.

[0065] Therefore, in this specification, to maximize the propagation distance of the light beam emitted by the ToF sensor and minimize its susceptibility to outdoor light, while also saving costs, a single-point ToF sensor can be installed on the unmanned device. Next, using a drone as an example, [the following section will discuss...] Figure 1 The positioning method shown will be explained.

[0066] In the embodiments of this specification, images within a preset range of the unmanned device can be acquired in real time using an image acquisition device located on the unmanned device. Simultaneously, depth information observed by the unmanned device while acquiring images within the preset range is obtained using a depth sensor located on the unmanned device. Then, an image can be randomly selected from the image sequence acquired in real time by the unmanned device as a keyframe; for example, the currently acquired image can be used as the keyframe. Alternatively, keyframes can be determined according to a keyframe selection strategy. These keyframes can be used to construct a raster map.

[0067] When determining keyframes based on the keyframe selection strategy, the first image captured after the unmanned device (UAV) is powered on can be directly used as the keyframe. Then, during the UAV's movement, the image captured at the current moment is used as the current image. The similarity between the current image and previously captured keyframes is determined. Based on this similarity, it is decided whether to re-identify the current image as a keyframe. If the similarity is greater than a preset threshold, the current image is not re-identified as a keyframe; if the similarity is not greater than the preset threshold, the current image is re-identified as a keyframe.

[0068] When determining the similarity between the current image and historically acquired keyframes, feature point matching can be used, or the current image and keyframes can be converted into vectors, and the similarity between the vector corresponding to the current image and the vector corresponding to the keyframes can be directly calculated. It should be noted that this specification does not restrict the method used to determine the similarity between the current image and keyframes. Figure 1 The positioning method shown can be applied to unmanned equipment or to servers that control unmanned equipment.

[0069] S102: Perform image segmentation on the keyframe to obtain multiple sub-images.

[0070] In the embodiments of this specification, after determining the keyframe, the keyframe can be segmented according to its image size to obtain multiple sub-images.

[0071] Specifically, keyframes can be divided into multiple grids using a fixed-size grid. For example... Figure 2 As shown. In Figure 2 In this example, the keyframe image size is 9×9, and the fixed grid size is 3×3. Thus, the keyframe contains sub-images 1 to 9.

[0072] The reason for segmenting keyframes is as follows: Firstly, when the depth sensor on the unmanned aerial vehicle (UAV) is a Time-of-Flight (ToF) sensor, the UAV simultaneously observes the distances (i.e., depth information) between itself and multiple target points during flight, allowing for feature point matching between adjacent frames. To improve matching accuracy, each frame can be divided into multiple sub-images. The UAV can move according to the regions corresponding to these sub-images, with the ToF sensor observing multiple depth information points during movement. Theoretically, the more sub-images in a keyframe image, the smaller the region corresponding to each sub-image, and the smaller the distance the UAV moves. This results in less movement of feature points in each frame, thus improving the matching accuracy between feature points in adjacent frames.

[0073] The second scenario: When the depth sensor on the unmanned device is a single-point ToF sensor, since a single-point ToF sensor can only emit one beam, the unmanned device can only observe the distance between itself and a target point if it is positioned directly above that target point. If the image acquisition area corresponding to the keyframe image is large enough, the unmanned device cannot observe every target point within that area. Therefore, the keyframe image can be segmented, assuming that all target points within each sub-image's area are at the same distance from the unmanned device. Furthermore, the depth information for each sub-image is independent of the others.

[0074] S104: For each sub-image, determine the actual depth information corresponding to the sub-image based on the matching relationship between the current image acquired by the UAV and the keyframe, and the initial depth information corresponding to the sub-image determined by the single-point TOF sensor located on the UAV.

[0075] In the embodiments of this specification, after obtaining each sub-image in the keyframe, the current image (i.e., the current frame) captured by the unmanned device at the current moment can be obtained, as well as the initial depth information corresponding to each sub-image observed by the depth sensor when the unmanned device captures the keyframe, which serves as the initial depth information corresponding to each sub-image determined under the keyframe. Then, based on the matching relationship between the current image and the keyframe, the initial depth information corresponding to each sub-image determined under the keyframe is updated to obtain the actual depth information corresponding to each sub-image. The matching relationship between the current image and the keyframe can be determined based on the matching degree between the current image and the keyframe.

[0076] During the process of updating the initial depth information corresponding to each sub-image in a keyframe, the keyframe may be changed. Specifically, the decision to change the keyframe can be based on the matching degree between the current image and the keyframe. If the matching degree is greater than a preset threshold, the current image is not re-identified as a keyframe; if the matching degree is not greater than the preset threshold, the current image is re-identified as a keyframe. The matching degree between the current image and the keyframe can be the similarity between the two; the greater the similarity, the greater the matching degree.

[0077] Next, we will explain the process of updating the initial depth information corresponding to each sub-image in the keyframe and the replacement of keyframes during the update process.

[0078] In the embodiments of this specification, after obtaining each sub-image in the keyframe, for each sub-image, the initial depth information corresponding to that sub-image can be determined first. Then, based on the depth information of the sub-image observed by the depth sensor when the unmanned device is located in the area corresponding to that sub-image, the initial depth information corresponding to that sub-image is updated. That is, based on the depth information of the sub-image observed by the depth sensor when the unmanned device is currently acquiring the current image, the initial depth information corresponding to that sub-image is updated. The updated depth information corresponding to that sub-image is re-determined as the initial depth information until the keyframe is re-determined from the image sequence acquired by the unmanned device. Before re-determining the keyframe from the image sequence acquired by the unmanned device, the latest updated depth information of the sub-image is used as the actual depth information corresponding to that sub-image. The actual depth information corresponding to each sub-image can represent the average distance between the area corresponding to each sub-image and the unmanned device. The actual depth information corresponding to each sub-image in the keyframe can also be the depth information corresponding to the center pixel of each sub-image.

[0079] Specifically, when the keyframe is the first image acquired after the unmanned device (UAV) is powered on (the first image acquired by the UAV is taken as the initial image, hereinafter referred to as the initial image), the depth information observed by the depth sensor when the UAV acquired the initial image can be directly used as the initial depth information for each sub-image. Then, as the UAV moves within the region corresponding to each sub-image in the initial image, the current image acquired by the UAV at the current moment is obtained. Based on the depth information observed by the depth sensor when the UAV acquired the current image, the initial depth information of the corresponding sub-image in the initial image is updated, and the updated depth information is used again as the initial depth information until the current image is used as the keyframe again. Then, for each sub-image in the initial image, the latest updated depth information is used as the actual depth information corresponding to that sub-image. In this way, after the current image is used as the keyframe again, the initial images with actual depth information constitute a raster map.

[0080] Furthermore, after re-identifying the current image as a keyframe, the re-identified keyframe can be segmented into multiple sub-images. Then, the initial depth information of each sub-image is determined, and based on the real-time depth information observed by the depth sensor of the unmanned device, the initial depth information of each sub-image is updated to obtain the actual depth information of each sub-image. In other words, the above process is repeated.

[0081] For keyframes, the initial depth information of each sub-image in the keyframe after each update can be weighted and averaged with the depth information observed by the depth sensor each time to obtain new initial depth information.

[0082] For example, take a sub-image from a keyframe as the first sub-image. If the initial depth information of the first sub-image after its first update is 6 meters, and the depth information observed by the depth sensor at this time is 7 meters, then the initial depth information of the first sub-image after its second update should be 6.5 meters.

[0083] In addition, for each sub-image in the keyframe, the initial depth information of the sub-image after each update can be Kalman filtered with the depth information observed by the depth sensor each time to obtain new initial depth information.

[0084] Furthermore, for each sub-image in the keyframe, the confidence level corresponding to the initial depth information of that sub-image is determined, and this confidence level is updated as the initial depth information changes. The confidence level corresponding to the initial depth information indicates whether the initial depth information corresponding to that sub-image is reliable. The confidence level corresponding to the initial depth information of each sub-image can be the covariance in the Kalman filtering algorithm.

[0085] The reason for continuously updating the depth information of each sub-image in the keyframe is twofold: First, to improve the accuracy of the depth information corresponding to each sub-image, thereby improving the positioning accuracy of the unmanned device. Second, when the depth sensor on the unmanned device is a single-point ToF sensor, the depth information observed by the unmanned device in the same sub-image region is also different. Therefore, it is necessary to perform weighted averaging or Kalman filtering on the depth information corresponding to the same sub-image.

[0086] S106: Determine the current positioning information of the UAV based on the depth information corresponding to each sub-image in the keyframe.

[0087] In the embodiments of this specification, the current image captured by the unmanned device at the current moment and the keyframes before the unmanned device captured the current image are acquired, and the current image and the keyframes are matched. Based on the matching result, the feature points (pixels) that match in the current image and the keyframes are determined as matching feature points. The sub-images corresponding to the matching feature points in the keyframes are determined as matching sub-images. Based on the actual depth information corresponding to the center pixel of the matching sub-image, the actual depth information corresponding to the matching feature points is determined. Based on the actual depth information corresponding to the matching feature points, the pose of the unmanned device when capturing the keyframes, and the current image, the pose of the unmanned device predicted by the visual inertial odometry system when capturing the current image is corrected to determine the true pose of the unmanned device when capturing the current image. That is, the current positioning information of the unmanned device is determined. The positioning information may include: position, attitude, velocity, acceleration, etc.

[0088] When determining the actual depth information corresponding to the matching feature point, you can directly use the actual depth information corresponding to the center pixel of the matching sub-image as the actual depth information corresponding to the matching feature point, or you can determine the actual depth information corresponding to the matching feature point based on the actual depth information of the center pixel of the matching sub-image and the actual depth information corresponding to the center pixels of other sub-images adjacent to the matching sub-image.

[0089] Furthermore, when locating unmanned equipment, the projected pixels of the unmanned equipment vertically projected onto the keyframe when the current image is captured can be determined, as well as the sub-images associated with the projected pixels in the keyframe, which serve as positioning sub-images. Based on the actual depth information corresponding to the center pixel of each positioning sub-image, the actual depth information corresponding to the projected pixels is determined. Based on the actual depth information corresponding to the projected pixels, the current positioning information of the UAV is determined. The positioning sub-images can be other sub-images adjacent to the sub-image where the projected pixels are located, as well as the sub-image where the projected pixels are located, and there can be multiple positioning sub-images.

[0090] When determining the actual depth information corresponding to the projected pixel, the actual depth information corresponding to the center pixel of each positioning sub-image is weighted and summed based on the pixel distance between the projected pixel and the positioning sub-image to obtain the actual depth information corresponding to the projected pixel.

[0091] exist Figure 2 If the projected pixel is located in sub-image 4, the selected positioning sub-images associated with the projected pixel are: sub-image 1, sub-image 2, sub-image 4, sub-image 5, sub-image 7, and sub-image 8. The actual depth information corresponding to the positioning sub-images is obtained as follows: 4 meters, 5 meters, 5 meters, 5 meters, 6 meters, 7 meters, and 4 meters. Based on the distance between the center pixel of the positioning sub-image and the projected pixel, the weights corresponding to the actual depth information of the positioning sub-images are determined as: 0.01, 0.01, 0.3, 0.25, 0.25, and 0.18, respectively. Thus, the actual depth information corresponding to the projected pixel is 5.56 meters.

[0092] In addition, a three-dimensional map can be constructed based on the keyframe images collected during the movement of the unmanned equipment and the actual depth information corresponding to each sub-image in each keyframe image. This three-dimensional map can be used for the positioning of the unmanned equipment.

[0093] Through the above Figure 1As shown in the method, this specification determines keyframes from images captured by the UAV, then segments the keyframes to obtain multiple sub-images. For each sub-image, the actual depth information of the sub-image is determined based on its initial depth information and the depth information observed when the UAV is located in the corresponding region. The current positioning information of the UAV is then determined based on the actual depth information of each sub-image. In this method, each sub-image in the keyframe corresponds to different actual depth information, which can describe the ground undulation information of the UAV image acquisition area, thereby improving the accuracy of determining the distance between the ground area and the UAV, and improving the positioning accuracy of the UAV.

[0094] Furthermore, in such Figure 1 In step S104, as shown, during the movement of the unmanned device, it can acquire images within a preset range in real time. Simultaneously, it uses a depth sensor to observe the distance between the target point within the area where the unmanned device is located and the device itself. Therefore, during movement, the unmanned device can update keyframes and the depth information corresponding to each sub-image within the keyframes.

[0095] For keyframes, they can be images captured historically by the unmanned equipment (UAV) or images captured at the current moment. When determining the initial depth information for each sub-image within a keyframe, the depth information observed by the depth sensor when the UAV captured the keyframe can be used as the initial depth information for all sub-images. Alternatively, the actual depth information for each sub-image in a historical keyframe can be projected onto the current keyframe as the initial depth information for each sub-image in the current keyframe. Then, the depth information for each sub-image within the keyframe is updated based on images captured by the UAV after the keyframe capture and the depth information observed by the depth sensor when capturing images after the keyframe capture.

[0096] Next, focusing on the current image captured by the unmanned device, we first determine whether the current image is a keyframe. If not, we update the initial depth information of each sub-image in the keyframe based on the current image to obtain the actual depth information of each sub-image in the keyframe. If it is a keyframe, we first determine the initial depth information of each sub-image in the current image, and then update the initial depth information of each sub-image in the current image based on subsequently captured images. Specifically, we can determine whether the current image is a keyframe based on the similarity between the current image and previously captured keyframes.

[0097] If the current image is a keyframe, historical keyframes can be determined first. Historical keyframes refer to keyframes determined before the UAV acquired the current image. Then, for each sub-image in the current image, it is determined whether the sub-image has a spatial mapping relationship with a historical keyframe. If the sub-image has a spatial mapping relationship with a keyframe, the initial depth information corresponding to the sub-image is determined based on a specified sub-image in the historical keyframes that has a spatial mapping relationship with the sub-image and the corresponding depth information of the specified sub-image. If the sub-image does not have a spatial mapping relationship with a historical keyframe, the depth information observed by the depth sensor located on the UAV when the current image was acquired is determined as the initial depth information corresponding to the sub-image.

[0098] Specifically, when determining whether a sub-image has a spatial mapping relationship with historical keyframes, for each sub-image in the current image, the pose transformation between the historical keyframes acquired by the unmanned device and the current image acquisition can be used to project the sub-image in the current image onto the keyframes to obtain projection points. If no projection point exists in the keyframe, it is determined that the sub-image does not have a spatial mapping relationship with the historical keyframes. If a projection point exists in the keyframe, it is determined that the sub-image has a spatial mapping relationship with the historical keyframes. Furthermore, the sub-images in the historical keyframes that have a spatial mapping relationship with the sub-image are designated as the specified sub-images.

[0099] When determining whether a sub-image has a spatial mapping relationship with a historical keyframe, the 3D information of the region corresponding to each sub-image in the historical keyframe can be determined based on the pose of the unmanned device when acquiring the historical keyframe and the actual depth information corresponding to each sub-image in the historical keyframe. Then, based on the pose transformation between the unmanned device acquiring the historical keyframe and acquiring the current image, each sub-image in the keyframe is projected onto the current image, and the sub-images with projection points in the current image are determined as projected sub-images. If there are no projection points in the current image, it is determined that the current image does not have a spatial mapping relationship with the historical keyframe. If there are projection points in the current image, it is determined that the current image has a spatial mapping relationship with the historical keyframe. Furthermore, the sub-images in the historical keyframe that have a spatial mapping relationship with the projected sub-images are designated images. Figure 3 As shown. In Figure 3 In the historical keyframes, there are three designated sub-images that have a spatial mapping relationship with the projected sub-images in the current image.

[0100] In addition, when determining the initial depth information of a sub-image based on a specified sub-image that has a spatial mapping relationship with the sub-image in the historical keyframes and the depth information corresponding to the specified sub-image, the depth information corresponding to the specified sub-image can be converted to the sub-image through the spatial mapping relationship between the specified sub-image and the sub-image to obtain the converted depth information, which is used as the initial depth information of the sub-image.

[0101] When a sub-image has multiple projection points, the converted depth information in the sub-image can be weighted and averaged to obtain the final initial depth information of the sub-image.

[0102] In addition, by specifying the spatial mapping relationship between the sub-image and the sub-image, the confidence level corresponding to the depth information of the specified sub-image can be transferred to the sub-image to obtain the confidence level corresponding to the initial depth information of the sub-image.

[0103] Specifically, the initial depth information corresponding to the sub-image in the current image that has a spatial mapping relationship with the specified sub-image can be determined through the prediction process in the Kalman filter algorithm, as well as the reliability of the initial depth information of the sub-image in the current image.

[0104] Specifically, the prediction process in the Kalman filter algorithm is as follows: Representation of the region corresponding to the specified sub-image c0 in the historical keyframes: ∑a=diag([σ a , σ a , σ a ]).in, Represented as a Gaussian distribution, This represents the 3D information of the region corresponding to a specified image c0 in the historical keyframes. ∑a is Gaussian white noise, representing the confidence level of the actual depth information corresponding to the specified sub-image c0 in the historical keyframes. σ a Calculate experience points.

[0105] Pose transformation between a specified sub-image c0 in the historical keyframes and the projected image c1 corresponding to the specified sub-image c0 in the current image: ∑g=diag([σ g , σ g , σ g ].in, Let represent the pose of the projected image c1. R represents the rotation matrix, assumed to be orthogonal. t represents the translation amount, and n(0, ∑g) represents the coordinate transformation noise. σ g Calculate experience points.

[0106] After projecting the specified sub-image c0 onto the projected image c1, the projected image c1 follows a Gaussian distribution: ∑a′=diag([σ a , σa , σ a ])+diag([σ g , σ g , σ g ] Here, ∑a′ represents the superposition of Gaussian white noise and coordinate transformation noise, and represents the confidence level of the initial depth information of the projected sub-image c1 in the current image.

[0107] If the current image is not a keyframe, the spatial mapping relationship between the keyframe and the current image is determined based on the pose of the unmanned device when acquiring the keyframe and the pose predicted by the visual inertial odometry system when the unmanned device acquires the current image. Based on the spatial mapping relationship and the target depth information currently observed by the depth sensor, a sub-image corresponding to the region in the keyframe where the target depth information observed by the unmanned device is located is determined as the target sub-image. The depth information to be converted from the target depth information to the target sub-image is then determined as the updated depth information. Based on the updated depth information, the initial depth information corresponding to the target sub-image is updated to obtain the actual depth information corresponding to the target sub-image. For example... Figure 4 As shown.

[0108] exist Figure 4 For greater clarity, we assume the current image is segmented into sub-images 1-9, even though the actual current image has not been segmented. The keyframe is segmented into sub-images 1-9. The target depth information observed by the unmanned device in the region of sub-image 5 within the current image corresponds to sub-image 6 in the keyframe, which is then used as the target sub-image. The initial depth information of sub-image 6 in the keyframe is updated based on the updated depth information derived from the target depth information to obtain the actual depth information.

[0109] When updating the initial depth information corresponding to the target sub-image, the reliability of the initial depth information corresponding to the target sub-image and the reliability of the target depth information observed by the depth sensor are determined. Based on the reliability of the initial depth information and the reliability of the target depth information, a reliability coefficient between the initial depth information and the target depth information corresponding to the target sub-image is determined. Based on the initial depth information, updated depth information, and reliability coefficient of the target sub-image, the initial depth information corresponding to the target sub-image is updated.

[0110] Specifically, the initial depth information of each sub-image in the keyframe can be updated through the update process in the Kalman filter algorithm, and the reliability of the initial depth information can also be updated.

[0111] The update process in the Kalman filter algorithm: Assume that when the unmanned device acquires the current image, the target depth information observed by the depth sensor (single-point ToF sensor) is converted into the updated depth information d in the target sub-image of the keyframe. tofBecause the center pixel of each sub-image was observed (c i c j The corresponding depth information is used, therefore, the observation matrix in the Kalman filter algorithm is: Then, determine the confidence coefficient between the initial depth information and the target depth information corresponding to the target sub-image: K = (HP) / (HPH) T +∑tof). Where K represents the confidence coefficient. P represents the confidence level of the initial depth information of the target sub-image, and ∑tof represents the confidence level of the target depth information observed by the depth sensor.

[0112] Update the initial depth information and its corresponding confidence level: X new =X+K(d tof -X), P new =P(I-KH). Where X represents the initial depth information of the target sub-image, and I represents the identity matrix.

[0113] The above are the positioning methods provided in the embodiments of this specification. Based on the same idea, this specification also provides corresponding devices, storage media and drones.

[0114] Figure 5 This is a schematic diagram of a positioning device provided in an embodiment of this specification. The device includes:

[0115] The keyframe determination module 501 is used to determine keyframes based on the image sequence acquired by the UAV;

[0116] Image segmentation module 502 is used to segment the keyframe to obtain multiple sub-images;

[0117] The depth information determination module 503 is used to determine the actual depth information of each sub-image based on the matching relationship between the current image collected by the UAV and the key frame, and the initial depth information of the sub-image determined by the single-point TOF sensor located on the UAV.

[0118] The positioning module 504 is used to determine the current positioning information of the UAV based on the actual depth information corresponding to each sub-image in the key frame.

[0119] Optionally, the depth information determination module 503 is specifically used to: acquire the current image collected in real time by the UAV and the initial depth information corresponding to the sub-image determined by the single-point TOF sensor located on the UAV when the UAV collects the key frame, as the initial depth information corresponding to the sub-image determined under the key frame; update the initial depth information corresponding to the sub-image determined under the key frame according to the matching relationship between the current image collected in real time and the key frame, and obtain the actual depth information corresponding to the sub-image.

[0120] Optionally, the depth information determination module 503 is specifically used to: determine the matching degree between the acquired current image and the key frame; if the matching degree is greater than a preset threshold, not to re-determine the current image as a key frame, and to update the initial depth information corresponding to the sub-image determined under the key frame based on the depth information observed by the single-point TOF sensor when the UAV acquires the current image; if the matching degree is not greater than the preset threshold, to re-determine the current image as a key frame, and to update the initial depth information corresponding to the sub-image determined under the current image for each sub-image in the re-determined key frame.

[0121] Optionally, the depth information determination module 503 is specifically used to: if the current image is the keyframe, determine historical keyframes, where the historical keyframes refer to keyframes determined before the UAV acquired the current image; for each sub-image in the current image, determine whether the sub-image has a spatial mapping relationship with the historical keyframes; if the sub-image has a spatial mapping relationship with the historical keyframes, determine the initial depth information corresponding to the sub-image based on the specified sub-image in the historical keyframes that has a spatial mapping relationship with the sub-image and the depth information corresponding to the specified sub-image.

[0122] Optionally, the depth information determination module 503 is further configured to, if the sub-image does not have a spatial mapping relationship with the historical keyframe, determine the depth information observed by the single-point TOF sensor when the UAV acquires the current image through the single-point TOF sensor located on the UAV, and use it as the initial depth information corresponding to the sub-image.

[0123] Optionally, the depth information determination module 503 is specifically configured to: if the current image is not the keyframe, determine the spatial mapping relationship between the keyframe and the current image based on the pose of the UAV when acquiring the keyframe and the predicted pose of the UAV when acquiring the current image; determine the sub-image in the keyframe corresponding to the region where the UAV observes the target depth information based on the spatial mapping relationship and the target depth information observed by the single-point TOF sensor when the UAV acquires the current image, as the target sub-image; and determine the depth information to be converted from the target depth information onto the target sub-image, as the updated depth information; and update the initial depth information corresponding to the target sub-image based on the updated depth information.

[0124] Optionally, the depth information determination module 503 is specifically used to: determine the credibility of the initial depth information corresponding to the target sub-image and the credibility of the target depth information observed by the single-point TOF sensor; determine a credibility coefficient between the initial depth information corresponding to the target sub-image and the target depth information based on the credibility of the initial depth information and the credibility of the target depth information; and update the initial depth information corresponding to the target sub-image based on the initial depth information, the updated depth information, and the credibility coefficient.

[0125] Optionally, the positioning module 504 is specifically configured to: determine the projected pixel point of the UAV vertically projected onto the keyframe at the current moment, and determine the sub-image associated with the projected pixel point in the keyframe as the positioning sub-image; determine the actual depth information corresponding to the center pixel point of the positioning sub-image based on the actual depth information corresponding to each sub-image in the keyframe; determine the actual depth information corresponding to the projected pixel point based on the actual depth information corresponding to the center pixel point of the positioning sub-image; and determine the current positioning information of the UAV based on the actual depth information corresponding to the projected pixel point.

[0126] This specification also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can be used to perform the above-described actions. Figure 1 The provided positioning method.

[0127] based on Figure 1 The positioning method shown in this specification is further provided in the embodiments. Figure 6 The diagram shows the structure of the drone. Figure 6At the hardware level, the drone includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for its operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then executes it to achieve the above-mentioned functions. Figure 1 The aforementioned positioning method.

[0128] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0129] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0130] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0131] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0132] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0133] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0134] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0135] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0136] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0137] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0138] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0139] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0140] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0141] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0143] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0144] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A positioning method, characterized in that, include: Keyframes are determined based on the image sequences captured by the drone; The keyframes are segmented to obtain multiple sub-images; For each sub-image, the actual depth information corresponding to the sub-image is determined based on the matching relationship between the current image captured by the UAV and the key frame, as well as the initial depth information corresponding to the sub-image determined by the single-point TOF sensor located on the UAV. The current positioning information of the UAV is determined based on the actual depth information corresponding to each sub-image in the keyframe; Specifically, determining the current positioning information of the UAV based on the actual depth information corresponding to each sub-image in the keyframe includes: Determine the projected pixel point of the UAV vertically projected onto the key frame at the current moment, and determine the sub-image associated with the projected pixel point in the key frame as the positioning sub-image; Based on the actual depth information corresponding to each sub-image in the keyframe, determine the actual depth information corresponding to the center pixel of the positioning sub-image; The actual depth information corresponding to the projected pixel is determined based on the actual depth information corresponding to the center pixel of the positioning sub-image. The current positioning information of the drone is determined based on the actual depth information corresponding to the projected pixels.

2. The method as described in claim 1, characterized in that, Based on the matching relationship between the current image captured by the UAV and the keyframe, and the initial depth information corresponding to the sub-image determined by the single-point TOF sensor located on the UAV, the actual depth information corresponding to the sub-image is determined, specifically including: The current image captured by the UAV and the initial depth information corresponding to the sub-image determined by the single-point TOF sensor on the UAV when the UAV captures the key frame are obtained, and used as the initial depth information corresponding to the sub-image determined under the key frame; Based on the matching relationship between the acquired current image and the keyframe, the initial depth information corresponding to the sub-image determined under the keyframe is updated to obtain the actual depth information corresponding to the sub-image.

3. The method as described in claim 2, characterized in that, Based on the matching relationship between the acquired current image and the keyframe, the initial depth information corresponding to the sub-image determined under the keyframe is updated, specifically including: Determine the matching degree between the acquired current image and the keyframe; If the matching degree is greater than a preset threshold, the current image will not be re-determined as a key frame, and the initial depth information corresponding to the sub-image determined under the key frame will be updated according to the depth information observed by the single-point TOF sensor when the UAV collects the current image. If the matching degree is not greater than a preset threshold, the current image is redefined as a keyframe, and for each sub-image in the current image that has been redefined as a keyframe, the initial depth information corresponding to the sub-image determined under the current image is updated.

4. The method as described in claim 3, characterized in that, Determine the initial depth information corresponding to this sub-image, specifically including: If the current image is the keyframe, determine the historical keyframe, which refers to the keyframe determined before the UAV acquired the current image; For each sub-image in the current image, determine whether the sub-image has a spatial mapping relationship with the historical keyframe; If the sub-image has a spatial mapping relationship with the historical keyframe, the initial depth information corresponding to the sub-image is determined based on the specified sub-image in the historical keyframe that has a spatial mapping relationship with the sub-image and the depth information corresponding to the specified sub-image.

5. The method as described in claim 4, characterized in that, The method further includes: If the sub-image does not have a spatial mapping relationship with the historical keyframe, the depth information observed by the single-point TOF sensor when the UAV acquires the current image is determined by the single-point TOF sensor located on the UAV, and used as the initial depth information corresponding to the sub-image.

6. The method as described in claim 3, characterized in that, Based on the depth information observed by the single-point TOF sensor when the UAV acquires the current image, the initial depth information corresponding to the sub-image determined in the keyframe is updated, specifically including: If the current image is not the keyframe, determine the spatial mapping relationship between the keyframe and the current image based on the pose of the UAV when acquiring the keyframe and the predicted pose of the UAV when acquiring the current image. Based on the spatial mapping relationship and the target depth information observed by the single-point TOF sensor when the UAV collects the current image, a sub-image corresponding to the region where the UAV observes the target depth information in the keyframe is determined as the target sub-image, and depth information that is converted from the target depth information to the target sub-image is determined as the updated depth information; The initial depth information corresponding to the target sub-image is updated based on the updated depth information.

7. The method as described in claim 6, characterized in that, Based on the updated depth information, the initial depth information corresponding to the target sub-image is updated, specifically including: Determine the reliability of the initial depth information corresponding to the target sub-image and the reliability of the target depth information observed by the single-point TOF sensor; Based on the credibility of the initial depth information and the credibility of the target depth information, a credibility coefficient between the initial depth information and the target depth information corresponding to the target sub-image is determined; The initial depth information corresponding to the target sub-image is updated based on the initial depth information, the updated depth information, and the confidence coefficient.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1-7.

9. A drone, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1-7.

Images