Aircraft visual navigation method and device based on map positioning correction

Abstract

The invention provides an aircraft visual navigation method and device based on map positioning correction, and is applied to the technical field of aircraft control. The method comprises the following steps: acquiring an environment image sequence of a flight environment of the aircraft; the environment image sequence comprises a current environment image collected at a current collection moment and a plurality of historical environment images collected at each historical collection moment. And predicting a predicted pose of the aircraft at the current moment based on the plurality of historical environment images. And matching the visual feature points in the current environment image with map feature points in the flight environment map to obtain a matching result of the plurality of feature point pairs. And determining a corrected pose of the aircraft in the flight environment based on the matching result and the predicted pose. And according to the corrected pose and the flight environment map, the flight path of the aircraft is optimized so as to drive the aircraft to fly, and the problems of matching failure and poor positioning correction reliability caused by the fact that the VIO pose with the accumulative error is depended as a matching initial value in a traditional method can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of aircraft control, and in particular to an aircraft visual navigation method and device based on map positioning correction. BACKGROUND

[0002] With the rapid development of low-altitude economy, electric vertical take-off and landing (eVTOL) aircraft, unmanned aerial vehicles and other equipment are increasingly widely used in fields such as logistics transportation, urban inspection and emergency rescue. These application scenarios have put forward very high requirements for the autonomous navigation capability of the aircraft, especially the positioning accuracy and robustness in complex environments have become key technical bottlenecks.

[0003] In order to improve the positioning accuracy and robustness of the aircraft, the related technology mainly adopts a navigation method combining visual-inertial odometry (VIO) and pre-stored map matching. Among them, VIO is used to provide real-time relative pose estimation, and then the accumulated error generated by VIO is corrected by matching with the pre-stored high-precision map.

[0004] However, this combination still has a series of problems in actual application. For example, map matching and pose solving need an initial pose estimation to narrow down the search range. This process usually directly uses the pose output by VIO, which has accumulated error, as the initial value. When VIO drifts seriously, this initial value will deviate seriously from the true position, causing the matching algorithm to search in the wrong area, which is easy to cause matching failure or a large number of false matches, so that the subsequent correction is completely invalid.

[0005] Therefore, an aircraft visual navigation method and device based on map positioning correction are needed to solve the problem of matching failure and poor positioning correction reliability caused by relying on VIO pose with accumulated error as the matching initial value in the traditional method. SUMMARY

[0006] The purpose of the present application is to provide an aircraft visual navigation method and device based on map positioning correction to solve the problem of matching failure and poor positioning correction reliability caused by relying on VIO pose with accumulated error as the matching initial value in the traditional method.

[0007] In a first aspect, an embodiment of the present application provides a method for visual navigation of an aircraft based on map positioning correction, which comprises: acquiring an environment image sequence of a flight environment in which the aircraft is located. The environment image sequence comprises a current environment image acquired at a current acquisition time and a plurality of historical environment images acquired at respective historical acquisition times. Based on the plurality of historical environment images, a predicted pose of the aircraft at the current time is predicted. A visual feature point in the current environment image is matched with a map feature point in a flight environment map to obtain a matching result of a plurality of feature point pairs. Based on the matching result and the predicted pose, a corrected pose of the aircraft in the flight environment is determined. According to the corrected pose and the flight environment map, a flight trajectory of the aircraft is optimized to drive the aircraft to fly.

[0008] The method for visual navigation of the aircraft based on map positioning correction provided by the embodiments of the present application acquires an environment image sequence of a flight environment in which the aircraft is located, predicts a predicted pose of the aircraft at a current time based on a plurality of historical environment images in the environment image sequence, then matches a visual feature point in a current environment image with a map feature point in a flight environment map to obtain a matching result of a plurality of feature point pairs, corrects the predicted pose based on the matching result, and obtains a corrected pose of the aircraft in the flight environment at a current acquisition time. Finally, based on the corrected pose and the flight environment map, a flight trajectory of the aircraft is optimized, so that the flight trajectory of the aircraft is highly consistent with the spatial structure of the flight environment, the accuracy and safety of flight control of the aircraft are ensured, and the autonomous navigation accuracy, robustness and overall flight task reliability of the aircraft in a complex and GNSS-free environment are improved.

[0009] In a possible implementation, the predicted pose of the aircraft at the current time is predicted based on the plurality of historical environment images, which comprises: for any historical time, based on a historical environment image of the corresponding historical time, a historical pose of the aircraft and a historical visual feature point of the aircraft in the historical environment image are extracted. The predicted pose is determined according to the historical poses of the plurality of historical times and the historical visual feature points of the plurality of historical times.

[0010] In a possible implementation, the matching result comprises a plurality of feature point pairs. One feature point pair comprises one visual feature point and one map feature point. Based on the matching result and the predicted pose, the corrected pose of the aircraft in the flight environment is determined, which comprises: acquiring a visual three-dimensional coordinate of each visual feature point photographed by the aircraft at the predicted pose. According to the visual three-dimensional coordinate, a ground coordinate of each map feature point in the flight environment map, and the predicted pose, a matching error value of the predicted pose is determined. The predicted pose is corrected according to the matching error value to obtain the corrected pose. Figure Three

[0011] In a possible implementation, according to the visual three-dimensional coordinate, the ground coordinate of each map feature point in the flight environment map, and the predicted pose, a matching error value of the predicted pose is determined. The predicted pose is corrected according to the matching error value to obtain the corrected pose.​Figure Three The matching error value of the predicted pose is determined according to the target visual three-dimensional coordinates of the target feature points corresponding to each target matching pair, the target map feature points corresponding to each target matching pair, and the predicted pose. Figure Three The sub-matching error value is calculated according to the target visual three-dimensional coordinates and the predicted pose.

[0012] In a possible implementation, the predicted pose is corrected according to the matching error value to obtain a corrected pose, including: determining the matching error values corresponding to all poses in a preset region based on the predicted pose. The pose with the minimum matching error value is determined as the corrected pose.

[0013] In a possible implementation, the predicted pose is corrected according to the matching error value to obtain a corrected pose, including: adjusting the predicted pose based on the matching error value to obtain a preliminary corrected pose of the aircraft. The corrected visual three-dimensional coordinates of each visual feature point photographed by the aircraft at the corrected pose are obtained. The corrected matching error value of the corrected pose is determined according to the corrected visual three-dimensional coordinates, the map feature points in the flight environment map, and the corrected pose. Figure Three The matching error value of the predicted pose is determined according to the target visual three-dimensional coordinates of the target feature points corresponding to each target matching pair, the target map feature points corresponding to each target matching pair, and the predicted pose.

[0014] In a possible implementation, the flight environment map is updated based on the corrected pose, including: determining the feature density of the visual feature points in a preset range during the flight of the aircraft. The map quality score of the flight environment map is determined according to the feature density and the matching result. Whether to switch the map type of the flight environment map is determined according to the map quality score.

[0015] In a possible implementation, the flight trajectory of the aircraft is optimized according to the corrected pose and the flight environment map, including: determining the global flight trajectory of the aircraft according to the corrected pose and the map semantic label marked in the flight environment map. The local flight trajectory of the aircraft is determined according to the plurality of path nodes in the global flight trajectory, so that the aircraft can fly smoothly.

[0016] In a possible implementation, the flight environment map is updated based on the corrected pose, including: determining the feature density of the visual feature points in a preset range during the flight of the aircraft. The map quality score of the flight environment map is determined according to the feature density and the matching result. Whether to switch the map type of the flight environment map is determined according to the map quality score.

[0017] In a possible implementation, the flight environment map is updated based on the corrected pose, including: determining the feature density of the visual feature points in a preset range during the flight of the aircraft. The map quality score of the flight environment map is determined according to the feature density and the matching result. Whether to switch the map type of the flight environment map is determined according to the map quality score.

[0018] The acquisition module is configured to acquire an environment image sequence of a flight environment in which the aerial vehicle is located. The environment image sequence includes a current environment image acquired at a current acquisition time and a plurality of historical environment images acquired at respective historical acquisition times.

[0019] The prediction module is configured to predict a predicted pose of the aerial vehicle at the current time based on the plurality of historical environment images.

[0020] The matching module is configured to match visual feature points in the current environment image with map feature points in the flight environment map to obtain a matching result of a plurality of feature point pairs.

[0021] The determination module is configured to determine a corrected pose of the aerial vehicle in the flight environment based on the matching result and the predicted pose.

[0022] The optimization module is configured to optimize a flight trajectory of the aerial vehicle according to the corrected pose and the flight environment map to drive the aerial vehicle to fly.

[0023] In a possible implementation, the prediction module is configured to predict a predicted pose of the aerial vehicle at the current time based on the plurality of historical environment images, and specifically configured to: for any historical time, extract a historical pose of the aerial vehicle and historical visual feature points of the aerial vehicle in a historical environment image based on a historical environment image of the corresponding historical time; and determine the predicted pose based on the historical poses of the plurality of historical times and the historical visual feature points of the plurality of historical times.

[0024] In a possible implementation, the matching result includes a plurality of feature point pairs. One feature point pair includes one visual feature point and one map feature point. The determination module is configured to determine a corrected pose of the aerial vehicle in the flight environment based on the matching result and the predicted pose, and specifically configured to: acquire a visual three-dimensional coordinate of each visual feature point photographed at the predicted pose; determine a matching error value of the predicted pose based on the visual three-dimensional coordinates, a ground three-dimensional coordinate of each map feature point in the flight environment map, and the predicted pose; and correct the predicted pose based on the matching error value to obtain the corrected pose. Figure Three

[0025] In a possible implementation, the determination module is configured to determine a matching error value of the predicted pose based on the visual three-dimensional coordinates, a ground three-dimensional coordinate of each map feature point in the flight environment map, and the predicted pose, and specifically configured to: for any target matching pair, calculate a sub-matching error value based on a target visual three-dimensional coordinate of a target visual feature point corresponding to the target matching pair, a target ground three-dimensional coordinate of a target map feature point corresponding to the target matching pair, and the predicted pose; and weight and sum the sub-matching error values of all matching pairs to obtain the matching error value. Figure Three Figure Three In a possible implementation, the determination module is configured to determine a matching error value of the predicted pose based on the visual three-dimensional coordinates, a ground three-dimensional coordinate of each map feature point in the flight environment map, and the predicted pose, and specifically configured to: for any target matching pair, calculate a sub-matching error value based on a target visual three-dimensional coordinate of a target visual feature point corresponding to the target matching pair, a target ground three-dimensional coordinate of a target map feature point corresponding to the target matching pair, and the predicted pose; and weight and sum the sub-matching error values of all matching pairs to obtain the matching error value.​​

[0026] In a possible implementation, the determining module is configured to correct the predicted pose according to the matching error values to obtain a corrected pose, and specifically configured to: determine matching error values corresponding to all poses in the preset region based on the predicted pose, and determine the pose with the minimum matching error value as the corrected pose.

[0027] In a possible implementation, the determining module is configured to correct the predicted pose according to the matching error values to obtain a corrected pose, and specifically configured to: adjust the predicted pose based on the matching error values to obtain a preliminary corrected pose of the aerial vehicle, obtain corrected visual three-dimensional coordinates of each visual feature point photographed by the aerial vehicle at the preliminary corrected pose, determine a corrected matching error value of the preliminary corrected pose according to the corrected visual three-dimensional coordinates, the three-dimensional coordinates of each map feature point in the flight environment map, and the preliminary corrected pose, and determine the preliminary corrected pose as the corrected pose in a case where the corrected matching error value is less than a threshold value. Figure Three

[0028] In a possible implementation, the aerial vehicle visual navigation device based on map positioning correction provided by the embodiments of the present application is further configured to: determine a feature density of the visual feature points in a preset range during flight of the aerial vehicle, determine a map quality score of the flight environment map according to the feature density and the matching result, and determine whether to switch a map type of the flight environment map according to the map quality score.

[0029] In a possible implementation, the optimization module is configured to optimize a flight trajectory of the aerial vehicle according to the corrected pose and the flight environment map, and specifically configured to: determine a global flight trajectory of the aerial vehicle according to the corrected pose and a map semantic label marked in the flight environment map, and determine a local flight trajectory of the aerial vehicle according to a plurality of path nodes in the global flight trajectory, so that the aerial vehicle flies stably.

[0030] In a possible implementation, the aerial vehicle visual navigation device based on map positioning correction provided by the embodiments of the present application is further configured to: extract new visual feature points based on all environment images collected by the aerial vehicle during flight, and add the new visual feature points to the flight environment map as new map feature points to perform incremental updating on the flight environment map.

[0031] In a third aspect, the embodiments of the present application provide an aerial vehicle visual navigation device based on map positioning correction, which has the function of implementing the aerial vehicle visual navigation method based on map positioning correction of the first aspect or any possible implementation of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0032] ​In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores instructions, when the instructions are run on a computer, cause the computer to execute the aircraft visual navigation method based on map positioning correction of the first aspect or any possible implementation manner of the first aspect.

[0033] In a fifth aspect, an embodiment of the present application provides a computer program product containing instructions, when the instructions are run on a computer, cause the computer to execute the aircraft visual navigation method based on map positioning correction of the first aspect or any possible implementation manner.

[0034] The technical effects brought by any design manner of the second aspect to the fifth aspect can refer to the technical effects brought by the first aspect or different possible implementation manners of the first aspect, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the specific embodiments or prior art in the present application, the drawings needed to be used in the specific embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0036] Figure One A structural schematic diagram of an aircraft provided by an embodiment of the present application is shown in the figure. Figure Two Another structural schematic diagram of an aircraft provided by an embodiment of the present application is shown in the figure. Figure Three A method flow chart of an aircraft visual navigation method based on map positioning correction provided by an embodiment of the present application is shown in the figure. Figure Four A structural schematic diagram of an aircraft visual navigation device based on map positioning correction provided by an embodiment of the present application is shown in the figure. Figure Five A system architecture diagram of an aircraft visual navigation system based on map positioning correction provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0038] The following detailed description of embodiments of the application provided in the accompanying drawings is not intended to limit the scope of the application as claimed, but merely represents selected embodiments of the application. Based upon the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present application.

[0039] With the wide application of autonomous flight systems such as unmanned aerial vehicles, unmanned aerial vehicles, etc. in the fields of industrial inspection, logistics distribution, aerial photography and mapping, etc., the demand for high-precision and high-reliability positioning and navigation of the aircraft is increasing. The traditional global navigation satellite system (GNSS) can provide relatively accurate position information in open environments, but in complex environments such as cities, canyons, indoors, forest areas or tunnels, GNSS signals are easily blocked or interfered, resulting in positioning failure or serious accuracy decline. Therefore, multi-sensor fusion navigation technology has become a research hotspot, among which VIO is widely used to realize real-time pose estimation of the aircraft under the condition of no GNSS due to its environmental perception ability of the camera and high-frequency response characteristics of the inertial measurement unit (IMU).

[0040] To further improve the long-term positioning accuracy and robustness of the VIO system, the related technology introduces a map matching method based on a pre-stored high-precision map. This method compares the real-time pose provided by the VIO with the known features in the map to identify the exact position of the aircraft in the global map, thereby correcting the cumulative error of the VIO. In a typical technical solution, the relative motion trajectory of the aircraft is first continuously output by the VIO, and then the current pose on the trajectory is used as the initial estimate value, which is input to the map matching module to search for the best matching position within the local search range, and the final pose output is adjusted accordingly.

[0041] However, in actual operation, due to the drift problem of VIO itself accumulated over time, especially in scenes with missing textures, rapid motion or severe changes in lighting, the output pose may have deviated significantly from the true value. When this pose with a large deviation is directly used as the initial estimate for map matching, it will cause the matching algorithm to search in the wrong area on the map, beyond the effective matching range, and further cause matching failure or a large number of false matching points. In this case, not only the error correction cannot be completed, but the pose estimation may be further deteriorated, seriously affecting the navigation performance and flight safety of the aircraft.

[0042] Based on this, the embodiment of the present application provides a kind of aircraft visual navigation method based on map positioning correction, which comprises obtaining the environment image sequence of the flight environment where the aircraft is located.The environment image sequence includes the current environment image collected at the current collection time and the multiple historical environment images collected at each historical collection time.Based on multiple historical environment images, the predicted pose of the aircraft at the current time is predicted.The visual feature points in the current environment image are matched with the map feature points in the flight environment map to obtain the matching result of multiple feature point pairs.Based on the matching result and the predicted pose, the corrected pose of the aircraft in the flight environment is determined.According to the corrected pose and the flight environment map, the flight trajectory of the aircraft is optimized to drive the aircraft to fly.

[0043] The aircraft visual navigation method based on map positioning correction provided by the embodiment of the present application, by obtaining the environment image sequence of the flight environment where the aircraft is located, based on the multiple historical environment images in the environment image sequence, the predicted pose of the aircraft at the current time is predicted, then the visual feature points in the current environment image are matched with the map feature points in the flight environment map to obtain the matching result of multiple feature point pairs, and then the predicted pose is corrected based on the matching result to obtain the corrected pose of the aircraft in the flight environment at the current collection time.Finally, based on the corrected pose and the flight environment map, the flight trajectory of the aircraft is optimized, so that the flight trajectory of the aircraft is highly consistent with the spatial structure of the flight environment, ensuring the accuracy and safety of the flight control of the aircraft, thereby improving the autonomous navigation accuracy, robustness and overall flight task reliability of the aircraft in complex, GNSS-free environment.

[0044] The method provided by the embodiment of the present application will be described below in conjunction with specific drawings.

[0045] In one aspect, the embodiment of the present application provides an aircraft. As shown in the figure, the aircraft 100 includes a binocular vision camera 101, an on-board processor 102, a flight controller 103, a servo steering engine 104 and a motor 105. Figure One

[0046] The binocular vision camera 101 can be fixedly installed on the head or abdomen of the aircraft 100.The binocular vision camera 101 is used to continuously collect the environment image sequence of the environment in front of or below the aircraft 100 during flight.

[0047] The on-board processor 102 is in communication connection with the binocular vision camera 101.The on-board processor 102 is used to receive the environment image sequence collected by the binocular vision camera 101, and determine the corrected pose of the aircraft in the flight environment map based on the environment image sequence and the flight environment map stored in advance by the aircraft visual navigation method based on map positioning correction provided by the embodiment of the present application. ​

[0048] The flight controller 103 is in communication connection with the onboard processor 102. The flight controller 103 is configured to receive the corrected pose determined by the onboard processor 102, and then plan a flight trajectory of the aerial vehicle based on the aerial vehicle visual navigation method based on map positioning correction provided by the embodiments of the present application, and the terrain and obstacle semantic information in the flight environment map. Then, the corresponding rudder control signal and motor signal are generated based on the flight trajectory.

[0049] The servo rudder 104 can be installed at the control surface rotation shaft of the aerial vehicle 100, and is in communication connection with the flight controller 103. The servo rudder 104 is configured to receive the rudder control signal sent by the flight controller 103, and drive the deflection of the rudder under the control of the rudder control signal.

[0050] The motor 105 is in communication connection with the flight controller 103. The motor 105 is configured to receive the motor signal sent by the flight controller 103, and adjust the rotation speed of the motor 105 under the control of the motor signal to provide thrust for the aerial vehicle 100.

[0051] Further, as shown in Figure Two The aerial vehicle 100 can further include an infrared thermal imager 106 and an inertial measurement unit 107.

[0052] The infrared thermal imager 106 can be installed side by side with the binocular vision camera 101. The infrared thermal imager 106 is configured to collect infrared images at night, in fog or in smoke environment, and provide infrared feature data to the onboard processor 102.

[0053] The inertial measurement unit 107 can be installed at a position close to the center of gravity of the fuselage, and is in communication connection with the onboard processor 102. The inertial measurement unit 107 is configured to measure the angular velocity and acceleration of the aerial vehicle in real time, and assist in pose prediction and motion compensation.

[0054] It should be noted that the aerial vehicle 100 shown in the above Figure One and Figure Two is only an example of the application scenario of the scheme of the present application, and is not a limitation on the application scenario of the scheme of the present application.

[0055] In one aspect, the embodiments of the present application provide an aerial vehicle visual navigation method based on map positioning correction. The method can be executed by the aerial vehicle 100 shown in Figure One As shown in Figure Three The method can include the following steps.

[0056] S301, acquiring an environment image sequence of a flight environment in which an aerial vehicle is located.

[0057] The environment image sequence includes a current environment image collected at a current collection time and a plurality of historical environment images collected at respective historical collection times. The current environment image can be a current image frame acquired by the aerial vehicle in a latest sampling period and used for performing instant positioning and matching. The historical environment images can be a plurality of image frames stored in chronological order before the current time, for example, consecutive image frames in the last 1 second.

[0058] Specifically, during flight, environment images of a flight environment in which the aerial vehicle is located are continuously collected at a fixed collection frequency, and the environment images are stored in chronological order of collection to obtain the environment image sequence.

[0059] For example, when a UAV performs a patrol task, a binocular camera carried by the UAV continuously captures a front scene at a frequency of 20 frames per second. An on-board processor maintains an image sequence with a length of 30 frames at each time, in which the latest frame is a current environment image and the previous 29 frames are historical environment images. The sequence is continuously updated as a new frame is added and an old frame is removed, thereby providing a continuous and time-sensitive visual data stream for a next prediction step.

[0060] S302, predicting a predicted pose of the aerial vehicle at the current time based on the plurality of historical environment images.

[0061] Specifically, for any historical time, a historical pose of the aerial vehicle and a historical visual feature point of the aerial vehicle in a historical environment image corresponding to the historical time are extracted based on the historical environment image. The predicted pose is determined according to the historical poses of the plurality of historical times and the historical visual feature points of the plurality of historical times.

[0062] In one possible implementation, the historical poses of the plurality of historical times and the historical visual feature points of the plurality of historical times are input into a pre-constructed time sequence prediction model to predict the predicted pose of the aerial vehicle at the current time. The time sequence prediction model learns and models a spatiotemporal variation rule of a motion pattern of the aerial vehicle and an environment feature by using a historical pose sequence and a historical visual feature point sequence of a plurality of historical times in a previous preset time length (for example, the last 0.5 seconds) as input, and then infers and outputs the predicted pose of the aerial vehicle at the current time. The time sequence prediction model can be a lightweight long short-term memory (LSTM).

[0063] Exemplarily, historical poses (each pose contains a three-dimensional position and an attitude angle, a total of 6 dimensions) of the past 10 consecutive historical moments and historical visual feature points (for example, 500 128-dimensional feature vectors are extracted from each image, and after pooling or encoding, fixed-length time sequence features are formed) extracted from the corresponding images are spliced and aligned and input into the time sequence prediction model. The time sequence prediction model is propagated forward, and a 6-dimensional vector is output, which represents the predicted pose (predicted position and predicted attitude) of the aircraft at the current moment. Compared with directly using the output of the VIO, the predicted pose usually has better smoothness and robustness to short-term visual loss, and can provide a more reliable pose prior for subsequent map matching to correct tracking loss caused by motion blur or short-term occlusion.

[0064] S303, match the visual feature points in the current environment image with the map feature points in the flight environment map to obtain a matching result of a plurality of feature point pairs.

[0065] Specifically, the process of extracting visual feature points from the current environment image can be to use a multi-scale feature extraction algorithm (speeded up robust features, SUFT), construct an image pyramid and detect features on each layer to ensure that significant ground or environmental details can be captured at different flight altitudes. Each extracted visual feature point contains its three-dimensional pixel coordinates and a feature description vector.

[0066] The flight environment map is a multi-modal map formed in advance through a map construction stage, for example, simultaneously containing a dense point cloud map, a sparse feature map, and an infrared fusion sub-map.

[0067] In one possible implementation, the construction process of the flight environment map can be: when the aircraft or the special mapping device performs a mapping task, multi-modal image data of the environment is collected by the mounted binocular vision camera and infrared camera, and the data is processed by using an improved simultaneous localization and mapping (SLAM) algorithm (for example, an ORB-SLAM4 algorithm combined with a long short-term memory network module).

[0068] The process simultaneously generates the geometric structure (point cloud) of the environment, extracts and optimizes stable sparse feature points, and labels semantic information (such as “building-static” and “road-passable”) for the features or regions in the map through a semantic segmentation model (such as YOLOv8). In addition, in a weak light or fog environment, thermal features are supplemented by the infrared camera to form an infrared fusion sub-map. Finally, the constructed map is scored for quality. The score can be determined by the following formula.

[0069]

[0070] wherein S is the map quality score. and are weight coefficients. is the map feature point density. is the feature density threshold, for example, 30 per square meter. is the matching error threshold, for example, 0.5 meters. is the map internal consistency error, i.e., the average re-projection error of feature points after multi-view geometry optimization (such as bundle adjustment) in the map construction process.

[0071] For example, from all the image frames collected during mapping, a part of the frames not directly involved in the final global optimization is reselected as a verification set. For each frame image in the verification set, its feature points are extracted. Using the camera pose obtained after SLAM optimization of this image, these feature points are inversely projected into the three-dimensional flight environment map just built. Thus, the matching pairs between the feature points of the verification set and the map feature points are obtained. For each matching pair, the Euclidean distance between its three-dimensional coordinates is calculated, and the average value of the Euclidean distances of all matching pairs in the verification set is calculated to obtain .

[0072] After determining the map quality score of the map, only when the score meets the threshold (such as S≥0.8), the map is put into the warehouse as a flight environment map for subsequent navigation. The map feature points usually carry their three-dimensional global coordinates, feature descriptors, and optional semantic labels.

[0073] After the visual feature points and the map feature points are extracted, the visual feature points in the current environment image are matched with the map feature points in the flight environment map to obtain the matching results of multiple feature point pairs.

[0074] For example, first, the visual feature points of the current frame are matched with the map feature points by a fast library for approximate nearest neighbors (FLANN) matcher to obtain an initial matching pair set. Then, a bidirectional matching verification method is used to screen the initial matching pairs: namely, the nearest neighbor search is performed in two directions of “current frame→map” and “map→current frame” in turn, and only the feature point pairs that are the optimal matches in both directions are retained to eliminate false matches.

[0075] Further, before or during matching, the type of map participating in matching can be dynamically selected according to the map quality score.

[0076] Specifically, during the flight of the aerial vehicle, a feature density of the visual feature points in a preset range is determined. According to the feature density and the matching result, a map quality score of the flight environment map is determined. According to the map quality score, it is determined whether to switch the map type of the flight environment map.

[0077] For example, when the map quality score S is greater than or equal to 0.8, a dense semantic map containing rich details is enabled for matching; when the score S is between 0.5 and 0.8, a sparse feature map and an infrared fusion sub-map with less data and stronger anti-interference capability are switched to for matching. Through dynamic switching of the map type of the flight environment map, the process can ensure stable and reliable matching results under different environmental conditions, such as urban high-rise areas, farmland, and water areas.

[0078] In S304, a corrected pose of the aerial vehicle in the flight environment is determined based on the matching result and the predicted pose.

[0079] The matching result includes a plurality of feature point pairs. A feature point pair includes a visual feature point and a map feature point.

[0080] In one possible implementation, a visual three-dimensional coordinate of each visual feature point photographed by the aerial vehicle at the predicted pose is obtained. According to the visual three-dimensional coordinate, a three-dimensional coordinate of each map feature point in the flight environment map, and the predicted pose, a matching error value of the predicted pose is determined. The predicted pose is corrected according to the matching error value to obtain a corrected pose. Figure Three

[0081] Specifically, first, a visual three-dimensional coordinate of each visual feature point photographed by the aerial vehicle at the predicted pose is obtained. The three-dimensional coordinate can be obtained by inversely projecting a two-dimensional feature point pixel coordinate in the environment image to a three-dimensional space by combining a camera intrinsic parameter and a camera projection model at the predicted pose. The visual three-dimensional coordinate represents an estimated observation position of the feature point in space under the assumption of the predicted pose. Then, according to the visual three-dimensional coordinate, a three-dimensional coordinate of each map feature point in the flight environment map, and the predicted pose, a matching error value of the predicted pose is determined. The error value quantifies the degree of inconsistency of all matching point pairs in space under the current predicted pose. Figure Three

[0082] Further, according to the visual three-dimensional coordinate, a three-dimensional coordinate of each map feature point in the flight environment map, and the predicted pose, the matching error value of the predicted pose is determined, including: for any target matching pair, according to a target visual three-dimensional coordinate of a target feature point corresponding to the target matching pair, a target three-dimensional coordinate of a target map feature point corresponding to the target matching pair, and the predicted pose, a sub-matching error value is calculated. The sub-matching error values of all matching pairs are weighted and summed to obtain the matching error value. Figure Three Figure Three Further, according to the visual three-dimensional coordinate, a three-dimensional coordinate of each map feature point in the flight environment map, and the predicted pose, the matching error value of the predicted pose is determined, including: for any target matching pair, according to a target visual three-dimensional coordinate of a target feature point corresponding to the target matching pair, a target three-dimensional coordinate of a target map feature point corresponding to the target matching pair, and the predicted pose, a sub-matching error value is calculated. The sub-matching error values of all matching pairs are weighted and summed to obtain the matching error value.​​​

[0083] For example, the calculation of the sub-matching error value corresponds to solving the re-projection error or the three-dimensional Euclidean distance of the matching point pair under the current predicted pose. The weighted sum of the errors of all matching pairs is a target function with the predicted pose as the variable. The target function is as follows.

[0084]

[0085] Wherein, the target function E (R1, t1) is the matching error value to be solved. is the corrected pose, wherein R1 is the corrected rotation matrix, t1 is the corrected translation vector. R is the predicted rotation matrix, is a 3x3 rotation matrix. T is the predicted pose, is a 3x1 translation vector. is the three-dimensional visual coordinates of the i-th visual feature point, is the three-dimensional coordinates of the corresponding i-th map feature point, Figure Three is the three-dimensional coordinates of the corresponding i-th map feature point, is the weight of the matching pair. The weight can be set according to the matching quality, for example, inversely proportional to the matching distance, to reduce the influence of false matching, for example which can be determined by the following formula.

[0086] is the weight is the i-th Euclidean distance of the matching point, is the smoothing coefficient.

[0087] After determining the matching error value (i.e. the target function), the predicted pose can be corrected according to the error value to obtain the optimal corrected pose.

[0088] In one possible implementation, the predicted pose is corrected according to the matching error value to obtain the corrected pose, including: based on the predicted pose, determining the matching error values corresponding to all poses in a preset region. The pose with the minimum matching error value is determined as the corrected pose.

[0089] In another possible implementation, the predicted pose is corrected according to the matching error value to obtain the corrected pose, including: adjusting the predicted pose based on the matching error value to obtain a preliminary corrected pose of the aerial vehicle. The corrected three-dimensional visual coordinates of each visual feature point photographed by the aerial vehicle in the corrected pose are obtained. The corrected matching error value of the corrected pose is determined according to the corrected three-dimensional visual coordinates, the three-dimensional coordinates of each map feature point in the flight environment map, and the corrected pose. In the case where the corrected matching error value is less than a threshold value, the preliminary corrected pose is determined as the corrected pose. Figure Three

[0090] ​Specifically, the predicted pose is taken as the initial value of the optimization iteration, the minimization of the matching error value (objective function) is taken as the optimization goal, and a nonlinear optimization algorithm is adopted to iteratively adjust the pose parameters (R, t). After each iteration, a new pose estimate is obtained. Then, the corrected visual three-dimensional coordinates of each visual feature point photographed by the aircraft at the new pose are obtained. According to the corrected visual three-dimensional coordinates, the three-dimensional coordinates of each map feature point in the flight environment map, and the new pose, the matching error value (i.e., the objective function value) is recalculated. This process is iterated until the change in the objective function value is less than a preset threshold or the maximum number of iterations is reached, at which time the obtained pose is the preliminary corrected pose. Finally, it is verified whether the final matching error value (i.e., the objective function value after optimization convergence) of the preliminary corrected pose is less than an acceptable threshold. If it is less than, it is determined that the preliminary corrected pose is the final corrected pose; if it is not satisfied, a re-matching or fault handling process may be triggered. Figure Four

[0091] S305, according to the corrected pose and the flight environment map, the flight trajectory of the aircraft is optimized to drive the aircraft to fly.

[0092] In one possible implementation, according to the corrected pose and the map semantic labels marked in the flight environment map, the global flight trajectory of the aircraft is determined. According to the plurality of path nodes in the global flight trajectory, the local flight trajectory of the aircraft is determined to enable the aircraft to fly smoothly.

[0093] Specifically, first, according to the corrected pose and the map semantic labels marked in the flight environment map, the global flight trajectory of the aircraft is determined. This process takes the current corrected pose of the aircraft as the starting point of path planning, and searches for a collision-free path from the starting point to the end point in the passable area provided by the map in combination with the task target point. The map semantic labels (such as "building-no-fly", "road-passable", and "open area-safe") provide key environmental semantic understanding for planning, ensuring that the generated global path meets the flight safety constraints in geometry and rules.

[0094] For example, a path search algorithm with improved search strategy (such as an A* algorithm extended from eight-direction search) is used for global planning, which can efficiently avoid no-fly zones in combination with semantic labels and generate a sequence of path points with roughly uniform spacing (e.g., 10-20 meters), constituting the global flight trajectory.

[0095] Then, according to the plurality of path nodes in the global flight trajectory, the local flight trajectory of the aircraft is determined to enable the aircraft to fly smoothly. Since the global path point sequence may not be smooth enough, direct tracking can cause the aircraft attitude to change dramatically, so the global path needs to be smoothed to generate a high-order continuous local trajectory with gentle curvature changes. ​

[0096] For example, a parametric curve fitting method is used, such as a fourth-order Bezier curve, to fit a smooth curve using the first several consecutive global path points as control points. The curve equation is as follows.

[0097]

[0098] where B(t) is a position vector at time t, is the current position, is the intermediate point at 10 m, is the intermediate point at 20 m, is the target point.

[0099] Finally, the optimized smooth local flight trajectory is converted into control instructions (such as rudder target angle, motor target speed) executable by the aircraft actuator, and sent to the flight control actuator through a specific communication interface (such as RS422), so as to accurately drive the aircraft to complete the autonomous flight task along the planned trajectory.

[0100] Further, after the aircraft completes the flight, the newly collected data in this flight task is used to continuously optimize and expand the existing flight environment map, so that the map remains timely, complete and high-precision, and provides a better map basis for subsequent navigation tasks.

[0101] In one possible implementation, based on all the environment images collected by the aircraft during flight, new visual feature points are extracted. The new visual feature points are added to the flight environment map as new map feature points to incrementally update the flight environment map.

[0102] Specifically, the sequence of environment images collected throughout the task is processed offline, and feature extraction algorithms (such as SURF and ORB) are used to detect and describe visual feature points from each image. By comparing and removing (for example, based on feature descriptor similarity and spatial position proximity) all map feature points that already exist in the flight environment map, visual feature points that cannot be successfully matched with existing map points or appear in areas not covered by the map are selected as new visual feature points. The new feature points represent environmental features that are newly observed by the aircraft in this flight and have not been recorded in the map.

[0103] Then, the new visual feature points are added to the flight environment map as new map feature points to incrementally update the flight environment map. For each new visual feature point, its precise three-dimensional coordinates need to be recovered. This can be achieved through multi-view geometry, for example, by using the observations of the feature point in multiple images and the corrected pose of the aircraft at the corresponding time, and calculating its three-dimensional spatial position through triangulation.

[0104] Further, the new points with three-dimensional coordinates and feature descriptors are fused into the original flight environment map as new map feature points.

[0105] For example, the fusion process can use an iterative closest point (ICP) algorithm or its variants to register and splice the new point cloud with the existing map point cloud, ensuring global consistency. At the same time, the semantic segmentation results in the collected images can be combined to assign semantic labels to the new map feature points.

[0106] Further, during or after the incremental update, map optimization and compression can be performed. For example, invalid feature points that have not been matched for a long time due to environmental changes (such as building demolition) can be deleted, or repeated feature points that are too close in space can be merged, thereby controlling the size of the map data and maintaining the efficiency of feature matching during navigation.

[0107] Finally, the updated and optimized new version of the flight environment map will be stored and can be used for the next navigation task of the aircraft or other aircraft of the same type, providing a high-precision map for the next task.

[0108] Further, during the flight of the aircraft, the method provided by the embodiments of the present application also monitors the health status of each component of the aircraft and the reliability of the output data in real time. Once a fault or performance degradation is detected, a pre-set redundancy emergency mechanism is triggered to ensure flight safety.

[0109] One possible implementation is to trigger the sensor redundancy switching mechanism when it is detected that the visual sensor (such as a binocular vision camera) is disconnected or the data stream is abnormally interrupted. At this time, the pure inertial navigation mode will be switched to immediately, that is, only relying on the acceleration, angular velocity and height data provided by the inertial measurement unit and barometer to perform dead reckoning to maintain the basic positioning ability for a short time. At the same time, the aircraft is controlled to enter a conservative flight mode of slow hovering or low-speed forward flight along the current heading, and an alarm is provided for the operator to prompt that the vision system is invalid.

[0110] Another possible implementation is that when the map matching module continuously reports matching failures or the map quality score continuously falls below a safety threshold (for example, S<0.5), it is determined that the map is invalid or unusable in this area, and the path planning redundancy mechanism is triggered. The online path planning relying on real-time positioning will be abandoned, and the pre-stored and verified emergency return path will be enabled. The path is usually a simplified and conservative route from the current airspace to a safe take-off point or a standby landing field. The aircraft will automatically return along the emergency path until it enters a known good navigation area or safely lands.

[0111] In another possible implementation, when the internal inconsistency (i.e. fusion error) of the output positioning result exceeds a set safety limit (e.g. horizontal error > 1 meter), a performance degradation mechanism is triggered. At this time, it is determined that the current positioning accuracy is unreliable and insufficient to support high maneuverability or high-precision route tracking. In response, the aircraft actively limits its maximum flight speed (e.g. to ≤ 5 m / s) and issues a high-level warning to prompt the operator to intervene. This process compensates for the decline in positioning accuracy by reducing the dynamicity to prevent accidents caused by control errors.

[0112] The above mainly introduces the scheme provided by the embodiments of the application from the perspective of the working principle of the device. It can be understood that the aircraft visual navigation device based on map positioning correction comprises a hardware structure and / or a software module for executing each function in order to realize the above functions. Those skilled in the art should easily realize that, in combination with the algorithm steps of each example described in the embodiments disclosed herein, the application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0113] The embodiments of the application can divide the functional modules of the aircraft visual navigation device based on map positioning correction according to the above method examples. For example, each functional module can be divided according to each function, or two or more functions can be integrated in one processing module. The above integrated module can be realized in the form of hardware or software functional module.

[0114] It should be noted that the division of modules in the embodiments of the application is illustrative, and is only a logical functional division. Actual implementation can have another division method. In the case of dividing each functional module according to each function, Figure Four A possible composition schematic diagram of the aircraft visual navigation device based on map positioning correction involved in the above and embodiments is shown. As shown in the figure, Figure Three The aircraft visual navigation device based on map positioning correction 400 can comprise an acquisition module 401, a prediction module 402, a matching module 403, a determination module 404, and an optimization module 405.

[0115] The acquisition module 401 is configured to support the aircraft visual navigation device based on map positioning correction 400 to perform the following functions: Figure Three S301 in the schematic aircraft visual navigation method based on map positioning correction.

[0116] The prediction module 402 is configured to support the aerial vehicle vision navigation device 400 based on map positioning correction to perform Figure Three S302 in the schematic aerial vehicle vision navigation method based on map positioning correction.

[0117] The matching module 403 is configured to support the aerial vehicle vision navigation device 400 based on map positioning correction to perform Figure Three S303 in the schematic aerial vehicle vision navigation method based on map positioning correction.

[0118] The determination module 404 is configured to support the aerial vehicle vision navigation device 400 based on map positioning correction to perform Figure Three S304 in the schematic aerial vehicle vision navigation method based on map positioning correction.

[0119] The optimization module 405 is configured to support the aerial vehicle vision navigation device 400 based on map positioning correction to perform Figure Three S305 in the schematic aerial vehicle vision navigation method based on map positioning correction.

[0120] In a possible implementation, the prediction module is configured to predict a predicted pose of the aerial vehicle at a current time based on a plurality of historical environment images, and specifically configured to: for any historical time, extract a historical pose of the aerial vehicle and a historical visual feature point of the aerial vehicle in a historical environment image based on a historical environment image corresponding to the historical time; and determine the predicted pose based on the historical poses of the plurality of historical times and the historical visual feature points of the plurality of historical times.

[0121] In a possible implementation, the matching result includes a plurality of feature point pairs. One feature point pair includes one visual feature point and one map feature point. The determination module is configured to determine a corrected pose of the aerial vehicle in the flight environment based on the matching result and the predicted pose, and specifically configured to: obtain a visual three-dimensional coordinate of each visual feature point photographed by the aerial vehicle at the predicted pose; determine a matching error value of the predicted pose based on the visual three-dimensional coordinates, a three-dimensional coordinate of each map feature point in a map of the flight environment, and the predicted pose; and correct the predicted pose based on the matching error value to obtain the corrected pose. Figure Three

[0122] In a possible implementation, the determination module is configured to determine the matching error value of the predicted pose based on the visual three-dimensional coordinates, the three-dimensional coordinate of each map feature point in the map of the flight environment, and the predicted pose, and specifically configured to: for any target matching pair, calculate a sub-matching error value based on a target visual three-dimensional coordinate of a target visual feature point corresponding to the target matching pair, a target three-dimensional coordinate of a target map feature point corresponding to the target matching pair, and the predicted pose; and weight and sum the sub-matching error values of all the matching pairs to obtain the matching error value. Figure Three Figure Three In a possible implementation, the determination module is configured to determine the matching error value of the predicted pose based on the visual three-dimensional coordinates, the three-dimensional coordinate of each map feature point in the map of the flight environment, and the predicted pose, and specifically configured to: for any target matching pair, calculate a sub-matching error value based on a target visual three-dimensional coordinate of a target visual feature point corresponding to the target matching pair, a target three-dimensional coordinate of a target map feature point corresponding to the target matching pair, and the predicted pose; and weight and sum the sub-matching error values of all the matching pairs to obtain the matching error value.​​

[0123] One possible implementation involves a determination module that corrects the predicted pose based on the matching error value. Specifically, when obtaining the corrected pose, the module determines the matching error value for all poses within a preset region based on the predicted pose. The pose with the smallest matching error value is then determined as the corrected pose.

[0124] One possible implementation involves a module that corrects the predicted pose based on the matching error value. Specifically, upon obtaining the corrected pose, this module adjusts the predicted pose based on the matching error value to obtain the initial corrected pose of the aircraft. It then acquires the corrected visual 3D coordinates of each visual feature point captured by the aircraft in the corrected pose. Finally, it uses the corrected visual 3D coordinates and the location of each map feature point on the flight environment map... Figure Three The system determines the 3D coordinates and the corrected pose, and then determines the corrected pose matching error value. If the corrected pose matching error value is less than a threshold, the initial corrected pose is determined as the corrected pose.

[0125] In one possible implementation, the aircraft visual navigation device based on map positioning correction provided in this application embodiment is further used to: determine the feature density of visual feature points within a preset range during aircraft flight; determine the map quality score of the flight environment map based on the feature density and matching results; and determine whether to switch the map type of the flight environment map based on the map quality score.

[0126] One possible implementation involves an optimization module that, when optimizing the aircraft's flight trajectory based on the corrected pose and the flight environment map, specifically: determining the aircraft's global flight trajectory based on the semantic labels marked on the map and the corrected pose; and determining the aircraft's local flight trajectory based on multiple path nodes in the global flight trajectory to ensure stable flight.

[0127] In one possible implementation, the aircraft visual navigation device based on map positioning correction provided in this application embodiment is further used to: extract newly added visual feature points based on all environmental images collected by the aircraft during flight; and add the newly added visual feature points as new map feature points to the flight environment map for incremental updates.

[0128] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0129] The aircraft visual navigation device 400 based on map positioning correction provided in this application embodiment is used to perform the above-mentioned... Figure ThreeThe illustrated aircraft visual navigation method based on map positioning correction can achieve the same effect as the above-mentioned aircraft visual navigation method based on map positioning correction.

[0130] The embodiments of the present application further provide an aircraft visual navigation device based on map positioning correction, which can execute the aircraft visual navigation method based on map positioning correction and related steps in the above-mentioned method embodiments.

[0131] The embodiments of the present application further provide a computer readable storage medium, which stores instructions, and the instructions are executed to execute the aircraft visual navigation method based on map positioning correction and related steps in the above-mentioned method embodiments.

[0132] The embodiments of the present application further provide a computer program product, which, when running on a computer, causes the computer to execute the aircraft visual navigation method based on map positioning correction and related steps in the above-mentioned method embodiments.

[0133] In some embodiments, the method illustrated in the present application can be implemented as computer program instructions encoded in a machine-readable storage medium in a machine-readable format or encoded in other non-transitory media or articles.

[0134] The embodiments of the present application further provide an aircraft visual navigation system 500 based on map positioning correction, as illustrated in the figure, Figure Five The aircraft visual navigation system 500 based on map positioning correction includes at least one processor 501 and at least one interface circuit 502.

[0135] For example, when the aircraft visual navigation system 500 based on map positioning correction includes one processor and one interface circuit, the one processor can be Figure Five the processor 501 (or the processor 501) illustrated in the solid line box in the figure, and the one interface circuit can be Figure Five the interface circuit 502 (or the interface circuit 502) illustrated in the solid line box in the figure. When the aircraft visual navigation system 500 based on map positioning correction includes two processors and two interface circuits, the two processors include Figure Five the processor 501 illustrated in the solid line box in the figure and the processor 501 illustrated in the dashed line box in the figure, and the two interface circuits include Figure Five the interface circuit 502 illustrated in the solid line box in the figure and the interface circuit 502 illustrated in the dashed line box in the figure. No limitation is made in this regard.

[0136] The processor 501 and the interface circuit 502 can be interconnected by a line. For example, the interface circuit 502 can be used to receive a signal. For another example, the interface circuit 502 can be used to send a signal to other devices (such as the processor 501). For example, the interface circuit 502 can read the computer instructions stored in the memory and send the computer instructions to the processor 501. The processor 501 executes the instructions and, in combination with the input and output devices, implements various steps in the above-described embodiments, such as implementing various steps performed in the method embodiments shown in the above-described embodiments. Figure Three The aircraft vision navigation system based on map positioning correction can also include other discrete devices, and the embodiments of the present application do not specifically limit this.

[0137] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional modules is exemplified, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0138] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0139] The units described as separate components can or can not be physically separated, and the components shown as units can be one physical unit or multiple physical units, that is, can be located in one place, or can be distributed to multiple different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0140] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0141] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product in essence or in the part that contributes to the present application, or the whole or part of the technical solutions can be embodied in the form of a software product stored in a storage medium, including a plurality of instructions for causing an apparatus (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the method described in the embodiments of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0142] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any change or replacement within the technical scope disclosed in the present application should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A visual navigation method for aircraft based on map positioning correction, characterized in that, The method includes: Acquire an environmental image sequence of the flight environment in which the aircraft is located; the environmental image sequence includes the current environmental image acquired at the current acquisition time and multiple historical environmental images acquired at various historical acquisition times; Based on multiple historical environmental images, predict the aircraft's pose at the current moment; The visual feature points in the current environment image are matched with the map feature points in the flight environment map to obtain the matching results of multiple feature point pairs. Based on the matching results and the predicted pose, the corrected pose of the aircraft in the flight environment is determined. Based on the corrected pose and the flight environment map, the flight trajectory of the aircraft is optimized to drive the aircraft to fly.

2. The method according to claim 1, characterized in that, The step of predicting the aircraft's pose at the current moment based on multiple historical environmental images includes: For any given historical moment, based on the historical environment image at the corresponding historical moment, the historical pose of the aircraft and the historical visual feature points of the aircraft in the historical environment image are extracted. The predicted pose is determined based on the historical poses at multiple historical moments and the historical visual feature points at multiple historical moments.

3. The method according to claim 1, characterized in that, The matching result includes multiple feature point pairs; each feature point pair includes a visual feature point and a map feature point. The step of determining the corrected pose of the aircraft in the flight environment based on the matching result and the predicted pose includes: Obtain the visual three-dimensional coordinates of each visual feature point captured by the aircraft in the predicted pose; The matching error value of the predicted pose is determined based on the visual three-dimensional coordinates, the map three-dimensional coordinates of each map feature point in the flight environment map, and the predicted pose. The predicted pose is corrected based on the matching error value to obtain the corrected pose.

4. The method according to claim 3, characterized in that, The step of determining the matching error value of the predicted pose based on the visual 3D coordinates, the map 3D coordinates of each map feature point on the flight environment map, and the predicted pose includes: For any target matching pair, the sub-matching error value is calculated based on the target visual 3D coordinates of the target feature points corresponding to the target matching pair, the target map 3D coordinates of the target map feature points corresponding to the target map feature points, and the predicted pose. The matching error value is obtained by weighted summing of the sub-matching error values ​​of all matching pairs.

5. The method according to claim 3, characterized in that, The step of correcting the predicted pose based on the matching error value to obtain the corrected pose includes: Based on the predicted pose, determine the matching error value corresponding to all poses within the preset area; The pose with the minimum matching error value is determined as the corrected pose.

6. The method according to claim 3, characterized in that, The step of correcting the predicted pose based on the matching error value to obtain the corrected pose includes: The predicted pose is adjusted based on the matching error value to obtain the preliminary corrected pose of the aircraft; Obtain the corrected visual three-dimensional coordinates of each visual feature point captured by the aircraft in the corrected pose; The correction matching error value of the correction pose is determined based on the corrected visual 3D coordinates, the map 3D coordinates of each map feature point in the flight environment map, and the corrected pose. If the corrected matching error value is determined to be less than the threshold, the preliminary corrected pose is determined to be the corrected pose.

7. The method according to claim 1, characterized in that, The method further includes: During the flight of the aircraft, the feature density of the visual feature points within a preset range is determined; The map quality score of the flight environment map is determined based on the feature density and the matching results. Based on the map quality score, determine whether to switch the map type of the flight environment map.

8. The method according to claim 1, characterized in that, The step of optimizing the flight trajectory of the aircraft based on the corrected pose and the flight environment map includes: The global flight trajectory of the aircraft is determined based on the corrected pose and the map semantic labels marked in the flight environment map. Based on multiple path nodes in the global flight trajectory, the local flight trajectory of the aircraft is determined to ensure stable flight of the aircraft.

9. The method according to claim 1, characterized in that, The method further includes: Based on all environmental images collected by the aircraft during flight, newly added visual feature points are extracted; The newly added visual feature points are added to the flight environment map as new map feature points to perform incremental updates to the flight environment map.

10. An aircraft visual navigation device based on map positioning correction, characterized in that, The device includes: The acquisition module is used to acquire a sequence of environmental images of the flight environment in which the aircraft is located; the sequence of environmental images includes the current environmental image acquired at the current acquisition time and multiple historical environmental images acquired at various historical acquisition times; The prediction module is used to predict the aircraft's pose at the current moment based on multiple historical environmental images. The matching module is used to match visual feature points in the current environment image with map feature points in the flight environment map to obtain matching results for multiple feature point pairs. The determination module is used to determine the corrected pose of the aircraft in the flight environment based on the matching result and the predicted pose. An optimization module is used to optimize the flight trajectory of the aircraft based on the corrected pose and the flight environment map, so as to drive the aircraft to fly.

Images