Calibration method, device and equipment for optical flow module of unmanned aerial vehicle and storage medium

Abstract

The embodiment of the invention discloses an unmanned aerial vehicle optical flow module calibration method and device, equipment and a storage medium, and belongs to the technical field of unmanned aerial vehicle data processing. The method comprises the following steps: acquiring a plurality of sensor heights of a calibration object captured by a sensor of an unmanned aerial vehicle, and correspondingly shot calibration object images; obtaining camera internal parameters of the unmanned aerial vehicle and a plurality of feature point coordinates of the calibration object in a real space; for each calibration object image, determining a pixel point coordinate corresponding to each feature point coordinate according to the calibration object image; for each calibration object image, determining the camera height from the camera of the unmanned aerial vehicle to the calibration object under the calibration object image according to the internal reference of the camera, the coordinate of each feature point and the coordinate of each pixel point, and determining a distance mapping relationship between the distances from the sensor and the camera of the unmanned aerial vehicle to the same object; and calibrating a camera of the unmanned aerial vehicle and a sensor of the unmanned aerial vehicle based on the distance mapping relation. The calibration accuracy of the unmanned aerial vehicle optical flow module can be improved.

Description

Calibration methods, apparatus, equipment, and storage media for UAV optical flow modules Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) data processing technology, specifically to a calibration method, apparatus, device, and storage medium for an UAV optical flow module. Background Technology

[0002] A drone is an aircraft operated by an autonomous flight control system. The optical flow module onboard a drone includes a downward-facing camera and sensors that measure the drone's altitude. The drone can estimate its relative speed or displacement in the horizontal direction using this module. To achieve this, the drone's sensors and cameras need to be calibrated to establish a unified coordinate system. Only within this coordinate system can high-precision pose estimation and environmental perception be achieved in tasks such as visual positioning, odometry, autonomous navigation, and precise landing.

[0003] In related technologies, drones need to fly horizontally at a known altitude to calibrate their optical flow modules based on the pixel movement distance in the camera and a preset horizontal flight distance. However, in actual flight, drones often struggle to maintain a horizontal flight state due to wind disturbances or attitude adjustments, resulting in poor calibration accuracy of the drone's optical flow modules. Summary of the Invention

[0004] This application provides a calibration method, apparatus, device, and storage medium for an unmanned aerial vehicle (UAV) optical flow module, which can improve the calibration accuracy of the UAV optical flow module.

[0005] To achieve the above objectives, one embodiment of this application provides a calibration method for an optical flow module of a UAV, comprising: controlling the UAV to fly vertically above a preset calibration object; when the calibration object is located within a preset field of view of the UAV's camera, acquiring multiple sensor heights of the UAV's sensors to the calibration object, and corresponding images of the calibration object captured at each sensor height; acquiring the UAV's camera intrinsic parameters and the coordinates of multiple feature points of the calibration object in real space; for each calibration object image, determining the pixel coordinates corresponding to each feature point coordinate based on the calibration object image, to obtain the pixel coordinates corresponding to each feature point coordinate in each calibration object image; for each calibration object image, determining the camera height between the UAV's camera and the calibration object in the calibration object image based on the camera intrinsic parameters, each feature point coordinate, and each pixel coordinate, to obtain the camera height corresponding to each calibration object image; determining the distance mapping relationship between the distances of the UAV's sensors and cameras to the same object based on each sensor height and the corresponding camera height, and calibrating the UAV's camera and sensors based on the distance mapping relationship.

[0006] In some embodiments, when the calibration object is located within a preset field of view of the drone's camera, acquiring multiple sensor heights of the calibration object captured by the drone's sensors, and corresponding images of the calibration object at each sensor height, includes: acquiring an initial image of the calibration object captured at the current flight altitude, and acquiring the center coordinates of the calibration object corresponding to the calibration object in the initial image, the center coordinates of the calibration object including the coordinates of a first horizontal axis center point and a first vertical axis center point; acquiring the camera center coordinates of the drone's camera, the camera center coordinates including the coordinates of a second horizontal axis center point and a second vertical axis center point; when the difference in lateral distance between the first horizontal axis center point coordinates and the second horizontal axis center point coordinates is less than a preset lateral distance difference, and the difference in longitudinal distance between the first vertical axis center point coordinates and the second vertical axis center point coordinates is less than a preset longitudinal distance difference, determining that the calibration object is located within a preset field of view of the drone's camera, and acquiring multiple sensor heights of the calibration object captured by the drone's sensors, and corresponding images of the calibration object captured at each sensor height.

[0007] In some embodiments, obtaining the coordinates of multiple feature points of the calibration object in real space includes: obtaining the size information of the calibration object and the origin of the real coordinate system of the real space where the calibration object is located; taking multiple corner points of the calibration object as corresponding feature points and determining the relative positional relationship between each corner point and the origin of the real coordinate system; and determining the corner point coordinates corresponding to each corner point based on the size information and the relative positional relationship corresponding to each corner point, so as to obtain the coordinates of multiple feature points of the calibration object in real space.

[0008] In some embodiments, determining the camera height between the UAV's camera and the calibration object in the calibration object image based on camera intrinsic parameters, coordinates of each feature point, and coordinates of each pixel point includes: obtaining a preset scale factor; determining the UAV's pose in the calibration object image based on the scale factor, camera intrinsic parameters, coordinates of each feature point, and coordinates of each pixel point, and determining the camera height between the UAV's camera and the calibration object based on the pose.

[0009] In some embodiments, determining the distance mapping relationship between the drone's sensors and cameras and the same object based on each sensor height and the corresponding camera height includes: obtaining a preset distance linear function, the distance linear function including an initial distance coefficient and an initial distance constant; updating the initial distance coefficient and the initial distance constant based on each sensor height and the corresponding camera height to obtain a target distance coefficient and a target distance constant; and determining the distance mapping relationship between the drone's sensors and cameras and the same object based on the target distance coefficient and the target distance constant.

[0010] In some embodiments, after calibrating the drone's camera and sensors based on a distance mapping relationship, the method further includes: obtaining the drone's initial movement distance; controlling the drone to fly horizontally, obtaining multiple ground altitude sensor values ​​captured by the drone's sensors, and a target image containing the same ground feature point captured at each altitude sensor value; obtaining the target pixel coordinates of the ground feature point in each target image, and updating the initial movement distance to obtain the flight distance based on each altitude sensor value and the target pixel coordinates of the ground feature point at each target sensing altitude; using the flight distance as the new initial movement distance, and returning to the steps of obtaining multiple ground altitude sensor values ​​captured by the drone's sensors, and a target image containing the same ground feature point captured at each altitude sensor value, until the drone reaches the target position, and using the last updated flight distance as the drone's target movement distance.

[0011] In some embodiments, updating the initial movement distance to obtain the flight distance based on each altitude sensor value and the target pixel coordinates corresponding to the ground feature point at each target sensing altitude includes: determining the spatial position of the UAV based on camera intrinsic parameters, the altitude sensor value, and the target pixel coordinates corresponding to the ground feature point at the target sensing altitude for each altitude sensor value; determining the additional movement distance of the UAV based on the spatial position of the UAV determined at different altitude sensor values; and updating the initial movement distance to obtain the flight distance based on the additional movement distance.

[0012] To achieve the above objectives, one embodiment of this application provides a calibration device for a drone, comprising: a first acquisition module, configured to control the drone to fly vertically above a preset calibration object, and when the calibration object is located within a preset field of view of the drone's camera, acquire multiple sensor heights of the calibration object captured by the drone's sensors, and corresponding images of the calibration object captured at each sensor height; a second acquisition module, configured to acquire the camera intrinsic parameters of the drone and the coordinates of multiple feature points of the calibration object in real space; and a pixel coordinate determination module, configured to determine the pixel coordinates corresponding to each feature point based on the calibration object image for each image of the calibration object. The system includes several modules: a pixel coordinate module to obtain the pixel coordinates corresponding to each feature point coordinate in each calibration object image; a camera height determination module to determine the camera height between the UAV's camera and the calibration object in each calibration object image based on camera intrinsic parameters, each feature point coordinate, and each pixel coordinate, thus obtaining the camera height corresponding to each calibration object image; and a calibration module to determine the distance mapping relationship between the UAV's sensors and cameras and the same object based on each sensor height and the corresponding camera height, and to calibrate the UAV's camera and sensors based on the distance mapping relationship.

[0013] To achieve the above objectives, one aspect of this application provides a computer-readable storage medium storing multiple instructions adapted for loading by a processor to execute the steps in the calibration method for the UAV optical flow module provided in this application.

[0014] To achieve the above objectives, one aspect of this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, it implements the steps in the calibration method for the UAV optical flow module provided in this application.

[0015] To achieve the above objectives, one aspect of this application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in the calibration method for the UAV optical flow module provided in this application.

[0016] The calibration method, apparatus, device, and storage medium for the UAV optical flow module proposed in this application control the UAV to fly vertically above a preset calibration object. When the calibration object is located within a preset field of view of the UAV's camera, the method acquires multiple sensor heights of the UAV's sensors to the calibration object, and corresponding images of the calibration object captured at each sensor height. It also acquires the UAV's camera intrinsic parameters and the coordinates of multiple feature points of the calibration object in real space. For each calibration object image, it determines the pixel coordinates corresponding to each feature point coordinate, thus obtaining the pixel coordinates corresponding to each feature point coordinate in each calibration object image. For each calibration object image, it determines the camera height between the UAV's camera and the calibration object based on the camera intrinsic parameters, each feature point coordinate, and each pixel coordinate, thus obtaining the camera height corresponding to each calibration object image. Finally, it determines the distance mapping relationship between the distances of the UAV's sensors and camera to the same object based on each sensor height and the corresponding camera height, and calibrates the UAV's camera and sensors based on this distance mapping relationship.

[0017] This application addresses the technical challenge of low calibration accuracy in UAVs due to difficulties in maintaining stable horizontal flight and susceptibility to wind disturbances. It transforms the complex horizontal flight calibration process into vertical flight calibration and establishes a distance mapping relationship between the distances of the UAV's sensors and cameras to the same object by using multiple sets of sensor heights and corresponding camera heights. This enables calibration between the UAV's sensors and cameras. Because this application avoids the stringent reliance on high-precision horizontal displacement control and long-term stable hovering found in related technologies, it improves the calibration accuracy of UAVs. Furthermore, the calibrated UAV can accurately convert sensor data into scale information in the camera coordinate system, thus improving the UAV's pose estimation and environmental perception, thereby enhancing the accuracy of the UAV in performing a series of tasks such as visual positioning, odometry, autonomous navigation, and precise landing.

[0018] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 is a schematic diagram of the system framework corresponding to the calibration method of the UAV optical flow module provided in the embodiment of this application; Figure 2 is a schematic flowchart of the calibration method of the UAV optical flow module provided in the embodiment of this application; Figure 3 is a schematic diagram of the calibration object image provided in the embodiment of this application; Figure 4 is another schematic diagram of the calibration object image provided in the embodiment of this application; Figure 5 is a schematic diagram of UAV flight of the calibration method of the UAV optical flow module provided in the embodiment of this application; Figure 6 is a schematic diagram of the module structure of the UAV calibration device provided in the embodiment of this application; Figure 7 is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of this application. Detailed Implementation

[0021] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] It should be noted that in all specific embodiments of this application, when it is necessary to obtain images of calibration objects, permission or consent from the relevant personnel managing the calibration objects is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when this application embodiment needs to obtain sensitive personal information of relevant personnel, separate permission or consent from the relevant personnel is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the separate permission or consent of the relevant personnel is the necessary calibration object image for the normal operation of this application embodiment acquired. Other data obtained in this application embodiment are all authorized and legal data, and will not be elaborated upon here.

[0023] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, programmable consumer computer devices, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0024] First, the technical problems existing in the relevant technology are described: A drone is an aircraft operated by an autonomous flight control system. The optical flow module on the drone includes a downward-facing camera and a sensor that measures the drone's altitude. The drone can estimate its relative speed or displacement in the horizontal direction through the optical flow module. To achieve this, the drone's sensors and cameras need to be calibrated to establish a unified coordinate system. Only under this coordinate system can high-precision pose estimation and environmental perception be achieved in tasks such as visual positioning, odometry, autonomous navigation, and precise landing.

[0025] In related technologies, drones need to fly horizontally at a known altitude to calibrate their optical flow modules based on the pixel movement distance in the camera and a preset horizontal flight distance. However, in actual flight, drones often struggle to maintain a horizontal flight state due to wind disturbances or attitude adjustments, resulting in poor calibration accuracy of the drone's optical flow modules.

[0026] For example, in urban logistics delivery scenarios, drones need to navigate at low altitudes between buildings to deliver goods precisely from delivery stations to users' balconies. To achieve centimeter-level landing accuracy, drones rely on the collaborative work of a downward-looking camera and a ranging sensor, both of which must establish a consistent distance scale through calibration. Traditional calibration methods require the drone to fly horizontally in a straight line for a certain distance at a fixed height (e.g., 2 meters), using the known flight displacement and pixel displacement of feature points in the image to infer the camera scale factor. However, in real urban environments, gusts of wind, turbulence between buildings, or obstacle avoidance maneuvers often cause uncontrollable roll or yaw in the drone, making it impossible for it to maintain ideal horizontal uniform motion. At this time, the actual trajectory of the drone deviates significantly from the preset horizontal flight trajectory, causing the displacement ratio between pixels and the actual physical location calculated based on this assumption to be inaccurate. Ultimately, this leads to visual odometry scale errors, landing position deviations, and even delivery failures or collision risks.

[0027] The calibration method, apparatus, device, and storage medium for the UAV optical flow module proposed in this application control the UAV to fly vertically above a preset calibration object. When the calibration object is located within a preset field of view of the UAV's camera, the method acquires multiple sensor heights of the UAV's sensors to the calibration object, and corresponding images of the calibration object captured at each sensor height. It also acquires the UAV's camera intrinsic parameters and the coordinates of multiple feature points of the calibration object in real space. For each calibration object image, it determines the pixel coordinates corresponding to each feature point coordinate, thus obtaining the pixel coordinates corresponding to each feature point coordinate in each calibration object image. For each calibration object image, it determines the camera height between the UAV's camera and the calibration object based on the camera intrinsic parameters, each feature point coordinate, and each pixel coordinate, thus obtaining the camera height corresponding to each calibration object image. Finally, it determines the distance mapping relationship between the distances of the UAV's sensors and camera to the same object based on each sensor height and the corresponding camera height, and calibrates the UAV's camera and sensors based on this distance mapping relationship.

[0028] This application addresses the technical challenge of low calibration accuracy in UAVs due to difficulties in maintaining stable horizontal flight and susceptibility to wind disturbances. It transforms the complex horizontal flight calibration process into vertical flight calibration and establishes a distance mapping relationship between the distances of the UAV's sensors and cameras to the same object by using multiple sets of sensor heights and corresponding camera heights. This enables calibration between the UAV's sensors and cameras. Because this application avoids the stringent reliance on high-precision horizontal displacement control and long-term stable hovering found in related technologies, it improves the calibration accuracy of UAVs. Furthermore, the calibrated UAV can accurately convert sensor data into scale information in the camera coordinate system, thus improving the UAV's pose estimation and environmental perception, thereby enhancing the accuracy of the UAV in performing a series of tasks such as visual positioning, odometry, autonomous navigation, and precise landing.

[0029] The specific details regarding the calibration method, apparatus, device, and storage medium for the UAV optical flow module provided in this application embodiment will be described in detail below.

[0030] Please refer to Figure 1, which is a schematic diagram of the system framework corresponding to the calibration method for the UAV optical flow module provided in this embodiment. The calibration method for the UAV optical flow module provided in this embodiment can be applied to this system framework.

[0031] It includes terminal 140, Internet 130, gateway 120, server 110, etc.

[0032] Terminal 140 or server 110 may be a device that performs a calibration method for the UAV optical flow module.

[0033] Terminal 140 includes, but is not limited to, mobile phones, tablets, computers, and intelligent computing centers. Terminal 140 can be a single device or a collection of multiple devices. For example, multiple computers can be interconnected via a local area network, sharing a single monitor to work collaboratively, thus forming a terminal 140. Terminal 140 can communicate with the Internet 130 via wired or wireless means to exchange data.

[0034] Server 110 refers to a computer system that can provide certain services to terminal 140. Compared to ordinary terminal 140, server 110 has higher requirements in terms of stability, security, and performance. Server 110 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0035] Gateway 120, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from terminal 140 to server 110 are forwarded to the corresponding server 110 via gateway 120. Messages sent from server 110 to terminal 140 are also forwarded to the corresponding terminal 140 via gateway 120.

[0036] The embodiments of this application can be applied to various scenarios, such as optical flow odometry, drone navigation, and target avoidance. The calibration of drones can be used in other drone-related scenarios. That is, before the drone is officially used, the method proposed in the embodiments of this application can be used to calibrate the drone's camera and sensors. This is only an example of optional scenario application and does not mean that the embodiments of this application limit the scenarios in which the calibration method for the drone's optical flow module can be applied.

[0037] Next, the description will focus on the calibration device of the UAV. As shown in Figure 2, Figure 2 is a flowchart illustrating the calibration method of the UAV optical flow module provided in this embodiment. The calibration method of the UAV optical flow module is applied to the UAV calibration device. The method in Figure 2 may include, but is not limited to, the following steps 210 to 250. When the UAV calibration device executes the calibration method of the UAV optical flow module, the specific process is as follows. It should be noted that this embodiment does not specifically limit the order of steps 210 to 250 in Figure 2. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0038] Step 210: Control the drone to fly vertically above a preset calibration object. When the calibration object is within the preset field of view of the drone's camera, acquire multiple sensor heights of the calibration object captured by the drone's sensors, and the corresponding calibration object image captured at each sensor height. Step 220: Acquire the drone's camera intrinsic parameters and the coordinates of multiple feature points of the calibration object in real space. Step 230: For each calibration object image, determine the pixel coordinates corresponding to each feature point coordinate based on the calibration object image, to obtain the pixel coordinates corresponding to each feature point coordinate in each calibration object image. Step 240: For each calibration object image, determine the camera height between the drone's camera and the calibration object based on the camera intrinsic parameters, each feature point coordinate, and each pixel coordinate, to obtain the camera height corresponding to each calibration object image. Step 250: Based on each sensor height and the corresponding camera height, determine the distance mapping relationship between the drone's sensor and camera and the distance to the same object, and calibrate the drone's camera and sensor based on the distance mapping relationship.

[0039] Steps 210 to 250 are described in detail below.

[0040] In step 210, the drone is controlled to fly vertically above a preset calibration object. When the calibration object is located in the preset field of view of the drone's camera, the drone's sensors capture multiple sensor heights of the calibration object and the corresponding images of the calibration object at each sensor height.

[0041] The drone used for target calibration is equipped with sensors and cameras. The drone's sensors refer to physical measurement devices used to sense flight status, environmental information, or relative relationships with external targets. In this embodiment, the sensors specifically refer to altitude sensors used to measure the distance between the drone and a target object (such as a calibration object or the ground), such as ultrasonic sensors, laser rangefinders, or barometers. The camera in this embodiment refers to a downward-facing camera, which is a camera mounted on the bottom of the drone with its lens facing the ground. It is used to capture images of the area below the drone. It should be noted that drones may also have forward-facing cameras, but these are not discussed in this embodiment. Other components or devices included in conventional drones will not be described further here. Furthermore, the drone's sensors and cameras are usually not located at the same height. Even if they are set to the same height at the factory, their positions may change due to vibration, aging, thermal expansion, and other reasons during actual use. Therefore, calibration is necessary before using the drone.

[0042] In this context, the calibration object is a reference object with known geometric information, such as a checkerboard pattern, concentric circles, or ArUco codes. During UAV calibration, the calibration object is fixed on the ground or at a specific location as a benchmark target for visual measurement. The calibration object used in this embodiment is an ArUco code. An ArUco code is a digital identifier based on a binary square matrix structure, specifically designed for machine vision applications. It typically consists of a black border and an internal black-and-white alternating coded area, possessing a predefined dictionary and a unique identifier (ID). Due to its ease of generation, fast detection speed, and high positioning accuracy, ArUco codes are often used as a fast and reliable visual benchmark or spatial feature point.

[0043] Furthermore, the drone can be controlled to fly vertically above a preset calibration target by a drone processing chip installed in the drone; alternatively, it can be controlled to fly vertically above a preset calibration target by an external connected device such as a computer or mobile phone. Vertical flight ensures that the calibration target remains within the center of the downward-looking camera's field of view during altitude changes, allowing the drone to use known object information about the calibration target as prior knowledge to determine the corresponding camera altitude, while avoiding a decrease in calibration accuracy caused by discrepancies between manually defined flight altitudes and actual altitudes.

[0044] Furthermore, during the vertical flight of the UAV, if the calibration object is located within the preset field of view of the UAV's camera, the UAV's sensors capture the current sensor altitude from the UAV to the calibration object in real time. Simultaneously with acquiring each sensor altitude, an image of the calibration object taken at that sensor altitude is also acquired. The calibration object image contains the calibration object, and the calibration object is located within the preset field of view of the calibration object image.

[0045] The preset field of view (FLP) refers to a specific range defined in advance within the imaging plane of the UAV camera, typically delineated with the image center as the reference. Only when the calibration object falls completely within or remains stably within the preset FLP in the real-time captured image will the UAV's sensors and camera acquire the corresponding data. This avoids introducing additional perspective errors or lens distortion due to excessive offset of the calibration object's position, ensuring the consistency, quality, and accuracy of the data used for subsequent calibration. As shown in Figure 3, which is a schematic diagram of the calibration object image provided in this embodiment, the solid line represents the UAV camera's field of view, the dashed line represents the UAV's preset FLP, and the ArUco code represents the calibration object. In the case shown in Figure 3, the UAV's sensor acquires the corresponding sensor height, and the UAV's camera acquires the corresponding calibration object image.

[0046] In some embodiments, when the calibration object is located within a preset field of view of the UAV's camera, the UAV's sensors capture multiple sensor heights to the calibration object and corresponding calibration object images captured at each sensor height, including: (1.1) acquiring an initial calibration object image captured at the current flight altitude, and acquiring the calibration object center coordinates corresponding to the calibration object in the initial calibration object image, the calibration object center coordinates including the first horizontal axis center point coordinates and the first vertical axis center point coordinates; (1.2) acquiring the camera center coordinates of the UAV's camera, the camera center coordinates including the second horizontal axis center point coordinates and the second vertical axis center point coordinates; (1.3) when the difference in lateral distance between the first horizontal axis center point coordinates and the second horizontal axis center point coordinates is less than a preset lateral distance difference, and the difference in longitudinal distance between the first vertical axis center point coordinates and the second vertical axis center point coordinates is less than a preset longitudinal distance difference, the calibration object is determined to be located within a preset field of view of the UAV's camera, and the UAV's sensors capture multiple sensor heights to the calibration object and corresponding calibration object images captured at each sensor height.

[0047] In some embodiments, during vertical flight, the UAV first uses a downward-looking camera to capture an initial image of the calibration object at the current flight altitude. The UAV processing chip analyzes the initial calibration object image using an image processing algorithm to identify the center coordinates of the calibration object in the camera coordinate system. This coordinate represents the pixel center position of the calibration object in the corresponding captured image, denoted as (u1, v1), where u1 is the coordinate of the center point of the first horizontal axis and v1 is the coordinate of the center point of the first vertical axis.

[0048] Furthermore, the camera center coordinates of the drone camera are obtained. The camera center coordinates represent the geometric center of the camera imaging sensor or the image frame itself in the pixel coordinate system. It is usually the origin of the camera coordinate system, denoted as (u0, y0), where u0 is the coordinate of the center point of the second horizontal axis and v0 is the coordinate of the center point of the second vertical axis.

[0049] Furthermore, by comparing the deviations of the calibration object's center coordinates from the camera's center coordinates along the horizontal and vertical axes, it is determined whether the calibration object is within the desired shooting range. The lateral distance difference is calculated based on the coordinates of the first and second horizontal axis center points, and the longitudinal distance difference is calculated based on the coordinates of the first and second vertical axis center points. Only when the lateral distance difference is less than a preset lateral distance difference and the longitudinal distance difference is less than a preset longitudinal distance difference is the calibration object considered to be within a preset field of view. Only then will the multiple sensor heights of the calibration object captured by the drone's sensors, and the corresponding images of the calibration object at each sensor height, be acquired.

[0050] For example, as shown in Figure 4, which is another schematic diagram of the calibration object image provided in an embodiment of this application, the drone may deviate from its vertical takeoff position due to uneven propeller control during takeoff. To ensure the drone can capture the ArUco marker within the optimal camera field of view and avoid oscillating flight to better estimate its distance from the ground sensor, the drone needs to be guided to the center region of the camera image. In pixel space, the camera center coordinates are C(u... c v c The coordinates of the calibration object center are generally half the image size; the coordinates of the calibration object center are T(u). t v t The marker is obtained through the ArUco detection operator during visual computing; the following formula is used to determine whether the marker is within the expected shooting range: ; Where ut represents the coordinates of the center point of the first horizontal axis; vt represents the coordinates of the center point of the first vertical axis; uc represents the coordinates of the center point of the second horizontal axis; and vc represents the coordinates of the center point of the second vertical axis. Indicates the difference in lateral distance; This represents the difference in longitudinal distance.

[0051] It is understood that by defining a preset field of view area, the embodiments of this application ensure that the starting position of subsequent data acquisition is optimal, thereby laying the foundation for obtaining high-quality calibration data and significantly improving the reliability of the calibration method proposed in the embodiments of this application.

[0052] In step 220, the camera intrinsic parameters of the UAV and the coordinates of multiple feature points of the calibration object in real space are obtained.

[0053] Camera intrinsic parameters refer to the internal parameters describing the imaging characteristics of the camera itself, mainly including focal length, principal point coordinates, lens distortion coefficient, and tangential distortion. Camera intrinsic parameters reflect how light is projected from three-dimensional space onto the two-dimensional image plane, and are the basis for converting pixel coordinates in the image to real-space orientation or for pose calculation. The coordinates of multiple feature points in real space refer to the coordinates of multiple feature points of the calibration object in the world coordinate system. Since the physical dimensions of the calibration object are known in advance, the coordinates of these feature points are also predetermined.

[0054] In some embodiments, obtaining the coordinates of multiple feature points of the calibration object in real space includes: (2.1) obtaining the size information of the calibration object and the origin of the real coordinates of the real space where the calibration object is located; (2.2) taking multiple corner points of the calibration object as corresponding feature points and determining the relative positional relationship between each corner point and the origin of the real coordinates; (2.3) determining the corner point coordinates corresponding to each corner point according to the size information and the relative positional relationship corresponding to each corner point, so as to obtain the coordinates of multiple feature points of the calibration object in real space.

[0055] In some embodiments, to determine the coordinates of feature points, it is first necessary to obtain the prior physical properties related to the calibration object: one is the size information of the calibration object, and the other is the origin of the real coordinate system corresponding to the real space. For example, if the calibration object is a square ArUco QR code, its size information is its side length; the origin of the real coordinate system is the origin of the real coordinate system set for the real space where the calibration object is located, so as to establish a local three-dimensional coordinate system on the calibration object itself. Usually, the origin of the real coordinate system is (0,0,0). In order to generate feature point coordinates more conveniently, the origin of the real coordinate system is usually selected as the geometric center of the calibration object or a specific corner point.

[0056] Furthermore, the corner points of a calibration object refer to feature points on the calibration object (such as ArUco QR codes, checkerboard patterns, etc.) that have clear geometric definitions and are easy to detect with high precision in images. They are usually located at the vertices of the intersection of black and white patterns. For example, in a square ArUco code, the corner points are the vertices of its four outer borders; in a checkerboard pattern, the corner points are the interior corners formed by the intersection of black and white squares. Since the corner points of a calibration object have high contrast, strong local texture variations, and good repeatability in the image, their positions can still be accurately located under different viewing angles and lighting conditions. Moreover, they have clear geometric definitions and fixed physical coordinates on the calibration object. Therefore, this application uses the corner points of the calibration object as feature points connecting the image pixel coordinates and the real-world three-dimensional coordinates.

[0057] Furthermore, the relative positional relationship between each selected feature point and the true coordinate origin is determined. The relative positional relationship describes the orientation relationship from the true coordinate origin to each corner point. For example, if the geometric center of the calibration object is selected as the true coordinate origin, then the four corner points of the calibration object are located at the upper left, upper right, lower left, and lower right of the true coordinate origin, respectively.

[0058] Furthermore, based on the relevant positional relationships and the obtained calibration object size information, the precise three-dimensional coordinates of each corner point in the real coordinate system are calculated, i.e., the corner point coordinates corresponding to each corner point. By combining the corner point coordinates corresponding to all corner points, the coordinates of multiple feature points of the calibration object in real space are obtained.

[0059] In step 230, for each calibration object image, the pixel coordinates corresponding to each feature point coordinate are determined based on the calibration object image, so as to obtain the pixel coordinates corresponding to each feature point coordinate in each calibration object image.

[0060] In some embodiments, the pixel coordinates corresponding to each feature point coordinate are determined from each calibration object image to obtain the pixel coordinates corresponding to each feature point coordinate in each calibration object image. Pixel coordinates refer to the coordinates of a certain feature point coordinate in each calibration object image. Pixel coordinates can be determined by a feature detection algorithm and are usually represented by coordinates (u, v), where u is the horizontal (column) index and v is the vertical (row) index.

[0061] In step 240, for each calibration object image, the camera height between the UAV camera and the calibration object is determined based on the camera intrinsic parameters, the coordinates of each feature point, and the coordinates of each pixel point, so as to obtain the camera height corresponding to each calibration object image.

[0062] Here, camera height refers to the vertical distance between the downward-looking camera and the calibration object. Specifically, using known camera intrinsic parameters, the coordinates of each feature point, and the corresponding pixel coordinates, the Perspective-n-Point Algorithm (PnP) is used to determine the camera height between the UAV's camera and the calibration object in the calibration object image.

[0063] In some embodiments, determining the camera height between the UAV camera and the calibration object in the calibration object image based on camera intrinsic parameters, coordinates of each feature point, and coordinates of each pixel point includes: (3.1) obtaining a preset scale factor; (3.2) determining the pose of the UAV in the calibration object image based on the scale factor, camera intrinsic parameters, coordinates of each feature point, and coordinates of each pixel point, and determining the camera height between the UAV camera and the calibration object based on the pose.

[0064] In some embodiments, the PnP algorithm estimates the camera pose or target object pose based on the mapping relationship between two-dimensional points in the pixel space of multiple cameras and three-dimensional points in the physical space. The solution of the PnP algorithm is generally given the camera intrinsic parameters, the pixel coordinates and world coordinates of the feature points, and is divided into n<3, 3≤n≤5 and n>6 according to the number of 2D-3D feature point pairs (n). In general, the pose is solved when n>6.

[0065] For example, as shown in the following formula, the pose of the UAV in the calibration image is determined based on the scale factor, camera intrinsic parameters, coordinates of each feature point, and coordinates of each pixel point, and then the camera height between the UAV's camera and the calibration object is determined based on the pose: .

[0066] Where s represents the scale factor; X, Y, and Z represent the coordinates of the feature points, and in this embodiment, Z is 0; u and v represent the coordinates of the pixel points; and the a matrix is ​​composed of camera intrinsic parameters and camera extrinsic parameters.

[0067] Among them, camera extrinsic parameters are parameters describing the camera's pose in the world coordinate system, including the rotation matrix and translation parameters. When the calibration object is placed on a horizontal surface and its coordinate system origin is set at the center of the calibration object, with the Z-axis pointing vertically upwards, the translation vector t = [t x , t y , t z ] t in z The component is the camera height, which is the vertical distance from the optical center of the downward-looking camera to the calibration plane.

[0068] It is understood that, in the UAV calibration process, this embodiment of the application, having acquired prior information about the calibration object, utilizes a visual pose estimation algorithm to accurately inversely determine the camera's height in three-dimensional space from a single two-dimensional image. This allows for precise determination of the corresponding camera height for each sensor height, providing a precise data foundation for subsequent UAV calibration based on the difference between multiple sensor heights and the camera height. Furthermore, it should be noted that the camera height is typically not directly determined using a visual pose estimation algorithm in actual UAV applications, as this algorithm relies on multiple definite target points of the target object, while sensors can continuously operate in environments without visual features or calibration objects.

[0069] In step 250, the distance mapping relationship between the drone's sensors and cameras and the distances to the same object is determined based on the height of each sensor and the camera height corresponding to the sensor height, and the drone's cameras and sensors are calibrated based on the distance mapping relationship.

[0070] In some embodiments, a data point set is formed by pairing each sensor altitude with its corresponding camera altitude value. Then, a distance mapping relationship describing the relationship between the two is determined by mathematically fitting this data point set, such as through linear regression analysis. This distance mapping relationship describes the inherent conversion rule between the values ​​measured by the UAV's sensors and camera when measuring the distance to the same object. Calibrating the UAV's camera and sensors based on this determined distance mapping relationship means that the relevant parameters are embedded into the UAV's control system. Subsequently, during flight, the UAV only needs to read the real-time altitude of the sensors to quickly and accurately calculate the altitude value in the camera coordinate system using this mapping relationship, providing precise scale information for tasks such as optical flow odometry.

[0071] It is understood that the embodiments of this application greatly reduce the difficulty of operation and the requirements for flight environment stability by simplifying the complex calibration process into an automated vertical flight, avoiding interference from factors such as wind disturbance. Moreover, by collecting multiple sets of data points in the continuous vertical motion trajectory, a distance mapping relationship between sensor height and camera height is established. Ultimately, this mapping relationship enables the UAV to accurately convert the easily obtainable sensor height into the camera height required by the visual algorithm in real time when performing a series of tasks such as optical flow odometry and visual positioning. This significantly improves the overall accuracy and reliability of pose estimation and environmental perception of the UAV in the case of Global Positioning System (GPS) denied environment.

[0072] In some embodiments, the distance mapping relationship between the UAV's sensors and cameras and the same object is determined based on each sensor height and the camera height corresponding to the sensor height, including: (4.1) obtaining a preset distance linear function, the distance linear function including an initial distance coefficient and an initial distance constant; (4.2) updating the initial distance coefficient and the initial distance constant based on each sensor height and the camera height corresponding to the sensor height to obtain a target distance coefficient and a target distance constant; (4.3) determining the distance mapping relationship between the UAV's sensors and cameras and the same object based on the target distance coefficient and the target distance constant.

[0073] In some embodiments, the distance linear function is a mathematical model used to describe the relationship between sensor height and camera height. For example, the distance linear function can be z = ah + b, where the initial value of a is the initial distance coefficient, the initial value of b is the initial distance constant, z represents the camera height, and h represents the corresponding sensor height.

[0074] Furthermore, using all previously acquired sensor-camera height pairs, the initial distance coefficients and initial distance constants are updated to find a set of optimal coefficient and constant values, such that the linear function defined by them can best fit all data points, i.e., minimize the overall error of all data points to the line, thus obtaining the target distance coefficients and target distance constants. Further, the value of 'a' is fixed according to the target distance coefficients, and the value of 'b' is fixed according to the target distance constants, thereby determining the distance mapping relationship z=ah+b between the distances of the UAV's sensors and cameras to the same object.

[0075] It is understood that the distance mapping relationship ultimately determined in the embodiments of this application is an optimized mathematical model that best represents the relationship between the sensor and the camera. This model enables the UAV to quickly and accurately convert the direct readings of the sensor into the camera altitude required for optical flow odometer calculation during subsequent flights, thereby providing a reliable basis for achieving high-precision pose estimation.

[0076] In some embodiments, after calibrating the camera and sensor of the UAV based on the distance mapping relationship, the method further includes: (5.1) obtaining the initial movement distance of the UAV; (5.2) controlling the UAV to fly horizontally, obtaining multiple ground altitude sensor values ​​captured by the UAV's sensors, and a target image containing the same ground feature point captured at each altitude sensor value; (5.3) obtaining the target pixel coordinates of the ground feature point in each target image, and updating the initial movement distance to obtain the flight distance based on each altitude sensor value and the target pixel coordinates of the ground feature point at each target sensing altitude; (5.4) using the flight distance as the new initial movement distance, and returning to the step of obtaining multiple ground altitude sensor values ​​captured by the UAV's sensors, and a target image containing the same ground feature point captured at each altitude sensor value, until the UAV reaches the target position, and using the last updated flight distance as the target movement distance of the UAV.

[0077] In some embodiments, after the calibration of the UAV is completed, the UAV can be put into practical application, such as when the UAV is performing an optical flow odometry task: the initial movement distance is obtained. The initial movement distance is the starting point for the odometry cumulative calculation. It is usually initialized to a zero value at the start of flight and serves as the cumulative reference. The subsequent calculated displacement increments of the UAV will be accumulated on this basis.

[0078] Furthermore, during the flight of the drone, the sensor altitude is acquired at different times (usually two different times), and the corresponding target image is captured at each sensor altitude. What these target images have in common is that they contain the same ground feature point in the real world. In this way, by using the optical flow transformation of the same ground feature point between different target images, the displacement of the drone can be determined, thereby realizing the odometry task.

[0079] Specifically, the coordinates of the target pixel of the ground feature point in each target image are obtained, and the change in the target pixel coordinates of the feature point is calculated. Then, based on each altitude sensor value and the target pixel coordinates of the ground feature point at each target sensing altitude, the actual three-dimensional movement distance corresponding to the change in the target pixel coordinates is determined, so as to obtain the flight distance by updating the initial movement distance.

[0080] Furthermore, after completing one distance update, the flight distance is used as the new initial movement distance, and the process returns to step 5.2 to continue acquiring new altitude sensor values ​​and target images. The calculation and update process in step 5.3 is repeated, ensuring that the UAV's movement distance is estimated and updated continuously in real time. This iterative process continues until the UAV reaches its preset final target position. At the end of the mission, the UAV processing chip uses the flight distance obtained from the last update as the final result of this flight and determines it as the UAV's target movement distance.

[0081] It is understood that the UAV calibrated through the embodiments of this application can serve as a visual optical flow odometry. In the absence of GPS signals, it can accurately perceive and track its own displacement by relying solely on onboard sensors and cameras, providing reliable location information for autonomous navigation, path planning, and precision operations, and greatly enhancing the UAV's autonomous operation capabilities in complex environments.

[0082] In some embodiments, the initial movement distance is updated to obtain the flight distance based on each altitude sensor value and the target pixel coordinates corresponding to the ground feature point at each target sensing altitude, including: (5.3.1) For each altitude sensor value, the spatial position of the UAV is determined based on the camera intrinsic parameters, the altitude sensor value, and the target pixel coordinates corresponding to the ground feature point at the target sensing altitude; (5.3.2) The additional movement distance of the UAV is determined based on the spatial position of the UAV determined at different altitude sensor values; (5.3.3) The initial movement distance is updated based on the additional movement distance to obtain the flight distance.

[0083] In some embodiments, geometric calculations are performed on each acquired altitude sensor value and its corresponding target image. The purpose is to determine the real spatial location of the UAV when capturing the corresponding target image by using camera intrinsic parameters, altitude sensor values, and the target pixel coordinates corresponding to ground feature points at the target sensing altitude.

[0084] Furthermore, the displacement increment is calculated by comparing spatial position information at consecutive time points. Specifically, based on the spatial position of the UAV determined at different altitude sensor values ​​(i.e., at different time points), the vector difference between the current spatial position and the previous spatial position is calculated, thereby determining the displacement of the UAV between these two time points. This displacement is the additional distance the UAV moves during this time interval, accurately quantifying the movement of the UAV during horizontal flight. Further, the initial movement distance is updated using the additional movement distance to obtain the flight distance.

[0085] To help readers better understand the calibration method of the UAV optical flow module proposed in this application, a complete example is given below: As shown in Figure 5, Figure 5 is a UAV flight diagram of the UAV optical flow module calibration method provided in this application. A UAV calibration system consists of a UAV 1 and a calibration object 2. The UAV flies vertically from a low altitude to a high altitude at a constant speed, and synchronously records the sensor altitude and the corresponding camera altitude to achieve UAV calibration. The specific steps of UAV calibration include: (1) starting the router, UAV, and computer in sequence, ensuring that the UAV and computer are connected to the same router; the router provides a communication link between the UAV and the computer; the UAV sends the UAV status and images to the computer through the router; after the computer obtains the UAV status information and images, it performs image processing and can send control commands.

[0086] (2) Place the drone and the ArUco QR code (select ID 0) on the ground respectively, with the drone on the QR code mark. The ArUco QR code is laid flat on a horizontal surface. In order to ensure the accuracy of the recognition result, the QR code is not deformed and is laid flat on the ground or a table. The drone is generally initialized to fly vertically. After the flight, the QR code needs to be in the camera's field of view. Therefore, the drone needs to be placed on the ArUco QR code.

[0087] (3) The computer starts the drone for automatic takeoff (e.g., 80cm) and guides the flight according to the ArUco QR code using the downward-facing camera. The drone initializes vertical flight and flies to the specified altitude in order to capture the ArUco QR code image; the specified altitude is set so that the downward-facing camera of the drone can fully capture the QR code image after takeoff; and guided centering flight is carried out according to the drone mark centering control strategy.

[0088] (4) After the QR code in the image is centered, the UAV descends to a certain height (e.g., 50cm) and simultaneously records the sensor height and the corresponding camera height. When the ArUco QR code image is in the center area of ​​the downward-looking camera image, the UAV descends to a certain height according to the size of the ArUco QR code, and the center point of the ArUco QR code image is still in the center area of ​​the downward-looking camera image; the current flight altitude of the UAV is used as the initial calibration position of the UAV optical flow module; and the sensor height is recorded by the UAV's rangefinder, and the corresponding camera height is determined by the image recorded by the UAV's camera.

[0089] (5) The UAV ascends slowly at a constant vertical speed (e.g., 20 mm / s), stops flying when it reaches a designated height (e.g., 1.5 m), completes data collection, and lands. To ensure the uniformity of data distribution and the clarity of the scene captured by the camera, the UAV needs to ascend slowly. When the UAV flies to the designated height, it cannot obtain the best QR code estimate if it exceeds this height. When it reaches this height, the UAV stops flying and descends at a certain speed (e.g., 40 mm / s), landing on the ground or a table.

[0090] (6) Based on the synchronously recorded data, determine the distance mapping relationship between the UAV's sensors and cameras and the distance to the same object, so as to achieve the calibration of the UAV.

[0091] As shown in Figure 6, Figure 6 is a schematic diagram of the module structure of the drone calibration device provided in this application embodiment. The drone-based calibration device 300 may include the following modules 310 to 350: a first acquisition module 310, used to control the drone to fly vertically above a preset calibration object, and when the calibration object is located in the preset field of view area of ​​the drone's camera, acquire multiple sensor heights of the calibration object captured by the drone's sensors, and the corresponding calibration object images captured at each sensor height; a second acquisition module 320, used to acquire the camera intrinsic parameters of the drone and the coordinates of multiple feature points of the calibration object in real space; and a pixel point coordinate determination module 330, used to determine the pixel coordinates for each calibration object image according to the specified coordinates. The calibration module 340 determines the pixel coordinates corresponding to each feature point coordinate in each calibration image to obtain the pixel coordinates corresponding to each feature point coordinate in each calibration image; the camera height determination module 340 is used to determine the camera height between the UAV camera and the calibration object in each calibration image based on the camera intrinsic parameters, each feature point coordinate, and each pixel coordinate, to obtain the camera height corresponding to each calibration image; the calibration module 350 is used to determine the distance mapping relationship between the distances of the UAV's sensors and cameras to the same object based on each sensor height and the camera height corresponding to the sensor height, and calibrate the UAV's camera and sensors based on the distance mapping relationship.

[0092] In some embodiments, the first acquisition module 310 is configured to: acquire an initial calibration object image captured at the current flight altitude, and acquire the calibration object center coordinates corresponding to the calibration object in the initial calibration object image, the calibration object center coordinates including the first horizontal axis center point coordinates and the first vertical axis center point coordinates; acquire the camera center coordinates of the UAV's camera, the camera center coordinates including the second horizontal axis center point coordinates and the second vertical axis center point coordinates; when the difference in lateral distance between the first horizontal axis center point coordinates and the second horizontal axis center point coordinates is less than a preset lateral distance difference, and the difference in longitudinal distance between the first vertical axis center point coordinates and the second vertical axis center point coordinates is less than a preset longitudinal distance difference, determine that the calibration object is located in a preset field of view area of ​​the UAV's camera, and acquire multiple sensor heights of the calibration object captured by the UAV's sensors, and the corresponding calibration object image captured at each sensor height.

[0093] In some embodiments, the second acquisition module 320 is used to: acquire the size information of the calibration object and the origin of the real coordinates of the real space where the calibration object is located; take multiple corner points of the calibration object as corresponding feature points and determine the relative positional relationship between each corner point and the origin of the real coordinates; and determine the corner point coordinates corresponding to each corner point according to the size information and the relative positional relationship corresponding to each corner point, so as to obtain the coordinates of multiple feature points of the calibration object in the real space.

[0094] In some embodiments, the camera height determination module 340 is used to: obtain a preset scale factor; determine the pose of the UAV in the calibration image based on the scale factor, camera intrinsic parameters, coordinates of each feature point and coordinates of each pixel point, and determine the camera height between the UAV's camera and the calibration object based on the pose.

[0095] In some embodiments, the calibration module 350 is configured to: obtain a preset distance linear function, the distance linear function including an initial distance coefficient and an initial distance constant; update the initial distance coefficient and the initial distance constant according to each sensor height and the camera height corresponding to the sensor height to obtain a target distance coefficient and a target distance constant; and determine the distance mapping relationship between the distances of the UAV's sensors and cameras to the same object according to the target distance coefficient and the target distance constant.

[0096] In some embodiments, the calibration module 350 is further configured to: obtain the initial movement distance of the UAV; control the UAV to fly horizontally, obtain multiple ground altitude sensor values ​​captured by the UAV's sensors, and target images containing the same ground feature point captured at each altitude sensor value; obtain the target pixel coordinates of the ground feature point in each target image, and update the initial movement distance to obtain the flight distance based on each altitude sensor value and the target pixel coordinates of the ground feature point at each target sensing altitude; use the flight distance as the new initial movement distance, and return to the steps of obtaining multiple ground altitude sensor values ​​captured by the UAV's sensors, and target images containing the same ground feature point captured at each altitude sensor value, until the UAV reaches the target position, and use the last updated flight distance as the target movement distance of the UAV.

[0097] In some embodiments, the calibration module 350 is further configured to: determine the spatial position of the UAV for each altitude sensor value based on camera intrinsic parameters, altitude sensor value, and the target pixel coordinates corresponding to ground feature points at the target sensing altitude; determine the additional movement distance of the UAV based on the spatial position of the UAV determined at different altitude sensor values; and update the initial movement distance based on the additional movement distance to obtain the flight distance.

[0098] The calibration method, apparatus, device, and storage medium for the UAV optical flow module proposed in this application control the UAV to fly vertically above a preset calibration object. When the calibration object is located within a preset field of view of the UAV's camera, the method acquires multiple sensor heights of the UAV's sensors to the calibration object, and corresponding images of the calibration object captured at each sensor height. It also acquires the UAV's camera intrinsic parameters and the coordinates of multiple feature points of the calibration object in real space. For each calibration object image, it determines the pixel coordinates corresponding to each feature point coordinate, thus obtaining the pixel coordinates corresponding to each feature point coordinate in each calibration object image. For each calibration object image, it determines the camera height between the UAV's camera and the calibration object based on the camera intrinsic parameters, each feature point coordinate, and each pixel coordinate, thus obtaining the camera height corresponding to each calibration object image. Finally, it determines the distance mapping relationship between the distances of the UAV's sensors and camera to the same object based on each sensor height and the corresponding camera height, and calibrates the UAV's camera and sensors based on this distance mapping relationship.

[0099] This application addresses the technical challenge of low calibration accuracy in UAVs due to difficulties in maintaining stable horizontal flight and susceptibility to wind disturbances. It transforms the complex horizontal flight calibration process into vertical flight calibration and establishes a distance mapping relationship between the distances of the UAV's sensors and cameras to the same object by using multiple sets of sensor heights and corresponding camera heights. This enables calibration between the UAV's sensors and cameras. Because this application avoids the stringent reliance on high-precision horizontal displacement control and long-term stable hovering found in related technologies, it improves the calibration accuracy of UAVs. Furthermore, the calibrated UAV can accurately convert sensor data into scale information in the camera coordinate system, thus improving the UAV's pose estimation and environmental perception, thereby enhancing the accuracy of the UAV in performing a series of tasks such as visual positioning, odometry, autonomous navigation, and precise landing.

[0100] As shown in Figure 7, which is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of this application, the electronic device includes: a processor 401, which can be implemented in the form of a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solution provided in the embodiment of this application; and a memory 402, which can be implemented in the form of read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 402 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 402 and is called and executed by the processor 401. The input / output interface 403 is used to implement information input and output. The communication interface 404 is used to realize communication interaction between this device and other devices. Communication can be realized by wired means (e.g., USB, network cable, etc.) or by wireless means (e.g., mobile network, WIFI, Bluetooth, etc.). The bus 405 transmits information between the various components of the device (e.g., processor 401, memory 402, input / output interface 403 and communication interface 404). The processor 401, memory 402, input / output interface 403 and communication interface 404 realize communication connection between each other within the device through the bus 405.

[0101] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described image analysis method.

[0102] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0103] The calibration method, apparatus, device, and storage medium for the UAV optical flow module proposed in this application control the UAV to fly vertically above a preset calibration object. When the calibration object is located within a preset field of view of the UAV's camera, the method acquires multiple sensor heights of the UAV's sensors to the calibration object, and corresponding images of the calibration object captured at each sensor height. It also acquires the UAV's camera intrinsic parameters and the coordinates of multiple feature points of the calibration object in real space. For each calibration object image, it determines the pixel coordinates corresponding to each feature point coordinate, thus obtaining the pixel coordinates corresponding to each feature point coordinate in each calibration object image. For each calibration object image, it determines the camera height between the UAV's camera and the calibration object based on the camera intrinsic parameters, each feature point coordinate, and each pixel coordinate, thus obtaining the camera height corresponding to each calibration object image. Finally, it determines the distance mapping relationship between the distances of the UAV's sensors and camera to the same object based on each sensor height and the corresponding camera height, and calibrates the UAV's camera and sensors based on this distance mapping relationship.

[0104] This application addresses the technical challenge of low calibration accuracy in UAVs due to difficulties in maintaining stable horizontal flight and susceptibility to wind disturbances. It transforms the complex horizontal flight calibration process into vertical flight calibration and establishes a distance mapping relationship between the distances of the UAV's sensors and cameras to the same object by using multiple sets of sensor heights and corresponding camera heights. This enables calibration between the UAV's sensors and cameras. Because this application avoids the stringent reliance on high-precision horizontal displacement control and long-term stable hovering found in related technologies, it improves the calibration accuracy of UAVs. Furthermore, the calibrated UAV can accurately convert sensor data into scale information in the camera coordinate system, thus improving the UAV's pose estimation and environmental perception, thereby enhancing the accuracy of the UAV in performing a series of tasks such as visual positioning, odometry, autonomous navigation, and precise landing.

[0105] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0106] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0107] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0108] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0109] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0110] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0111] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0112] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0113] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0114] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0115] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A calibration method for an optical flow module of an unmanned aerial vehicle (UAV), characterized in that, The method includes: controlling a drone to fly vertically above a preset calibration object; when the calibration object is located within a preset field of view of the drone's camera, acquiring multiple sensor heights of the calibration object captured by the drone's sensors, and corresponding calibration object images captured at each sensor height; acquiring the drone's camera intrinsic parameters and multiple feature point coordinates of the calibration object in real space; for each calibration object image, determining the pixel coordinates corresponding to each feature point coordinate based on the calibration object image, to obtain the pixel coordinates corresponding to each feature point coordinate in each calibration object image; for each calibration object image, determining the camera height between the drone's camera and the calibration object in the calibration object image based on the camera intrinsic parameters, each feature point coordinate, and each pixel coordinate, to obtain the camera height corresponding to each calibration object image; determining the distance mapping relationship between the drone's sensors and cameras and the distance to the same object based on each sensor height and the corresponding camera height, and calibrating the drone's camera and sensors based on the distance mapping relationship.

2. The calibration method for the optical flow module of a UAV according to claim 1, characterized in that, When the calibration object is located within a preset field of view of the UAV's camera, acquiring multiple sensor heights of the calibration object captured by the UAV's sensors, and corresponding calibration object images captured at each sensor height, includes: acquiring an initial calibration object image captured at the current flight altitude, and acquiring the calibration object center coordinates corresponding to the calibration object in the initial calibration object image, the calibration object center coordinates including a first horizontal axis center point coordinate and a first vertical axis center point coordinate; acquiring the camera center coordinates of the UAV's camera, the camera center coordinates including a second horizontal axis center point coordinate and a second vertical axis center point coordinate; when the lateral distance difference between the first horizontal axis center point coordinate and the second horizontal axis center point coordinate is less than a preset lateral distance difference, and the longitudinal distance difference between the first vertical axis center point coordinate and the second vertical axis center point coordinate is less than a preset longitudinal distance difference, determining that the calibration object is located within a preset field of view of the UAV's camera, and acquiring multiple sensor heights of the calibration object captured by the UAV's sensors, and corresponding calibration object images captured at each sensor height.

3. The calibration method for the optical flow module of a UAV according to claim 1, characterized in that, The step of obtaining the coordinates of multiple feature points of the calibration object in real space includes: obtaining the size information of the calibration object and the origin of the real coordinate system of the real space where the calibration object is located; taking multiple corner points of the calibration object as corresponding feature points and determining the relative positional relationship between each corner point and the origin of the real coordinate system; and determining the corner point coordinates corresponding to each corner point based on the size information and the relative positional relationship of each corner point, so as to obtain the coordinates of multiple feature points of the calibration object in real space.

4. The calibration method for a UAV according to claim 1, characterized in that, The step of determining the camera height between the UAV's camera and the calibration object in the calibration object image based on the camera intrinsic parameters, the coordinates of each feature point, and the coordinates of each pixel point includes: obtaining a preset scale factor; determining the pose of the UAV in the calibration object image based on the scale factor, the camera intrinsic parameters, the coordinates of each feature point, and the coordinates of each pixel point; and determining the camera height between the UAV's camera and the calibration object based on the pose.

5. The calibration method for the optical flow module of a UAV according to claim 1, characterized in that, The step of determining the distance mapping relationship between the drone's sensors and cameras and the same object based on each sensor height and the corresponding camera height includes: obtaining a preset distance linear function, the distance linear function including an initial distance coefficient and an initial distance constant; updating the initial distance coefficient and the initial distance constant based on each sensor height and the corresponding camera height to obtain a target distance coefficient and a target distance constant; and determining the distance mapping relationship between the drone's sensors and cameras and the same object based on the target distance coefficient and the target distance constant.

6. The calibration method for the optical flow module of a UAV according to claim 1, characterized in that, After calibrating the UAV's camera and sensors based on the distance mapping relationship, the method further includes: obtaining the initial movement distance of the UAV; controlling the UAV to fly horizontally, obtaining multiple ground altitude sensor values ​​captured by the UAV's sensors, and a target image containing the same ground feature point captured at each altitude sensor value; obtaining the target pixel coordinates of the ground feature point in each target image, and updating the initial movement distance to obtain the flight distance based on each altitude sensor value and the target pixel coordinates of the ground feature point at each target sensing altitude; using the flight distance as the new initial movement distance, and returning to execute the steps of obtaining multiple ground altitude sensor values ​​captured by the UAV's sensors, and a target image containing the same ground feature point captured at each altitude sensor value, until the UAV reaches the target position, and using the last updated flight distance as the target movement distance of the UAV.

7. The calibration method for the optical flow module of a UAV according to claim 6, characterized in that, The step of updating the initial movement distance to obtain the flight distance based on each altitude sensor value and the target pixel coordinates corresponding to the ground feature point at each target sensing altitude includes: for each altitude sensor value, determining the spatial position of the UAV based on the camera intrinsic parameters, the altitude sensor value, and the target pixel coordinates corresponding to the ground feature point at the target sensing altitude; determining the additional movement distance of the UAV based on the spatial position of the UAV determined at different altitude sensor values; and updating the initial movement distance to obtain the flight distance based on the additional movement distance.

8. A calibration device for a drone, characterized in that, include: The first acquisition module is used to control the drone to fly vertically above a preset benchmark. When the benchmark is located in the preset field of view area of ​​the drone's camera, the module acquires multiple sensor heights of the benchmark captured by the drone's sensors, and the corresponding image of the benchmark captured at each sensor height. The second acquisition module is used to acquire the camera intrinsic parameters of the UAV and the coordinates of multiple feature points of the calibration object in real space; the pixel coordinate determination module is used to determine the pixel coordinates corresponding to each feature point coordinate for each calibration object image, so as to obtain the pixel coordinates corresponding to each feature point coordinate in each calibration object image. The camera height determination module is used to determine the camera height between the UAV's camera and the calibration object in each calibration object image based on the camera intrinsic parameters, the coordinates of each feature point, and the coordinates of each pixel point, so as to obtain the camera height corresponding to each calibration object image; The calibration module is used to determine the distance mapping relationship between the distances of the drone's sensors and cameras to the same object based on the height of each sensor and the camera height corresponding to the sensor height, and to calibrate the drone's cameras and sensors based on the distance mapping relationship.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the calibration method for the UAV optical flow module according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the calibration method of the UAV optical flow module according to any one of claims 1 to 7.

Images