Unmanned aerial vehicle camera internal parameter repair method and electronic device

Abstract

The present disclosure provides a UAV camera internal parameter repair method and electronic equipment, and relates to the technical field of UAVs. The method comprises the following steps: acquiring flight images collected by an on-board camera and measured poses synchronously collected by a pose collection device when a UAV executes a flight task; recalibrating the internal parameters of the UAV based on the flight images and the measured poses to obtain algorithm-calibrated internal parameters; calculating the pose of the UAV when the flight images are collected as a theoretical pose based on original internal parameters currently used by the on-board camera and the flight images; comparing the theoretical pose with the measured pose to calculate a pose deviation; if the pose deviation exceeds a first preset threshold, it is determined that the internal parameters of the on-board camera have degenerated; and updating and repairing the internal parameters of the on-board camera based on the algorithm-calibrated internal parameters. The present disclosure can detect the internal parameters of the UAV camera and accurately complete the repair without sensing.

Description

Technical Field

[0001] This disclosure relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a method for repairing the intrinsic parameters of a UAV camera and an electronic device. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this disclosure as set forth in the claims. The description herein is not intended to be a prior art simply because it is included in this section.

[0003] Autonomous flight of drones relies on robust and accurate positioning. The core of this is visual positioning, which depends on accurate camera intrinsics. However, in real-world scenarios, the camera intrinsics of drones can change due to factors such as non-ideal initial calibration, mechanical vibration, structural deformation, or lens aging, leading to degradation of the camera intrinsics and consequently affecting the accuracy of global visual positioning.

[0004] Therefore, there is a need for an effective mechanism to detect changes in the intrinsic parameters of drones and for efficient and automated methods to calibrate the intrinsic parameters to achieve online repair. Summary of the Invention

[0005] The purpose of this disclosure is to provide a method and electronic device for repairing camera intrinsic parameters of a drone, which can automatically and imperceptibly detect camera intrinsic parameter degradation and complete accurate repair during the daily operation of the drone, effectively ensuring the long-term accuracy and reliability of the visual positioning system.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] This disclosure provides a method for repairing the intrinsic parameters of a drone camera, comprising: acquiring flight images captured by an airborne camera and measured poses simultaneously acquired by a pose acquisition device during the drone's flight mission; recalibrating the intrinsic parameters of the drone based on the flight images and the measured poses to obtain algorithm-calibrated intrinsic parameters; calculating the pose of the drone when acquiring the flight images based on the original intrinsic parameters currently used by the airborne camera and the flight images, as the theoretical pose; comparing the theoretical pose with the measured pose to calculate the pose deviation; if the pose deviation exceeds a first preset threshold, determining that the intrinsic parameters of the airborne camera have degraded; and updating and repairing the intrinsic parameters of the airborne camera based on the algorithm-calibrated intrinsic parameters.

[0008] In some embodiments, the method further includes: comparing the original intrinsic parameters with the algorithm calibration intrinsic parameters to determine the intrinsic parameter deviation; and determining that the intrinsic parameters of the airborne camera have degraded when the intrinsic parameter deviation exceeds a second preset threshold and the pose deviation exceeds a first preset threshold.

[0009] In some embodiments, the flight image includes a first image; wherein, based on the original intrinsic parameters currently used by the airborne camera and the flight image, calculating the pose of the UAV when acquiring the flight image as a theoretical pose includes: acquiring a positioning map corresponding to the flight mission, the positioning map including the three-dimensional coordinates of multiple physical objects in the environment and their corresponding image feature descriptors; the first image including a first physical object belonging to the multiple physical objects; performing feature matching processing on the first image based on the image feature descriptors in the positioning map to determine the three-dimensional coordinate values ​​of the first physical object and the two-dimensional coordinate values ​​of the first physical object in the first image; and determining the theoretical pose of the UAV when acquiring the first image based on the original intrinsic parameters, the three-dimensional coordinate values ​​of the first physical object, and the two-dimensional coordinate values ​​of the first physical object in the first image.

[0010] In some embodiments, at least one flight image contains a second physical object; wherein, based on the flight image and the measured pose, the intrinsic parameters of the UAV are recalibrated to obtain algorithm calibration intrinsic parameters, including: acquiring the three-dimensional coordinate values ​​of the second physical object; using the original intrinsic parameters of the UAV as initial intrinsic parameters, combining the three-dimensional coordinate values ​​of the second physical object and the measured pose of the UAV when acquiring the at least one flight image, determining the reprojection pixels corresponding to the second physical object in the at least one flight image; acquiring the observation pixels of the second physical object from the at least one flight image; determining the sum of errors between each reprojection pixel and the corresponding observation pixel as the reprojection error; minimizing the reprojection error with the intrinsic parameters as variables to obtain the algorithm calibration intrinsic parameters.

[0011] In some embodiments, the three-dimensional coordinates of the second physical object are obtained by the following method: performing feature matching and image tracking on multiple flight images acquired by the UAV during flight to determine the feature point tracking chain of the second physical object; taking the measured pose of the UAV when acquiring the at least one flight image as the prior pose; and triangulating the second physical object based on the prior pose and the feature point tracking chain to determine the three-dimensional coordinates of the second physical object.

[0012] In some embodiments, the flight image includes a second image, which includes multiple rows of pixels; wherein the second image is preprocessed by the following method: obtaining the angular velocity acquired by the UAV through an inertial measurement unit during flight; determining a rotation matrix corresponding to the acquisition of each row of pixels in the second image by the UAV based on the angular velocity; rotating each row of pixels in the second image based on the rotation matrix to compensate for each row of pixels in the second image; and preprocessing the second image based on the rotated pixels.

[0013] In some embodiments, preprocessing the second image based on the rotated pixels includes: determining the optical flow vector corresponding to each pixel in the second image based on the rotated pixels; wherein the optical flow vector is used to describe the displacement of the rotated pixel relative to the pixel before rotation; performing backward mapping on the pixels in the second image based on the optical flow vector; and obtaining the preprocessed second image based on the backward mapping result.

[0014] In some embodiments, the multiple rows of pixels in the second image include a first row of pixels; wherein, rotating each row of pixels in the second image based on the rotation matrix to compensate for each row of pixels in the second image includes: converting the pixel coordinates of the first row of pixels into a direction vector in the camera coordinate system; rotating the direction vector corresponding to the first row of pixels using the rotation matrix corresponding to the first row of pixels to obtain a rotated direction vector; and backprojecting the rotated direction vector back to the pixel coordinate system to obtain the compensated pixels corresponding to the first row, so as to preprocess the second image based on the compensated pixels.

[0015] In some embodiments, determining the rotation matrix corresponding to the UAV acquiring each row of pixels in the second image based on the angular velocity includes: determining the cumulative angular displacement corresponding to each row of the second image based on the angular velocity; and determining the rotation angular velocity corresponding to each row of the second image based on the cumulative angular displacement corresponding to each row.

[0016] This disclosure provides an internal parameter repair device for a drone camera, comprising: a flight image acquisition module, an algorithm calibration internal parameter determination module, a theoretical pose determination module, a pose deviation calculation module, an internal parameter degradation judgment module, and an internal parameter repair module.

[0017] The flight image acquisition module is used to acquire flight images captured by the airborne camera and measured poses synchronously acquired by the pose acquisition device when the UAV is performing a flight mission. The algorithm calibration intrinsic parameter determination module can be used to recalibrate the intrinsic parameters of the UAV based on the flight images and the measured poses to obtain algorithm calibration intrinsic parameters. The theoretical pose determination module can be used to calculate the pose of the UAV when acquiring the flight images based on the original intrinsic parameters currently used by the airborne camera and the flight images, as the theoretical pose. The pose deviation calculation module can be used to compare the theoretical pose with the measured pose to calculate the pose deviation. The intrinsic parameter degradation judgment module can be used to determine that the intrinsic parameters of the airborne camera have degraded if the pose deviation exceeds a first preset threshold. The intrinsic parameter repair module can be used to update and repair the intrinsic parameters of the airborne camera based on the algorithm calibration intrinsic parameters.

[0018] This disclosure provides an electronic device comprising: a memory and a processor; the memory for storing computer program instructions; and the processor for calling the computer program instructions stored in the memory to implement the UAV camera intrinsic parameter repair method described above.

[0019] This disclosure provides a computer-readable storage medium storing computer program instructions to implement the UAV camera intrinsic parameter repair method as described in any of the preceding embodiments.

[0020] This disclosure provides a computer program product or computer program that includes computer program instructions stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and the processor executes the computer program instructions to implement the aforementioned UAV camera intrinsic parameter repair method.

[0021] The drone camera intrinsic parameter repair method, apparatus, electronic device, computer-readable storage medium, and computer program product provided in this disclosure enable drones to automatically determine whether their camera calibration parameters have "drifted" while performing normal flight missions (such as delivery), and immediately use the latest flight data to calculate more accurate parameters for online update and repair. The entire process is fully automated, requiring no manual intervention or recall of drones for additional traditional calibration that relies on special scenarios (such as checkerboard patterns). This significantly improves maintenance efficiency and the long-term operational stability of the drone fleet while ensuring visual positioning accuracy.

[0022] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] Figure 1 A schematic diagram of a scenario is shown that can be applied to the drone camera intrinsic parameter repair method or drone camera intrinsic parameter repair device in the embodiments of this disclosure.

[0025] Figure 2 This is a flowchart illustrating a method for repairing the intrinsic parameters of a drone camera according to an exemplary embodiment.

[0026] Figure 3 This is a flowchart illustrating a method for repairing the intrinsic parameters of a drone camera according to an exemplary embodiment.

[0027] Figure 4 This is a flowchart illustrating a theoretical pose determination method according to an exemplary embodiment.

[0028] Figure 5 This is a flowchart illustrating an algorithm calibration intrinsic parameter determination method according to an exemplary embodiment.

[0029] Figure 6 This is a flowchart illustrating a method for determining three-dimensional coordinate values ​​according to an exemplary embodiment.

[0030] Figure 7 This is a flowchart illustrating an image preprocessing method according to an exemplary embodiment.

[0031] Figure 8 This is a flowchart illustrating an image preprocessing method according to an exemplary embodiment.

[0032] Figure 9 This is a flowchart illustrating a pixel processing method according to an exemplary embodiment.

[0033] Figure 10 This is a schematic diagram of a method for repairing the intrinsic parameters of a drone camera, according to an exemplary embodiment.

[0034] Figure 11 This is a block diagram illustrating a drone camera intrinsic parameter repair device according to an exemplary embodiment.

[0035] Figure 12 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0036] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0037] Those skilled in the art will recognize that embodiments of this disclosure can be a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0038] The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0039] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0040] The accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus omitting repeated descriptions of them. Some block diagrams shown in the drawings do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0041] The flowchart shown in the accompanying drawings is merely illustrative and does not necessarily include all content and steps, nor does it require execution in the described order. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0042] In the description of this disclosure, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences; the terms "contains," "includes," and "has" are used to indicate an open-ended meaning of inclusion and refer to the existence of additional elements / components / etc. besides those listed.

[0043] This disclosure embodiment can be implemented by a terminal and / or a server. The terminal can obtain data from a computer device and display that data. The computer device can interact with the terminal, and can be a server hosting the application, or it can belong to the terminal (i.e., the terminal's backend), etc., without limitation.

[0044] The terminal can be a mobile phone, a laptop computer, or a playback device in a vehicle, etc., without limitation. The terminal can be considered a playback device in a vehicle, and it can display the target application. The terminal is only one example of the devices listed; the terminal in this disclosure is not limited to the listed devices. The target application in this disclosure can be any application capable of displaying multimedia information.

[0045] It is understood that the terminal mentioned in the embodiments of this disclosure can be a computer device, including but not limited to a terminal or a server. In other words, the computer device can be a server or a terminal, or a system composed of a server and a terminal. The terminal mentioned above can be an electronic device, including but not limited to mobile phones, tablets, desktop computers, laptops, handheld computers, in-vehicle devices, augmented reality / virtual reality (AR / VR) devices, head-mounted displays, smart TVs, wearable devices, smart speakers, digital cameras, webcams, and other mobile internet devices (MIDs) with network access capabilities, or terminals in scenarios such as trains, ships, and flights.

[0046] The servers mentioned above can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, vehicle-road cooperation, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0047] Optionally, the data involved in the embodiments of this disclosure may be stored in a computer device or may be stored based on cloud storage technology, without limitation.

[0048] To better understand the above-mentioned objectives, features and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0049] The exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0050] Figure 1 A schematic diagram of a scenario is shown that can be applied to the drone camera intrinsic parameter repair method or drone camera intrinsic parameter repair device in the embodiments of this disclosure.

[0051] Please refer to Figure 1 The diagram illustrates an implementation environment provided by an exemplary embodiment of this disclosure.

[0052] like Figure 1As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0053] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, desktop computers, wearable devices, virtual reality devices, smart home devices, etc.

[0054] Server 105 can be a server that provides various services, such as a backend management server that supports the devices operated by users using terminal devices 101, 102, and 103. The backend management server can analyze and process received requests and other data, and feed the processing results back to the terminal devices.

[0055] A server can be a standalone physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. This disclosure does not impose any restrictions on this.

[0056] Server 105 may, for example, acquire flight images captured by an airborne camera and measured poses simultaneously acquired by a pose acquisition device during the UAV's flight mission; Server 105 may, for example, recalibrate the UAV's intrinsic parameters based on the flight images and measured poses to obtain algorithm calibration intrinsic parameters; Server 105 may, for example, calculate the UAV's pose when acquiring flight images based on the original intrinsic parameters currently used by the airborne camera and the flight images, as the theoretical pose; Server 105 may, for example, compare the theoretical pose with the measured pose to calculate the pose deviation; Server 105 may, for example, determine that the airborne camera's intrinsic parameters have degraded if the pose deviation exceeds a first preset threshold; Server 105 may, for example, update and repair the airborne camera's intrinsic parameters based on the algorithm calibration intrinsic parameters.

[0057] It should be understood that Figure 1The number of terminal devices, networks, and servers shown is merely illustrative. Server 105 can be a single physical server or a combination of multiple servers. Depending on actual needs, it can have any number of terminal devices, networks, and servers.

[0058] Under the above system architecture, this disclosure provides a method for repairing the intrinsic parameters of a drone camera, which can be executed by any electronic device with computing power.

[0059] Figure 2 This is a flowchart illustrating a method for repairing intrinsic parameters of a drone camera according to an exemplary embodiment. The method provided in this disclosure can be executed by any electronic device with computing power, for example, the method can be performed by the above-described... Figure 1 The execution can be performed by a server or terminal device in the embodiments, or it can be performed by both a server and a terminal device. In the following embodiments, the server is used as the execution subject for illustration, but this disclosure is not limited to this.

[0060] Reference Figure 2 The drone camera intrinsic parameter repair method provided in this disclosure may include the following steps.

[0061] Step S202: Acquire flight images captured by the airborne camera and measured poses simultaneously acquired by the pose acquisition device when the UAV is performing a flight mission.

[0062] A pose acquisition device can refer to a sensor module on a UAV used to measure its position (latitude and longitude, altitude) and attitude (pitch, roll, yaw) in three-dimensional space in real time and with high precision. For example, it can be an RTK-GNSS (Real-Time Kinematic Global Navigation Satellite System), or it can be a combination of RTK-GNSS and IMU (Inertial Measurement Unit).

[0063] An airborne camera can refer to an image acquisition device that is fixedly mounted on the fuselage of a drone (usually located on the belly or front of the drone) and used to continuously collect visual information about the flight path environment during flight missions.

[0064] Flight images can refer to a sequence of digital images captured continuously or at a preset frequency by an onboard camera (which can be a global shutter camera, a rolling shutter camera, or others) during a flight mission of an unmanned aerial vehicle (UAV). The aforementioned flight images can be two-dimensional images.

[0065] Measured pose refers to physical measurement data that reflects the real position and attitude of the UAV in three-dimensional space, collected in real time during flight by high-precision pose acquisition equipment on the UAV (such as a combination of RTK-GNSS and IMU).

[0066] Step S204: Based on the flight image and measured pose, the intrinsic parameters of the UAV are recalibrated to obtain the algorithm calibration intrinsic parameters.

[0067] In some embodiments, the process of intrinsic parameter recalibration based on flight images and measured poses can be achieved through joint optimization by fusing visual (flight images) and high-precision measurement data (measured poses). Specifically, the measured poses (such as RTK / IMU fusion data) can first be used to provide high-precision initial camera pose ground truth for each flight image, effectively avoiding the uncertainty in the initialization stage of traditional visual methods. Simultaneously, feature points are extracted from the image sequence and multi-view matching is performed to establish the trajectory chain of feature points. Subsequently, triangulation is performed based on the known initial pose and feature matching relationship to initially reconstruct the sparse 3D scene. Finally, a nonlinear optimization problem is constructed with the goal of minimizing the reprojection error. Camera intrinsic parameters (such as focal length, principal point, distortion coefficient, etc.), camera pose (fine-tuned near the measured pose), or 3D point coordinates are used as variables to be optimized and jointly solved. The final optimized intrinsic parameters are output as the algorithm calibration intrinsic parameters. This method significantly improves the robustness, convergence speed, and accuracy of calibration by introducing strongly constrained measured pose priors.

[0068] In some embodiments, a "real-time recursive calibration method" can also be used to determine the algorithm calibration intrinsic parameters. This method treats the camera intrinsic parameters as a continuously monitored "health status" and integrates it into the UAV's real-time navigation system. The core process is as follows: during flight, the UAV can perform a highly frequent cycle of "prediction-comparison-fine-tuning"—it uses the previous state and inertial data to predict the current image, then quickly compares the predicted image with the actual captured image and the precise real-time position provided by RTK; once a deviation is detected, the system not only corrects the position estimate but also makes a small, optimal adjustment to the current intrinsic parameter value based on a mathematical model. Through thousands of such instantaneous fine-tunings during flight, the intrinsic parameters are gradually and stably calibrated to the correct value, much like being precisely "tightened screws," thus achieving true "on-the-fly calibration."

[0069] This application does not restrict the implementation process of "recalibrating the intrinsic parameters of the UAV based on flight images and measured poses to obtain the algorithm calibration intrinsic parameters".

[0070] Step S206: Based on the original intrinsic parameters and flight images currently used by the airborne camera, calculate the pose of the UAV when acquiring flight images, and use it as the theoretical pose.

[0071] In some embodiments, the theoretical pose can be determined using the PnP (Perspective-n-Point) algorithm in visual positioning. Simply put, a pre-constructed positioning map can be used (this map may contain a large number of feature points with known 3D spatial locations and their corresponding 2D image descriptions (such as image descriptors / image description features)). When the drone captures a new flight image, feature points can be extracted from the image and quickly matched with features in the map, thus establishing a series of correspondences between numerous 2D pixels in the image and 3D spatial points in the map. With these "2D-3D point pairs" and the original intrinsic parameters used by the onboard camera, the PnP algorithm can accurately calculate the position and orientation of the drone camera in space when the image was captured, much like solving an equation; this calculated pose is the theoretical pose.

[0072] For example, for shots that may be degraded, the corresponding drone index can be used to retrieve the pre-generated positioning map composed of image feature points and 3D points for that route. .in, It is a three-dimensional eigenvalue. It is a two-dimensional eigenvalue.

[0073] With location maps constructed from different flights on the same route, features can be extracted from the image of the currently calibrated flight data and matched with map feature points to construct a location map. Figure 3 Point D With image 2D points Correspondence: .

[0074] Given 3D points With image 2D points Correspondence and camera intrinsic parameters The pose of the current image can then be solved using the PnP algorithm. .

[0075] Step S208: Compare the theoretical pose with the measured pose and calculate the pose deviation.

[0076] Based on the above principles, the old UAV intrinsic parameters are used to solve the pose corresponding to each image using the PnP algorithm. By comparing the pose solved by the PnP algorithm with the high-precision RTK pose carried by the UAV, and solving for the corresponding positioning error, the accuracy of the UAV intrinsic parameter solution can be evaluated.

[0077] For the same image, the difference in pose obtained by PnP arises solely from the accuracy of the intrinsic parameter calibration. First, Gaussian fitting is performed on the positioning error of PnP using the old intrinsic parameters, and the mean values ​​in the x, y, and z directions are determined: .

[0078] At this point, the mean positioning error in each of the three directions indicates whether the PnP positioning suffers from constant bias in the three directions due to inaccurate intrinsic parameters. Simultaneously, the mean values ​​in the three directions... Find the 2-norm, which characterizes the global constant bias: .

[0079] Determine whether the overall constant bias exceeds a preset threshold. If there is a significant difference between the algorithm calibration intrinsic parameters and the current original intrinsic parameters, and the positioning constant bias exceeds the threshold, it can be determined that the drone lens has degraded, and the original intrinsic parameters are no longer applicable.

[0080] In step S210, if the pose deviation exceeds the first preset threshold, it is determined that the intrinsic parameters of the airborne camera have degraded.

[0081] Step S212: Based on the algorithm calibration intrinsic parameters, update and repair the intrinsic parameters of the airborne camera.

[0082] In some embodiments, new intrinsic parameters (such as algorithm calibration intrinsic parameters) can be used to solve the PnP pose, and the same method can be applied to statistically determine the constant deviations of the three axes x, y, and z, and calculate the overall constant deviation.

[0083] If the newly calibrated intrinsic parameters are consistently less than the set threshold, and the highest quantile of the positioning error for the entire flight path is less than the preset threshold, it indicates that the new intrinsic parameter calibration is correct and should be approved. The calibration data of the corresponding lens for that UAV will then be marked as accurate calibration.

[0084] If the newly calibrated intrinsic parameters are consistently greater than or equal to the set threshold, it indicates that the newly calibrated intrinsic parameters are also inaccurate, and the drone can be recalled for intrinsic parameter calibration.

[0085] In some embodiments, calibration results (such as algorithm calibration interpolation) can be uploaded to the Internet of Things.

[0086] In some embodiments, the full-process algorithm provided in this application can be run on a cloud server to perform data feedback and intrinsic parameter calibration for all currently operating drones. For a drone with degraded intrinsic parameters, the cloud server can integrate the corresponding drone number, the original intrinsic parameters, and the calibrated intrinsic parameters into a JSON format, and upload it to an IoT server that can communicate with the drone via an HTTP interface.

[0087] In some embodiments, the IoT server receives data sent by the cloud algorithm deployment server and distributes it to the corresponding drone according to the corresponding drone ID.

[0088] In some embodiments, the drone pulls relevant data from the Internet of Things when it is powered on daily, periodically, or when a preset event occurs. After receiving the data, the drone first verifies the drone ID in the received data packet to confirm whether the parameters sent to the drone are correct. At the same time, it verifies the original calibration parameters to determine whether they are existing parameters on the drone. It then confirms the accuracy of the current data packet and finally updates the newly calibrated parameters to the local calibration file to update and repair the degraded parameters.

[0089] The aforementioned technical solution achieves intrinsic parameter self-calibration without manual intervention or dedicated calibration scenarios by simultaneously acquiring flight images and high-precision measured poses using RTK / IMU during routine drone operations, and then integrating these two data for joint optimization of motion recovery structures. Furthermore, by comparing the systematic deviations between the theoretical pose calculated based on the original intrinsic parameters and the measured pose, intrinsic parameter degradation can be automatically and reliably detected. Finally, relying on an end-to-end cloud communication link, the verified new calibration intrinsic parameters can be remotely and securely distributed to the drone for online repair. This process automates the entire chain from data acquisition and analysis to parameter updates, enabling drones to perform autonomous "check-ups" and "calibrations" while continuously executing tasks, significantly improving the operational reliability and maintenance efficiency of large-scale fleets.

[0090] Figure 3 This is a flowchart illustrating a method for repairing the intrinsic parameters of a drone camera according to an exemplary embodiment.

[0091] refer to Figure 3 The above-mentioned method for repairing the intrinsic parameters of a drone camera may include the following steps.

[0092] Step S302: Acquire flight images captured by the onboard camera and measured poses simultaneously acquired by the pose acquisition device when the UAV is performing a flight mission.

[0093] Step S304: Based on the flight image and measured pose, the intrinsic parameters of the UAV are recalibrated to obtain the algorithm calibration intrinsic parameters.

[0094] Step S306: Based on the original intrinsic parameters and flight images currently used by the airborne camera, calculate the pose of the UAV when acquiring flight images, and use it as the theoretical pose.

[0095] Step S308: Compare the theoretical pose with the measured pose and calculate the pose deviation.

[0096] Step S310: Compare the original intrinsic parameters with the algorithm calibration intrinsic parameters to determine the intrinsic parameter deviation.

[0097] In some embodiments, a threshold for the percentage change of camera intrinsic parameters can be preset. For the intrinsic parameters calibrated by the aforementioned algorithm, they can be compared with the original intrinsic parameters to determine whether the percentage difference between the focal length f and the principal point c of the original intrinsic parameters and the newly calibrated intrinsic parameters exceeds the threshold. .in This represents the recalibrated focal length. This represents the focal length before recalibration. This represents the recalibrated principal point. This represents the principal point before recalibration.

[0098] In some embodiments, lenses that exceed a threshold can be marked as degraded lenses, meaning that the lens intrinsic parameters may have significant changes.

[0099] Step S312: When the intrinsic parameter deviation exceeds the second preset threshold and the pose deviation exceeds the first preset threshold, it is determined that the intrinsic parameters of the airborne camera have degraded.

[0100] Step S314: Based on the algorithm calibration intrinsic parameters, update and repair the intrinsic parameters of the airborne camera.

[0101] This solution significantly improves the accuracy and reliability of intrinsic parameter degradation detection through an innovative dual-threshold joint judgment mechanism: After completing intrinsic parameter self-calibration using measured pose and images, the system not only compares the numerical differences between the original intrinsic parameters and the newly calibrated intrinsic parameters (intrinsic parameter deviation) to directly perceive parameter changes, but also simultaneously checks the consistency between the pose calculated based on the original intrinsic parameters and the measured pose (pose deviation) to verify positioning failure caused by parameter degradation; only when both deviations exceed a preset threshold is the system ultimately judged as having intrinsic parameter degradation and triggers an update. This design effectively avoids misjudgments caused by single calibration errors or data noise, ensuring the rigor of repair decisions, thus balancing sensitivity and robustness in the automated process, and achieving accurate and safe monitoring and maintenance of the intrinsic parameter status of large-scale operational UAVs.

[0102] Figure 4 This is a flowchart illustrating a theoretical pose determination method according to an exemplary embodiment.

[0103] In some embodiments, the flight image may include a first image.

[0104] Below, this application will use the first image as an example to explain how to calculate the theoretical pose of the UAV when acquiring flight images.

[0105] refer to Figure 4 The above-mentioned method for determining theoretical pose may include the following steps.

[0106] Step S402: Obtain the positioning map corresponding to the flight mission. The positioning map includes the three-dimensional coordinates of multiple physical objects in the environment and their corresponding image feature descriptors; the first image includes the first physical object belonging to the multiple physical objects.

[0107] A positioning map can be a high-precision digital reference map pre-created specifically for UAV visual positioning. It's not an ordinary image map, but rather a record of the precise three-dimensional spatial coordinates of a series of key environmental feature points (such as building corners, unique textures, etc.) along a specific flight path, along with two-dimensional image feature descriptors (such as SIFT, ORB, etc.) extracted from historical images that strictly correspond to each three-dimensional point. In short, it's a database that precisely correlates three-dimensional world coordinates (where it is) with two-dimensional visual features (what it looks like). When a UAV takes images during a new flight, by matching real-time image features with features in this map, it can quickly and accurately calculate its position and attitude within the known map.

[0108] In some embodiments, the operating route corresponding to the drone index can be retrieved from pre-generated image feature points. With 3D points The resulting location map: .

[0109] Physical objects can refer to static scene elements in the real world that can be captured by a camera and stably identified as visual feature points or feature regions, and that have obvious textures or geometric structures.

[0110] Step S404: Perform feature matching processing on the first image based on the image feature symbols in the positioning map to determine the three-dimensional coordinates of the first physical object and the two-dimensional coordinates of the first physical object in the first image.

[0111] In some embodiments, feature points can be extracted from the first image and their image feature descriptors (such as SIFT or ORB descriptors) can be calculated. Then, these descriptors are compared with image feature descriptors pre-stored in the positioning map and associated with three-dimensional coordinates (e.g., using nearest neighbor search) to find the map feature entry that best matches the feature point in the first image. Upon successful matching, the three-dimensional coordinates corresponding to the first physical object can be directly read from the positioning map. Simultaneously, the two-dimensional coordinates of the first physical object in the first image represent the detection position of the feature point in the image pixel coordinate system. Through this process, the system establishes a mapping from two-dimensional pixels of the image to the location of the first physical object. Figure 3 The accurate correspondence between points in 3D space.

[0112] Step S406: Based on the original intrinsic parameters, the three-dimensional coordinates of the first physical object, and the two-dimensional coordinates of the first physical object in the first image, determine the theoretical pose of the UAV when acquiring the first image.

[0113] In some embodiments, the theoretical pose can be solved using the PnP (Perspective-n-Point) algorithm: for example, the three-dimensional coordinates of the first physical object can be used as a spatial point set, and its corresponding two-dimensional pixel coordinates in the first image can be used as an observation point set. The original intrinsic parameters currently used by the camera are input to construct a perspective projection equation. The PnP algorithm solves the equation through iterative optimization (such as using EPnP, iterative methods, etc.) to find an optimal camera rotation matrix R and translation vector t, such that when the three-dimensional point is projected onto the image plane according to the pose and intrinsic parameters, the reprojection error between it and the observed two-dimensional point is minimized. The final solution R and t are the theoretical pose of the UAV when acquiring the first image.

[0114] This method achieves accurate and robust pose calculation in known environments by rapidly matching pre-constructed high-precision visual positioning maps with real-time images. First, the system efficiently matches the "3D coordinate-feature descriptor" pairs stored in the positioning map with the current image features, quickly establishing multiple reliable 2D-3D point correspondences. Then, combining the original intrinsic parameters of the camera currently used by the UAV, the precise position and orientation (theoretical pose) of the UAV in space when the image was captured is directly calculated using the PnP optimization algorithm. The entire process does not rely on external signals such as GPS and can provide stable, low-latency positioning output in structured or semi-structured environments, providing a reliable pose ground truth comparison benchmark for subsequent intrinsic parameter degradation determination.

[0115] Figure 5 This is a flowchart illustrating an algorithm calibration intrinsic parameter determination method according to an exemplary embodiment.

[0116] refer to Figure 5 The above algorithm calibration intrinsic parameter determination method may include the following steps.

[0117] In some embodiments, some or all of the flight images contain an identifiable second physical object.

[0118] Below, this application will use the second physical object as an example to explain how to recalibrate the intrinsic parameters of the UAV based on flight images and measured poses to obtain the algorithm calibration intrinsic parameters.

[0119] Step S502: Obtain the three-dimensional coordinate values ​​of the second physical object.

[0120] In some embodiments, the three-dimensional coordinates of the second physical object can be obtained directly from the positioning map, and multi-view mapping can also be used.Figure 3 The above three-dimensional coordinate values ​​are obtained by angularization, and this application does not impose any restrictions on this.

[0121] In some embodiments, feature points can be triangulated based on high-precision prior pose and feature point tracking chains to obtain 2D image matching feature points. Corresponding 3D points .

[0122] Step S504: Using the original intrinsic parameters of the UAV as initial intrinsic parameters, and combining them with the three-dimensional coordinates of the second physical object and the measured pose of the UAV when acquiring at least one flight image, determine the reprojection pixels corresponding to the second physical object in at least one flight image.

[0123] In some embodiments, the known three-dimensional coordinates of the second physical object can be used, along with the original intrinsic parameters of the UAV and the measured pose (from RTK / IMU) during image acquisition, to calculate the theoretically expected pixel position of the object on the image plane using a camera projection model. Then, based on this initial projection position, feature matching or alignment optimization is performed in a local area of ​​the image, fine-tuning the projection points to more accurately correspond to the actually observed feature positions in the image. Finally, the precise reprojected pixel coordinates of the second physical object in the flight image are obtained. This process provides crucial two-dimensional observation data for subsequent calculation of reprojection errors and evaluation of intrinsic parameter accuracy.

[0124] Specifically, it can read the global position and attitude information of the camera provided by the high-precision RTK-GNSS system carried by the drone. , which serves as the initial pose of the image in the sfM algorithm.

[0125] In some embodiments, the camera intrinsics can be parameterized first, whereby the camera intrinsics include optimization parameters: .in, The radial distortion coefficient is... denoted as the tangential distortion coefficient.

[0126] In some embodiments, a projection function can be used. Determine the reprojected pixels of the second physical object in the flight image: .

[0127] Step S506: Obtain the observation pixels of the second physical object from at least one flight image.

[0128] In some embodiments, feature extraction and matching algorithms can be used to automatically identify and locate the specific pixel position of the second physical object from one or more flight images.

[0129] Step S508: Determine the sum of the errors between each reprojected pixel and its corresponding observed pixel as the reprojection error.

[0130] In some embodiments, the predicted pixel position (reprojected pixel) of each second physical object is compared one by one with the actual pixel position detected in the image (observed pixel), and the sum of the deviations of all point pairs is calculated as the overall indicator for measuring the accuracy of intrinsic parameters and pose—reprojection error.

[0131] Step S510: Minimize the reprojection error using the intrinsic parameters as variables to obtain the algorithm calibration intrinsic parameters.

[0132] In some embodiments, the reprojection error can be defined as the difference between the observed pixel coordinates and the theoretical projected coordinates: .in, Let `projection` be the projection function, representing the projection of 3D points onto 2D points in an image. This is achieved by combining a kernel function... (e.g., L2 norm, Huber kernel, Cauchy kernel, etc.) Nonlinear optimization of camera pose and intrinsic parameters and 3D point coordinates Minimize reprojection error: After the convergence criterion is met, the intrinsic parameter optimization result is used as the algorithm calibration to achieve high-precision intrinsic parameter calibration.

[0133] This scheme introduces high-precision measured pose as a strong constraint, transforming the intrinsic parameter calibration problem into an optimization problem subject to strict geometric conditions, effectively improving the robustness and accuracy of the calibration. Specifically, the system uses the known 3D physical point coordinates, measured pose, and initial intrinsic parameters to forward calculate the theoretical projection position (reprojected pixel) of the point in the image. This is then compared with the detected feature positions (observed pixels) in the actual image, with the sum of reprojection errors for all matching point pairs serving as the optimization objective. A nonlinear optimization algorithm continuously adjusts the camera intrinsic parameters until the error is minimized, thereby obtaining accurate algorithm calibration intrinsic parameters. This method avoids the local optimum problem caused by pose uncertainty in traditional pure vision SfM calibration, achieving stable, reliable, and high-precision online self-calibration in real-world operational scenarios.

[0134] Figure 6 This is a flowchart illustrating a method for determining three-dimensional coordinate values ​​according to an exemplary embodiment.

[0135] refer to Figure 6 The above-mentioned method for determining three-dimensional coordinate values ​​may include the following steps.

[0136] Step S602: Perform feature matching and image tracking on multiple flight images collected by the UAV during flight to determine the feature point tracking chain of the second physical object.

[0137] In some embodiments, feature points can be extracted from the aforementioned calibrated UAV-captured images using Scale Invariant Feature Transform (SIFT). Let the first... Zhang Image The extracted feature points are: .in, For the first The first image The pixel coordinates of each feature point are obtained, and then the descriptors of the corresponding feature points are extracted for feature point matching.

[0138] In some embodiments, feature correspondences can be established between adjacent images by comparing feature descriptors, and feature chains of the same physical point can be maintained in multiple image sequences through a tracking mechanism. For the first... Zhang Hedi Feature points in an image are analyzed, and corresponding matching point pairs are determined based on the distance between feature descriptors. Matching point pairs that satisfy this constraint are retained as valid matches. Feature chains of the same physical point are traced in the image sequence. Let the physical point... In the The projections in the images are respectively Build a tracking chain: The subscript indicates the feature's number in the corresponding image.

[0139] Step S604: The measured pose of the UAV when it acquires at least one flight image is taken as the prior pose.

[0140] Step S606: Based on the prior pose and feature point tracking chain, the second physical object is triangulated to determine the three-dimensional coordinate values ​​of the second physical object.

[0141] In some embodiments, a feature point tracking chain can be established based on multiple flight images containing a second physical object, ensuring that the same physical point is continuously tracked and matched from multiple viewpoints. Subsequently, using the high-precision prior poses (camera position and attitude) corresponding to these images provided by the RTK / IMU, combined with camera intrinsic parameters, the optimal 3D coordinates of the feature point in the world coordinate system are solved in reverse using triangulation methods such as least squares or singular value decomposition. This process essentially involves finding the optimal intersection point in 3D space for multiple lines of sight originating from different viewpoints and passing through the same image point, thereby directly reconstructing the 3D position of key points in the environment using aerial photography sequences without prior acquisition of map data.

[0142] This solution effectively addresses the problem of obtaining 3D coordinates of environmental points in the absence of a pre-set map: by establishing and maintaining a feature point tracking chain for a second physical object across multiple flight images, it ensures stable correspondence of the same scene point under different viewpoints; furthermore, combined with the high-precision prior pose provided by RTK / IMU, the system utilizes multi-view... Figure 3 The principle of angular convergence allows multiple lines of sight from different shooting positions to converge in three-dimensional space, thereby reliably calculating the precise three-dimensional coordinates of the physical object in the world coordinate system. This process not only enables online reconstruction from two-dimensional image sequences to three-dimensional structures, but also provides crucial spatial geometric constraints for subsequent steps such as camera intrinsic parameter calibration and reprojection error calculation, enhancing the adaptability and reliability of the entire calibration system in unknown or dynamic environments.

[0143] Figure 7 This is a flowchart illustrating an image preprocessing method according to an exemplary embodiment.

[0144] In some embodiments, the flight image may include a second image, which includes multiple rows of pixels.

[0145] In some embodiments, reference may be made to Figure 7 The method preprocesses the second image: Step S702: Obtain the angular velocity of the UAV during flight through the inertial measurement unit.

[0146] In some embodiments, the angular velocity data described above can be filtered by applying moving average filtering and Butterworth low-pass filtering to the raw gyroscope data of the inertial measurement unit (IMU) to smooth noise. Subsequently, the IMU data is sampled for each row of the image. For the pixel in the i-th row, the exposure time is: .in, The start time of image exposure. For single-line readout time, , This represents the number of rows in the image.

[0147] In some embodiments, a time-sampling grid can be constructed for each row: .

[0148] In some embodiments, binary search and linear interpolation can be used to sample the angular velocity corresponding to each row of the image: .in Let be the angular velocity corresponding to the i-th row in the k-th image, thus enabling fine-grained sampling of the angular velocity for each row of data in each frame. Both k and i are integers not greater than 0.

[0149] Step S704: Determine the rotation matrix corresponding to each row of pixels in the second image by the acquisition drone based on the angular velocity.

[0150] In some embodiments, the cumulative angular displacement corresponding to each row of the second image can be determined based on the angular velocity; then, the rotational angular velocity corresponding to each row of the second image can be determined based on the cumulative angular displacement corresponding to each row.

[0151] In some embodiments, the cumulative angular displacement of a row (e.g., row i) can be obtained based on the row index: And the rotation matrix is ​​calculated using the Rodriguez formula. : .in, For antisymmetric matrices: .

[0152] Step S706: Rotate each row of pixels in the second image based on the rotation matrix to compensate for each row of pixels in the second image.

[0153] Step S708: Preprocess the second image based on the rotated pixels.

[0154] This image preprocessing scheme targets the rolling shutter cameras commonly used in UAVs, achieving precise online correction of image distortion caused by flight shake. Using high-frequency sampled IMU gyroscope data, the system accurately reconstructs the camera angular velocity at the moment of exposure for each row of pixels and cumulatively calculates the rotational displacement of that row relative to the frame's start time. Utilizing the Rodriguez rotation formula, each pixel is rotated inversely from its original shake-affected direction to its theoretical global shutter imaging position, thereby eliminating rolling shutter distortions such as the "jelly effect" caused by the UAV's rapid movement or vibration line by line. This processing significantly improves the accuracy of subsequent feature extraction and matching, laying a high-quality image data foundation for highly reliable visual intrinsic parameter calibration and positioning in dynamic flight environments.

[0155] Figure 8 This is a flowchart illustrating an image preprocessing method according to an exemplary embodiment.

[0156] refer to Figure 8 Preprocessing the second image based on the rotated pixels may include the following steps.

[0157] Step S802: Based on the rotated pixels, determine the optical flow vector corresponding to each pixel in the second image; wherein the optical flow vector is used to describe the displacement of the rotated pixel relative to the pixel before rotation.

[0158] In some embodiments, an optical flow vector can be constructed: This represents the distance each pixel needs to move to eliminate rolling shutter distortion.

[0159] Step S804: Back-mapping is performed on the pixels in the second image based on the optical flow vector.

[0160] In some embodiments, reverse mapping calculates the position of the corresponding rolling shutter pixel to the global shutter pixel based on optical flow and maps it to the corresponding position. Multi-source pixel accumulation addresses the situation where multiple rolling shutter pixels are mapped to the same global shutter pixel.

[0161] In some embodiments, optical flow can be applied using backward mapping: for pixels in the original rolling shutter image Calculate the source location from which it should originate: .

[0162] Then, after rounding, boundary constraints are applied. Using the cumulative method to handle cases where multiple source pixels are mapped to the same target pixel: .

[0163] in, This is a global shutter (after rolling shutter correction) image. This is the original rolling shutter image.

[0164] In some embodiments, for holes in passive pixels, median filtering is used for interpolation: .

[0165] The image is now complete, but the rolling shutter effect caused by the drone's flight vibrations has affected the image.

[0166] Step S806: Obtain the preprocessed second image based on the reverse mapping result.

[0167] This preprocessing workflow achieves a stable conversion from a rolling shutter image to an equivalent global shutter image through optical flow-guided pixel-level reverse rearrangement and intelligent filling. First, based on the optical flow vector calculated by rotation compensation, the system "pushes back" each pixel in the distorted image to its theoretical position before distortion by reverse mapping, and uses a pixel cumulative averaging strategy to handle coordinate overlap to maintain texture smoothness. Subsequently, for sourceless pixel "holes" caused by excessive motion or occlusion, adaptive interpolation repair is performed using mean square filtering. This series of operations ultimately generates a corrected image with strong spatiotemporal consistency and effectively eliminated geometric distortion, providing reliable input for subsequent high-precision visual feature extraction and matching, and significantly improving the robustness and accuracy of intrinsic parameter calibration and visual positioning under severe shaking flight conditions.

[0168] Figure 9 This is a flowchart illustrating a pixel processing method according to an exemplary embodiment.

[0169] In some embodiments, the multiple rows of pixels in the second image may include the first row of pixels.

[0170] refer to Figure 9 The process of rotating each row of pixels in the second image based on a rotation matrix to compensate for the pixel differences in the second image may include the following steps.

[0171] Step S902: Convert the pixel coordinates of the first row of pixels into direction vectors in the camera coordinate system.

[0172] Step S904: Rotate the direction vector corresponding to the first row of pixels using the rotation matrix corresponding to the first row of pixels to obtain the rotated direction vector.

[0173] Step S906: The rotated direction vector is back-projected back to the pixel coordinate system to obtain the compensated pixels corresponding to the first row, so as to preprocess the second image based on the compensated pixels.

[0174] In some embodiments, an optical flow image can be used to represent the optical flow for each pixel. The displacement is caused by the drone's shaking. Calculating the optical flow image is to determine the offset of each pixel relative to the global shutter image due to the rolling shutter effect, and then calibrating the rolling shutter image based on this offset.

[0175] First, convert the pixel coordinates to unit direction vectors in the camera coordinate system: .in, As the initial principal point, This is the initial focal length.

[0176] In some embodiments, the cumulative angular displacement of a row can be obtained based on the row index, and the rotation matrix can be calculated using the Rodriguez formula. And calculate the coordinates of the unit direction vector after rotation based on the rotation matrix: And project it back to the pixel coordinate system: To obtain the compensated pixels corresponding to the first row, so as to preprocess the second image based on the compensated pixels.

[0177] The pixel-level processing method described above achieves precise geometric correction of the rolling shutter effect line by line: by rotating and compensating the coordinates of each row of pixels in the image in reverse from the original observation direction affected by jitter to its theoretical static imaging direction based on the accumulated rotational displacement of the current row (calculated from IMU data), and then reprojecting it back onto the image plane, the correct position of each pixel after distortion correction is calculated. This process provides a direct and accurate geometric basis for generating optical flow maps describing pixel position changes, and is a key foundational step for subsequent overall image correction and restoration of spatiotemporal consistency.

[0178] In related technologies, autonomous flight of drones relies on robust and accurate positioning. The core of this is visual positioning, which depends on accurate camera intrinsics. However, in real-world scenarios, the camera intrinsics of drones may change due to factors such as non-ideal initial calibration, mechanical vibration, structural deformation, or lens aging, leading to degradation of the camera intrinsics and affecting the accuracy of global visual positioning. Therefore, effective mechanisms are needed to detect changes in the drone's intrinsics, and efficient, automated methods are required to calibrate the intrinsics and achieve online repair.

[0179] The existing solutions suffer from several key limitations: First, they rely on specific calibration objects (such as checkerboard patterns, random calibration images, or airborne calibration boards), requiring dedicated environments or additional loads and preventing the UAV from operating normally during calibration, making them unsuitable for large-scale, routine commercial flight scenarios. Second, self-calibration methods based on pure vision-based SfM are susceptible to shutter distortion and feature matching errors, often getting stuck in local optima, resulting in poor calibration stability and a lack of reliable verification mechanisms. Third, while data-driven methods such as deep learning improve calibration convenience, they suffer from insufficient model generalization ability and uninterpretable results, and also lack effective accuracy verification processes. Overall, existing solutions cannot achieve robust, high-precision, and self-verifiable intrinsic parameter degradation detection and online repair while the UAV continuously performs tasks.

[0180] Below, this application is approved. Figure 10 A technical solution is proposed to address the degradation of UAV camera intrinsic parameters caused by factors such as mechanical vibration, structural deformation, or lens aging during routine operation. This solution utilizes SfM self-calibration and validity verification based on multi-source data fusion to achieve the effect of detecting and repairing UAV camera intrinsic parameter degradation based on operational data without manual intervention or the need to photograph a custom calibration board.

[0181] Figure 10 This is a schematic diagram of a method for repairing the intrinsic parameters of a drone camera, according to an exemplary embodiment.

[0182] This solution provides a complete process for automatic detection and updating of camera intrinsic parameter degradation for drones carrying RTK and IMU for batch patrol operations in commercial scenarios such as low-altitude delivery. Figure 10 As shown, the framework diagram corresponding to the complete process of automatic detection and updating of camera intrinsic parameter degradation can include four modules: image preprocessing, intrinsic parameter self-calibration, intrinsic parameter degradation detection and verification, and intrinsic parameter distribution.

[0183] The image preprocessing module uses IMU data to perform motion calibration on images captured by the rolling shutter camera mounted on the UAV, compensating for the rolling shutter effect caused by UAV shaking, and laying an accurate foundation for subsequent calibration.

[0184] The intrinsic parameter self-calibration module utilizes images collected during a single flight operation and high-precision pose information provided by RTK, and employs the SfM algorithm to jointly optimize intrinsic parameters such as camera focal length, principal point, and distortion.

[0185] The intrinsic parameter degradation detection and verification module detects the degradation of the UAV camera intrinsic parameters based on preset intrinsic parameter variation thresholds and visual positioning error thresholds, and verifies the effectiveness of the intrinsic parameters generated by the self-calibration module.

[0186] The internal parameter distribution module distributes verified calibration internal parameters to drones with degraded internal parameters through the communication link between the cloud server, the IoT server, and the drone, thereby achieving internal parameter repair.

[0187] Below, this application will provide a detailed explanation of each module.

[0188] 1. Image preprocessing module.

[0189] This module aims to compensate for rolling shutter images affected by severe shaking by combining IMU data, recover the global shutter image, improve the spatiotemporal consistency of the image, and provide a more accurate image for subsequent calibration algorithms.

[0190] (1) IMU data preprocessing.

[0191] First, the IMU data is filtered. Moving average filtering and Butterworth low-pass filtering are applied to the raw gyroscope data to smooth out noise. Then, the IMU data is sampled for each row of the image. For the pixel in the i-th row, the exposure time is: .

[0192] in, The start time of image exposure. For single-line readout time, , This represents the number of rows in the image.

[0193] Construct a time sampling grid for each row: .

[0194] Binary search and linear interpolation are used to sample the angular velocity corresponding to each row of the image: .

[0195] in This allows for fine-grained sampling of the angular velocity of each row of data in each frame of the image, corresponding to the angular velocity of each row.

[0196] (2) Generation of optical flow mapping map.

[0197] Optical flow image representation for each pixel The displacement is caused by the drone's shaking. Calculating the optical flow image is to determine the offset of each pixel relative to the global shutter image due to the rolling shutter effect, and then calibrating the rolling shutter image based on this offset.

[0198] First, convert the pixel coordinates to unit direction vectors in the camera coordinate system: .

[0199] in, As the initial principal point, This is the initial focal length. The cumulative angular displacement of the row is obtained based on the row index: .

[0200] Calculate the rotation matrix using the Rodriguez formula : .in, For antisymmetric matrices: .

[0201] And calculate the coordinates of the unit direction vector after rotation based on the rotation matrix: .

[0202] And project it back into the pixel coordinate system: .

[0203] Therefore, the optical flow vector is constructed: .

[0204] This represents the distance each pixel needs to move to eliminate rolling shutter distortion.

[0205] (3) Reverse mapping and multi-source pixel accumulation.

[0206] Reverse mapping calculates the position of the corresponding rolling shutter pixel to the global shutter pixel based on optical flow and maps it to the corresponding position. Multi-source pixel accumulation is used to handle the situation where multiple rolling shutter pixels are mapped to the same global shutter pixel.

[0207] Applying optical flow using backward mapping: For pixels in the original rolling shutter image Calculate the source location from which it should originate: .

[0208] Then, after rounding, boundary constraints are applied. .

[0209] Using the cumulative method to handle cases where multiple source pixels are mapped to the same target pixel: .

[0210] in, This is a global shutter (after rolling shutter correction) image. This is the original rolling shutter image.

[0211] (4) Filling holes.

[0212] For holes in passive pixels, median filtering is used for interpolation: .

[0213] The image is now complete, but the rolling shutter effect caused by the drone's flight vibrations has affected the image.

[0214] 2. Intrinsic parameter self-calibration module: This module aims to achieve camera intrinsic parameter self-calibration based on multi-view geometry and feature matching, combined with high-precision prior pose, and through the sfM algorithm, providing a foundation for subsequent intrinsic parameter degradation detection.

[0215] (1) Feature extraction and description.

[0216] For the images captured by the aforementioned calibrated UAV, feature points are extracted using Scale Invariant Feature Transform (SIFT). Let the... Zhang Image The extracted feature points are: .

[0217] in, For the first The first image The pixel coordinates of each feature point are obtained, and then the descriptors of the corresponding feature points are extracted for feature point matching.

[0218] (2) Feature matching and multi-image tracking.

[0219] By comparing feature descriptors, feature correspondences are established between adjacent images, and a tracking mechanism is used to maintain the feature chain of the same physical point across multiple image sequences. For the first... Zhang Hedi Feature points in an image are analyzed, and corresponding matching point pairs are determined based on the distance between feature descriptors. The RANSAC algorithm is then used to verify the geometric consistency of the matches, generating the final in-place matching points. Matching point pairs that satisfy this constraint are retained as valid matches. Feature chains of the same physical point are traced in the image sequence. Let the physical point... In the The projections in the images are respectively Build a tracking chain: .

[0220] The subscript indicates the number of the feature in the corresponding image.

[0221] (3) High-precision prior pose initialization.

[0222] Read the global position and attitude information of the camera provided by the high-precision RTK-GNSS system on the UAV. , which serves as the initial pose of the image in the sfM algorithm.

[0223] (4) Multiple views Figure 3 Keratinization is used to obtain the initial 3D points.

[0224] Triangulation of feature points is performed based on high-precision prior pose and feature point tracking chain to obtain matching feature points in 2D images. Corresponding 3D points .

[0225] (5) Nonlinear optimization intrinsic parameter calibration.

[0226] First, the camera intrinsics are parameterized, including optimization parameters: .

[0227] in, The radial distortion coefficient is... denoted as the tangential distortion coefficient.

[0228] The reprojection error is defined as the difference between the observed pixel coordinates and the theoretical projected coordinates: .

[0229] in, Let `projection` be the projection function, representing the projection of 3D points onto 2D points in an image. This is achieved by combining a kernel function... (e.g., L2 norm, Huber kernel, Cauchy kernel, etc.) Nonlinear optimization of camera pose and intrinsic parameters and 3D point coordinates Minimize reprojection error: .

[0230] After the convergence criterion is met, the result of the intrinsic parameter optimization is used as the algorithm calibration to achieve high-precision intrinsic parameter calibration.

[0231] 3. Intrinsic parameter degradation detection and verification module: This module aims to detect whether camera intrinsic parameters have degraded and to verify the accuracy of the calibration results through the positioning map.

[0232] (1) Internal reference degradation detection.

[0233] A pre-set threshold for the percentage change in camera intrinsic parameters is used. The intrinsic parameters calculated by the aforementioned intrinsic parameter self-calibration module are compared with the intrinsic parameters in the original calibration file on the UAV to determine whether the percentage difference between the focal length and principal point between the original intrinsic parameters and the newly calibrated intrinsic parameters exceeds the threshold. .

[0234] Lenses exceeding the threshold are marked as degraded lenses, meaning that the lens intrinsic parameters may have significant changes.

[0235] (2) Location map construction.

[0236] For shots where the internal parameters might be degraded as determined in the previous step, the corresponding operational flight path is used based on the corresponding drone index, and a pre-generated positioning map composed of image feature points and 3D points for that flight path is retrieved: .

[0237] (3) Obtain new and old PNP location data based on the location map.

[0238] With location maps constructed from different flights on the same route, features can be extracted from the image of the currently calibrated flight data and matched with map feature points to construct a location map. Figure 3 The correspondence between point D and 2D points in the image: .

[0239] Given the correspondence between 3D points and 2D points in the image, as well as the camera intrinsics, the pose of the current image can be solved using the PnP algorithm. .

[0240] Based on the above principles, the pose of each image is solved using the PnP algorithm, applying both the old and newly calibrated UAV intrinsic parameters. The pose solved using the PnP algorithm is then compared with the high-precision RTK pose derived from the UAV, and the corresponding positioning error is calculated to evaluate the accuracy of the pose solution.

[0241] (4) Judgment of the accuracy of calibration results.

[0242] For the same map and the same image, the difference in pose obtained by PnP arises solely from the accuracy of the intrinsic parameter calibration. First, Gaussian fitting is performed on the positioning error of PnP using the old intrinsic parameters, and the mean values ​​in the x, y, and z directions are determined: .

[0243] At this point, the mean positioning error in each of the three directions indicates whether the PnP positioning suffers from constant bias in the three directions due to inaccurate intrinsic parameters. Simultaneously, the mean values ​​in the three directions... Find the 2-norm, which characterizes the global constant bias: .

[0244] Determine if the overall constant bias exceeds a preset threshold. If there is a significant difference between the algorithm calibration intrinsic and the current original intrinsic, and the positioning constant bias exceeds the threshold, it indicates that the UAV lens has degraded, and the original intrinsics are no longer applicable. Simultaneously, use the new intrinsics to solve for PnP pose, applying the same method to statistically analyze the constant biases along the x, y, and z axes, and calculate the overall constant bias. If the constant bias of the newly calibrated intrinsics is less than the set threshold, and the high quantile of the positioning error for the entire flight path is less than the preset threshold, it indicates that the new intrinsic calibration is correct and is recognized. The calibration data for the corresponding lens of that UAV is then marked as accurately calibrated.

[0245] 4. The internal parameter distribution module has the following functions and effects.

[0246] (1) Upload the calibration results to the Internet of Things.

[0247] The entire algorithm runs on a cloud server, performing data feedback and intrinsic parameter calibration for all currently operational drones. For a drone with degraded intrinsic parameters, the cloud server integrates the corresponding drone number, the original intrinsic parameters, and the calibrated intrinsic parameters into a JSON format, and uploads it via an HTTP interface to an IoT server that can communicate with the drone.

[0248] (2) The Internet of Things issues calibration parameters.

[0249] The IoT server receives data sent by the cloud algorithm deployment server and distributes it to the corresponding drone according to the drone ID.

[0250] (3) Verification and updating of UAV side parameters.

[0251] When the drone is powered on each day, it pulls relevant data from the Internet of Things. After receiving the data, the drone first verifies the drone ID in the received data packet to confirm whether the parameters sent to the drone are correct. At the same time, it verifies the original calibration parameters to determine whether they are the parameters that already exist on the drone. It then confirms again whether the current data packet is accurate. Finally, it updates the newly calibrated parameters to the local calibration file to update and repair the degraded parameters.

[0252] It should be particularly noted that the steps in each embodiment of the above-described UAV camera intrinsic parameter repair method can be interchanged, substituted, added, or deleted. Therefore, these reasonable permutations and combinations of the UAV camera intrinsic parameter repair method should also fall within the protection scope of this disclosure, and the protection scope of this disclosure should not be limited to the embodiments.

[0253] It should be noted that the scope of protection of this application should include, but is not limited to, the specific implementation methods described in the embodiments. Any alternative solution that uses a different name but substantially performs the same function and achieves the same technical effect falls within the scope of protection defined by the claims of this application.

[0254] Based on the same inventive concept, this disclosure also provides a drone camera intrinsic parameter repair device, as shown in the following embodiment. Since the principle by which this device solves the problem is similar to that of the method embodiment described above, the implementation of this device embodiment can refer to the implementation of the method embodiment described above, and repeated details will not be elaborated further.

[0255] Figure 11 This is a block diagram illustrating a drone camera intrinsic parameter repair device according to an exemplary embodiment. (Refer to...) Figure 11 The UAV camera intrinsic parameter repair device 1100 provided in this embodiment may include: a flight image acquisition module 1101, an algorithm calibration intrinsic parameter determination module 1102, a theoretical pose determination module 1103, a pose deviation calculation module 1104, an intrinsic parameter degradation judgment module 1105, and an intrinsic parameter repair module 1106.

[0256] The flight image acquisition module 1101 can be used to acquire flight images captured by the airborne camera and measured poses synchronously acquired by the pose acquisition device when the UAV is performing a flight mission; the algorithm calibration intrinsic parameter determination module 1102 can be used to recalibrate the intrinsic parameters of the UAV based on the flight images and the measured poses to obtain algorithm calibration intrinsic parameters; the theoretical pose determination module 1103 can be used to calculate the pose of the UAV when acquiring the flight images based on the original intrinsic parameters currently used by the airborne camera and the flight images, as the theoretical pose; the pose deviation calculation module 1104 can be used to compare the theoretical pose with the measured pose to calculate the pose deviation; the intrinsic parameter degradation judgment module 1105 can be used to determine that the intrinsic parameters of the airborne camera have degraded if the pose deviation exceeds a first preset threshold; and the intrinsic parameter repair module 1106 can be used to update and repair the intrinsic parameters of the airborne camera based on the algorithm calibration intrinsic parameters.

[0257] It should be noted that the aforementioned flight image acquisition module 1101, algorithm calibration intrinsic parameter determination module 1102, theoretical pose determination module 1103, pose deviation calculation module 1104, intrinsic parameter degradation judgment module 1105, and intrinsic parameter repair module 1106 correspond to S202 to S212 in the method embodiment. The examples and application scenarios implemented by these modules and their corresponding steps are the same, but they are not limited to the content disclosed in the above method embodiment. It should be noted that these modules, as part of a device, can be executed in a computer system such as a set of computer-executable instructions.

[0258] In some embodiments, the UAV camera intrinsic parameter repair device 1100 may include an intrinsic parameter deviation determination module and a degradation judgment module.

[0259] The intrinsic parameter deviation determination module can be used to compare the original intrinsic parameters with the algorithm calibration intrinsic parameters to determine the intrinsic parameter deviation; the degradation judgment module can be used to determine that the intrinsic parameters of the airborne camera have degraded when the intrinsic parameter deviation exceeds a second preset threshold and the pose deviation exceeds a first preset threshold.

[0260] In some embodiments, 3. The method according to claim 1, wherein the flight image includes a first image; wherein, based on the original intrinsic parameters currently used by the airborne camera and the flight image, calculating the pose of the UAV when acquiring the flight image as a theoretical pose includes: acquiring a positioning map corresponding to the flight mission, the positioning map including the three-dimensional coordinates of multiple physical objects in the environment and their corresponding image feature descriptors; the first image including a first physical object belonging to the multiple physical objects; performing feature matching processing on the first image based on the image feature descriptors in the positioning map to determine the three-dimensional coordinate values ​​of the first physical object and the two-dimensional coordinate values ​​of the first physical object in the first image; and determining the theoretical pose of the UAV when acquiring the first image based on the original intrinsic parameters, the three-dimensional coordinate values ​​of the first physical object, and the two-dimensional coordinate values ​​of the first physical object in the first image.

[0261] In some embodiments, at least one flight image contains a second physical object; wherein, based on the flight image and the measured pose, the intrinsic parameters of the UAV are recalibrated to obtain algorithm calibration intrinsic parameters, including: acquiring the three-dimensional coordinate values ​​of the second physical object; using the original intrinsic parameters of the UAV as initial intrinsic parameters, combining the three-dimensional coordinate values ​​of the second physical object and the measured pose of the UAV when acquiring the at least one flight image, determining the reprojection pixels corresponding to the second physical object in the at least one flight image; acquiring the observation pixels of the second physical object from the at least one flight image; determining the sum of errors between each reprojection pixel and the corresponding observation pixel as the reprojection error; minimizing the reprojection error with the intrinsic parameters as variables to obtain the algorithm calibration intrinsic parameters.

[0262] In some embodiments, the three-dimensional coordinates of the second physical object are obtained by the following method: performing feature matching and image tracking on multiple flight images acquired by the UAV during flight to determine the feature point tracking chain of the second physical object; taking the measured pose of the UAV when acquiring the at least one flight image as the prior pose; and triangulating the second physical object based on the prior pose and the feature point tracking chain to determine the three-dimensional coordinates of the second physical object.

[0263] In some embodiments, the flight image includes a second image, which includes multiple rows of pixels; wherein the second image is preprocessed by the following method: obtaining the angular velocity acquired by the UAV through an inertial measurement unit during flight; determining a rotation matrix corresponding to the acquisition of each row of pixels in the second image by the UAV based on the angular velocity; rotating each row of pixels in the second image based on the rotation matrix to compensate for each row of pixels in the second image; and preprocessing the second image based on the rotated pixels.

[0264] In some embodiments, preprocessing the second image based on the rotated pixels includes: determining the optical flow vector corresponding to each pixel in the second image based on the rotated pixels; wherein the optical flow vector is used to describe the displacement of the rotated pixel relative to the pixel before rotation; performing backward mapping on the pixels in the second image based on the optical flow vector; and obtaining the preprocessed second image based on the backward mapping result.

[0265] In some embodiments, the multiple rows of pixels in the second image include a first row of pixels; wherein, rotating each row of pixels in the second image based on the rotation matrix to compensate for each row of pixels in the second image includes: converting the pixel coordinates of the first row of pixels into a direction vector in the camera coordinate system; rotating the direction vector corresponding to the first row of pixels using the rotation matrix corresponding to the first row of pixels to obtain a rotated direction vector; and backprojecting the rotated direction vector back to the pixel coordinate system to obtain the compensated pixels corresponding to the first row, so as to preprocess the second image based on the compensated pixels.

[0266] In some embodiments, determining the rotation matrix corresponding to the UAV acquiring each row of pixels in the second image based on the angular velocity includes: determining the cumulative angular displacement corresponding to each row of the second image based on the angular velocity; and determining the rotation angular velocity corresponding to each row of the second image based on the cumulative angular displacement corresponding to each row. Since the functions of the device 1100 have been described in detail in their corresponding method embodiments, they will not be repeated here.

[0267] The modules and / or sub-modules and / or units described in the embodiments of this disclosure can be implemented in software or hardware. The described modules and / or sub-modules and / or units can also be located in a processor. The names of these modules and / or sub-modules and / or units do not, in some cases, constitute a limitation on the module and / or sub-module and / or unit itself.

[0268] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a portion of a module or program segment containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer program instructions.

[0269] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0270] Figure 12 A schematic diagram of an electronic device suitable for implementing embodiments of the present disclosure is shown. It should be noted that... Figure 12 The illustrated electronic device 1200 is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0271] like Figure 12 As shown, the electronic device 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage section 1208 into a random access memory (RAM) 1203. The RAM 1203 also stores various programs and data required for the operation of the electronic device 1200. The CPU 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0272] The following components are connected to I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to I / O interface 1205 as needed. Removable media 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1210 as needed so that computer programs read from them can be installed into storage section 1208 as needed.

[0273] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing computer program instructions for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1209, and / or installed from removable medium 1211. When the computer program is executed by central processing unit (CPU) 1201, it performs the functions defined above in the system of this disclosure.

[0274] It should be noted that the computer-readable storage medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable computer program instructions. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Computer program instructions contained on a computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0275] In another aspect, this disclosure also provides a computer-readable storage medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable storage medium carries one or more programs that, when executed by the device, enable the device to perform the following functions: acquiring flight images captured by an airborne camera and measured poses synchronously acquired by a pose acquisition device when the UAV is performing a flight mission; recalibrating the intrinsic parameters of the UAV based on the flight images and the measured poses to obtain algorithm calibration intrinsic parameters; calculating the pose of the UAV when acquiring the flight images based on the original intrinsic parameters currently used by the airborne camera and the flight images, as the theoretical pose; comparing the theoretical pose with the measured pose to calculate the pose deviation; if the pose deviation exceeds a first preset threshold, determining that the intrinsic parameters of the airborne camera have degraded; and updating and repairing the intrinsic parameters of the airborne camera based on the algorithm calibration intrinsic parameters.

[0276] According to one aspect of this disclosure, a computer program product or computer program is provided, comprising computer program instructions stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and a processor executes the computer program instructions to implement the methods provided in various optional implementations of the above embodiments.

[0277] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions of the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or portable hard drive) and includes several computer program instructions to cause an electronic device (such as a server or terminal device) to execute the method according to the embodiments of this disclosure.

[0278] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0279] It should be understood that this disclosure is not limited to the detailed structures, drawing arrangements or implementations shown herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A method for repairing intrinsic parameters of a drone camera, characterized in that, include: Acquire flight images captured by the onboard camera and measured poses simultaneously acquired by the pose acquisition device when the UAV is performing a flight mission; Based on the flight image and the measured pose, the intrinsic parameters of the UAV are recalibrated to obtain the algorithm calibration intrinsic parameters; Based on the original intrinsic parameters currently used by the airborne camera and the flight image, the pose of the UAV when acquiring the flight image is calculated as the theoretical pose. The theoretical pose is compared with the measured pose, and the pose deviation is calculated. If the pose deviation exceeds a first preset threshold, it is determined that the intrinsic parameters of the airborne camera have degraded. Based on the algorithm, the intrinsic parameters of the airborne camera are calibrated and updated and repaired.

2. The method according to claim 1, characterized in that, The method further includes: The original intrinsic parameters are compared with the algorithm calibration intrinsic parameters to determine the intrinsic parameter deviation; When the intrinsic parameter deviation exceeds a second preset threshold and the pose deviation exceeds a first preset threshold, it is determined that the intrinsic parameters of the airborne camera have degraded.

3. The method according to claim 1, characterized in that, The flight image includes a first image; wherein, based on the original intrinsic parameters currently used by the airborne camera and the flight image, the pose of the UAV when acquiring the flight image is calculated as the theoretical pose, including: Obtain a positioning map corresponding to the flight mission. The positioning map includes the three-dimensional coordinates of multiple physical objects in the environment and their corresponding image feature descriptors. The first image includes a first physical object belonging to the multiple physical objects. Based on the image feature symbols in the positioning map, feature matching processing is performed on the first image to determine the three-dimensional coordinates of the first physical object and the two-dimensional coordinates of the first physical object in the first image. Based on the original intrinsic parameters, the three-dimensional coordinates of the first physical object, and the two-dimensional coordinates of the first physical object in the first image, the theoretical pose of the UAV when acquiring the first image is determined.

4. The method according to claim 1, characterized in that, At least one flight image contains a second physical object; wherein, based on the flight image and the measured pose, the intrinsic parameters of the UAV are recalibrated to obtain algorithm calibration intrinsic parameters, including: Obtain the three-dimensional coordinates of the second physical object; Using the original intrinsic parameters of the UAV as initial intrinsic parameters, combined with the three-dimensional coordinate values ​​of the second physical object and the measured pose of the UAV when acquiring the at least one flight image, the reprojection pixels corresponding to the second physical object in the at least one flight image are determined. Obtain the observed pixels of the second physical object from the at least one flight image; The sum of the errors between each reprojected pixel and its corresponding observed pixel is determined as the reprojection error; The reprojection error is minimized using the intrinsic parameters as variables to obtain the calibration intrinsic parameters of the algorithm.

5. The method according to claim 4, characterized in that, The three-dimensional coordinates of the second physical object are obtained using the following method: Feature matching and image tracking are performed on multiple flight images collected by the UAV during flight to determine the feature point tracking chain of the second physical object; The measured pose of the UAV when it acquires the at least one flight image is taken as the prior pose; Based on the prior pose and the feature point tracking chain, the second physical object is triangulated to determine the three-dimensional coordinates of the second physical object.

6. The method according to claim 1, characterized in that, The flight image includes a second image, which comprises multiple rows of pixels; wherein the second image is preprocessed using the following method: Obtain the angular velocity of the UAV during flight via an inertial measurement unit; Based on the angular velocity, determine the rotation matrix corresponding to the acquisition of each row of pixels in the second image by the UAV; The pixels in each row of the second image are rotated based on the rotation matrix to compensate for the pixels in each row of the second image. The second image is preprocessed based on the rotated pixels.

7. The method according to claim 6, characterized in that, The second image is preprocessed based on the rotated pixels, including: Based on the rotated pixels, the optical flow vector corresponding to each pixel in the second image is determined; wherein the optical flow vector is used to describe the displacement of the rotated pixel relative to the pixel before rotation. Back-mapping is performed on the pixels in the second image based on the optical flow vector; The preprocessed second image is obtained based on the reverse mapping result.

8. The method according to claim 6, characterized in that, The second image contains multiple rows of pixels, including a first row of pixels; wherein, rotating each row of pixels in the second image based on the rotation matrix to compensate for each row of pixels in the second image includes: Convert the pixel coordinates of the first row of pixels into direction vectors in the camera coordinate system; The direction vector corresponding to the first row of pixels is rotated using the rotation matrix corresponding to the first row of pixels to obtain the rotated direction vector; The rotated direction vector is back-projected back to the pixel coordinate system to obtain the compensated pixels corresponding to the first row, so that the second image can be preprocessed based on the compensated pixels.

9. The method according to claim 6, characterized in that, Determining the rotation matrix corresponding to the UAV acquiring each row of pixels in the second image based on the angular velocity includes: The cumulative angular displacement corresponding to each row of the second image is determined based on the angular velocity; Based on the cumulative angular displacement corresponding to each row, the rotational angular velocity corresponding to each row of the second image is determined.

10. An electronic device, characterized in that, include: Memory and processor; The memory is used to store computer program instructions; the processor calls the computer program instructions stored in the memory to implement the UAV camera intrinsic parameter repair method as described in any one of claims 1-9.

Images