Unmanned aerial vehicle video sensor online calibration method and system based on three-dimensional topographic data

Abstract

The invention relates to the technical field of computer vision, in particular to an unmanned aerial vehicle video sensor online calibration method and system based on three-dimensional topographic data. Acquiring three-dimensional topographic data of a target area to be inspected, wherein the three-dimensional topographic data comprises a reference elevation model and reference point cloud data; dividing an inspection area corresponding to the three-dimensional topographic data to obtain an uncalibrated area and a to-be-calibrated area; evaluating an online calibration demand index of each to-be-calibrated area; determining whether to start online calibration according to the online calibration demand index; when the unmanned aerial vehicle starts flight inspection, real-time video data and real-time point cloud data are collected in real time through a video sensor, and online calibration and abnormal point detection are carried out on the video sensor. By evaluating the online calibration demand index of the inspection area, whether online calibration is started or not is intelligently determined, and the precision and real-time performance of the unmanned aerial vehicle during three-dimensional terrain data acquisition and processing in a dynamic environment are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an online calibration method and system for a UAV video sensor based on three-dimensional terrain data. Background Art

[0002] At present, image-based calibration methods have been widely used in sensor calibration, especially binocular vision and multi-viewing calibration. Common calibration methods include: Zhang's calibration method: Zhang's calibration method uses a checkerboard pattern image to obtain the internal and external parameters and distortion parameters of the camera. The advantages of this method are simple operation, high accuracy, and the ability to reduce errors through optimization algorithms. Epipolar geometry calibration: In a binocular vision system, epipolar geometry calibration is based on the relative position and direction relationship between the two cameras, and is calculated based on the corresponding points of the image to obtain the internal and external parameters and epipolar constraints of the camera. SIFT feature matching: In visual calibration, SIFT features (scale-invariant feature transform) are used to extract key points in the image, and the geometric relationship between cameras is established by matching feature points from multiple perspectives.

[0003] For example, the invention patent with publication number: CN110458947A discloses a three-dimensional reconstruction method and system for the sag of a transmission line, including: calibrating the parameters of the binocular camera including: using Zhang's calibration method to determine the internal parameters and distortion parameters of each camera based on a checkerboard image; wherein the internal parameters are used to characterize the internal structure of the binocular camera, and the distortion parameters are used to characterize radial distortion and tangential distortion, then, based on the image captured by each camera, extracting the corner points where the pixel values ​​on the image suddenly change, and calculating the SIFT feature description values ​​on these corner points, using the Hamming distance to match the corresponding control points in different images in pairs, and establishing a posture relationship between the two cameras; wherein the control point is a pixel point in the image with prominent attributes in some aspect, and is an isolated point or the end point of a line segment with the maximum or minimum intensity in some attributes.

[0004] For example, the invention patent with announcement number: CN115731303B discloses a method for three-dimensional reconstruction of the sag of a large-span power transmission line based on bidirectional binocular vision, including: using Zhang's calibration method to calibrate the parameters of the binocular camera and perform epipolar correction, including: step 101, calibrating the internal, external and distortion parameters of the binocular camera; step 102, performing epipolar correction with reference to the internal, external and distortion parameters; step 103, setting and determining the focal length f of the camera and the distance b between the centers of the left and right cameras; wherein the internal parameters are the internal structural parameters of the binocular camera; the external parameters are the rotation matrix and translation matrix of the binocular camera; and the distortion parameters are radial distortion and tangential distortion. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention is proposed.

[0006] Therefore, the problem to be solved by the present invention is how to use the existing technology. Although Zhang’s calibration method and other image matching algorithms can provide relatively accurate calibration results in static environments, in actual applications, especially during the flight of drones, due to environmental factors, dynamic motion, sensor offset, etc., the dynamic calibration accuracy will be insufficient. At the same time, most of the existing calibration methods are offline calibrations, and the parameters of the camera or sensor cannot be updated in real time during the flight of the drone. This limits the data collection capabilities of drones in dynamic environments, especially for rapidly changing terrain or dynamic obstacle detection. In summary, the existing calibration methods have the problem of low accuracy when collecting and processing three-dimensional terrain data of drones in dynamic environments.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] In a first aspect, an embodiment of the present invention provides an online calibration method for a UAV video sensor based on three-dimensional terrain data, which includes obtaining three-dimensional terrain data of a target area to be inspected, wherein the three-dimensional terrain data includes a reference elevation model and reference point cloud data;

[0009] Dividing the inspection area corresponding to the three-dimensional terrain data to obtain an uncalibrated area and an area to be calibrated;

[0010] Evaluate the online calibration requirement index of each area to be calibrated, where the online calibration requirement index is used to reflect the degree to which the inspection area needs online calibration;

[0011] Determining whether to start online calibration of the area to be calibrated according to the online calibration demand index, when the online calibration demand index exceeds a set threshold, starting online calibration, and when the online calibration demand index does not exceed the set threshold, not starting online calibration;

[0012] When the UAV starts flying inspection, real-time video data and real-time point cloud data are collected in real time through the video sensor, and the video sensor is calibrated online and abnormal points are detected.

[0013] As a preferred solution of the online calibration method of the UAV video sensor based on three-dimensional terrain data of the present invention, the specific step of dividing the inspection area includes: uniformly dividing the flight area based on the grid division method to obtain the initial area;

[0014] Marking the change area in each initial area, wherein the change area includes a strong illumination change area, a strong vibration area and a strong height difference area;

[0015] According to the changed area, the initial area is adjusted into an area to be calibrated and an uncalibrated area, and an area shift index of each area to be calibrated is analyzed. The area shift index is used to reflect the distance that each area to be calibrated needs to be expanded or reduced.

[0016] As a preferred solution of the online calibration method of the UAV video sensor based on three-dimensional terrain data of the present invention, the step of obtaining the regional offset index includes:

[0017] Number the areas to be calibrated;

[0018] Obtain the maximum light intensity difference in the strong light change area, the maximum wind vibration coefficient in the strong vibration area, and the maximum height difference in the strong height difference area, and perform preprocessing and data standardization on them;

[0019] The objective weighting method is used to obtain the weights of the maximum light intensity difference in the strong light change area, the maximum wind vibration coefficient in the strong vibration area, and the maximum height difference in the strong height difference area on the regional deviation index after data standardization.

[0020] Obtain the first boundary coordinates and the second boundary coordinates of the area to be calibrated in the direction of the transmission line, the third boundary coordinates and the fourth boundary coordinates of the circular area in the area to be calibrated, and the fifth center change position and the sixth center change position closest to the area to be calibrated;

[0021] The regional offset index is calculated according to the boundary coordinates, the center change position, the maximum light intensity difference, the maximum wind vibration coefficient, the maximum height difference and their corresponding weights.

[0022] As a preferred solution of the online calibration method of the UAV video sensor based on three-dimensional terrain data of the present invention, the specific steps of adjusting the initial area into the area to be calibrated and the uncalibrated area include:

[0023] Extract the central change position of the change area and the boundary position of the transmission line direction;

[0024] In the initial area, a circle is drawn with the central change position of the change area as the center and the farthest boundary position of the transmission line as the radius to obtain a circular area to be calibrated, which is recorded as the area to be calibrated;

[0025] When there is no area to be calibrated in two adjacent initial areas, the two initial areas are merged into an uncalibrated area;

[0026] When the to-be-calibrated areas corresponding to two adjacent center change positions overlap, the two to-be-calibrated areas are merged, and all the initial areas corresponding to the to-be-calibrated areas are merged into a new to-be-calibrated area;

[0027] When the merging is completed, the target area to be inspected is traversed, the area offset index of the area to be calibrated is calculated, the size of the calibrated area is adjusted according to the area offset index, the blank area is filled as the uncalibrated area, and the adjacent uncalibrated areas are merged.

[0028] As a preferred solution of the online calibration method of the UAV video sensor based on three-dimensional terrain data of the present invention, the step of obtaining the online calibration requirement index includes:

[0029] Obtain the intensity difference, wind vibration coefficient, and height difference corresponding to the unit length in the area to be calibrated, and perform preprocessing and data standardization;

[0030] The weights of the light intensity difference, wind vibration coefficient, and height difference after data standardization on the online calibration demand index are obtained through an objective weighting method.

[0031] An online calibration requirement index is calculated according to the strong light difference, the wind vibration coefficient, the height difference and their corresponding weights.

[0032] As a preferred solution of the online calibration method of the UAV video sensor based on three-dimensional terrain data of the present invention, the online calibration process includes:

[0033] Extract reference elevation model and reference point cloud data from 3D terrain data;

[0034] Preprocess the real-time video data and real-time point cloud data;

[0035] Perform feature matching on the preprocessed real-time video data and the preprocessed real-time point cloud data;

[0036] A real-time elevation model is constructed according to the preprocessed real-time video data, and feature matching is performed between the real-time elevation model and the real-time point cloud data and the reference elevation model and the reference point cloud data, respectively.

[0037] As a preferred solution of the online calibration method of the UAV video sensor based on three-dimensional terrain data of the present invention, the process of abnormal point detection includes:

[0038] Preprocess the real-time video data and real-time point cloud data;

[0039] Perform feature matching on the preprocessed real-time video data and the real-time point cloud data, and mark the position where the comparison difference result is greater than the difference threshold as the first abnormal point;

[0040] A real-time elevation model is constructed according to the preprocessed real-time video data, and the preprocessed real-time elevation model and real-time point cloud data are feature matched with the reference elevation model and reference point cloud data respectively, and the position that is abnormal to both the reference elevation model and the reference point cloud data is obtained and marked as the second abnormal point;

[0041] The positions of the first abnormal point and the second abnormal point are compared. When the positions are repeated, an acoustic signal of a warning notification and a visual annotation are issued. When the positions are not repeated, only a visual annotation is issued.

[0042] In a second aspect, an embodiment of the present invention provides an online calibration system for a UAV video sensor based on three-dimensional terrain data, which includes a data acquisition module for acquiring three-dimensional terrain data of a target area to be inspected, wherein the three-dimensional terrain data includes a reference elevation model and reference point cloud data;

[0043] An area division module is used to divide the inspection area corresponding to the three-dimensional terrain data into an uncalibrated area and an area to be calibrated;

[0044] A demand assessment module, which assesses the online calibration demand index of each area to be calibrated, wherein the online calibration demand index is used to reflect the degree to which the inspection area needs online calibration;

[0045] An online calibration module, which determines whether to start the online calibration of the area to be calibrated according to the online calibration demand index, and when the online calibration demand index exceeds a set threshold, the online calibration is started, and when the online calibration demand index does not exceed the set threshold, the online calibration is not started;

[0046] The anomaly detection module collects real-time video data and real-time point cloud data through the video sensor when the UAV starts flying inspection, and performs online calibration and anomaly point detection on the video sensor.

[0047] In a third aspect, an embodiment of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program instructions are executed by the processor, the steps of the online calibration method of a drone video sensor based on three-dimensional terrain data as described in the first aspect of the present invention are implemented.

[0048] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program instructions are executed by a processor, the steps of the online calibration method for a drone video sensor based on three-dimensional terrain data as described in the first aspect of the present invention are implemented.

[0049] The beneficial effects of the present invention are as follows: by evaluating the online calibration demand index of the inspection area, it is intelligently decided whether to start the online calibration, thereby improving the accuracy and real-time performance of three-dimensional terrain data collection and processing by UAVs in dynamic environments, and effectively solving the problem of low accuracy of existing calibration methods when collecting and processing three-dimensional terrain data of UAVs in dynamic environments.

[0050] By matching the features of real-time video data with real-time point cloud data and comparing them with the reference elevation model and reference point cloud data, abnormal points can be accurately identified and warnings can be issued when the positions overlap, thereby achieving real-time and automated detection and annotation of abnormal points, effectively improving the accuracy and efficiency of UAV three-dimensional terrain data collection and processing in dynamic environments.

[0051] By extracting the central change position of the changed area and the boundary position of the transmission line, and combining the relationship between adjacent areas to merge and adjust the areas, while ensuring the accurate calibration of the area, the division of the area to be calibrated and the area not to be calibrated is optimized, thereby achieving more efficient and accurate regional calibration and improving the data processing and analysis accuracy in transmission line inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 It is a flow chart of the online calibration method of UAV video sensor based on three-dimensional terrain data;

[0054] Figure 2 Figure 1 is a computer device diagram for an online calibration method of a UAV video sensor based on three-dimensional terrain data. DETAILED DESCRIPTION

[0055] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0056] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0057] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.

[0058] Example 1

[0059] Reference Figure 1-2 , which is the first embodiment of the present invention, provides an online calibration method for a UAV video sensor based on three-dimensional terrain data, comprising:

[0060] S100: Acquire three-dimensional terrain data of the target area to be inspected, where the three-dimensional terrain data includes a reference elevation model and reference point cloud data;

[0061] S200: Divide the inspection area corresponding to the three-dimensional terrain data to obtain an uncalibrated area and an area to be calibrated;

[0062] S201: The specific steps of dividing the inspection area include: uniformly dividing the flight area based on the grid division method to obtain an initial area;

[0063] Mark the change areas in each initial area, including strong light change areas, strong vibration areas and strong height difference areas;

[0064] According to the changed area, the initial area is adjusted into the area to be calibrated and the uncalibrated area, and the area shift index of each area to be calibrated is analyzed. The area shift index is used to reflect the distance that each area to be calibrated needs to be expanded or reduced.

[0065] S202: The step of obtaining the regional offset index includes:

[0066] Number the areas to be calibrated;

[0067] Obtain the maximum light intensity difference in the strong light change area, the maximum wind vibration coefficient in the strong vibration area, and the maximum height difference in the strong height difference area, and perform preprocessing and data standardization on them;

[0068] The objective weighting method is used to obtain the weights of the maximum light intensity difference in the strong light change area, the maximum wind vibration coefficient in the strong vibration area, and the maximum height difference in the strong height difference area on the regional deviation index after data standardization.

[0069] Obtain the first boundary coordinates and the second boundary coordinates of the area to be calibrated in the direction of the transmission line, the third boundary coordinates and the fourth boundary coordinates of the circular area in the area to be calibrated, and the fifth center change position and the sixth center change position closest to the area to be calibrated;

[0070] The regional offset index is calculated based on the boundary coordinates, center change position, maximum light intensity difference, maximum wind vibration coefficient, maximum height difference and their corresponding weights.

[0071] S203: The specific steps of adjusting the initial area into the area to be calibrated and the area without calibration include:

[0072] Extract the central change position of the change area and the boundary position of the transmission line direction;

[0073] In the initial area, a circle is drawn with the central change position of the change area as the center and the farthest boundary position of the transmission line as the radius to obtain the circular area to be calibrated, which is the area to be calibrated;

[0074] When there is no area to be calibrated in two adjacent initial areas, the two initial areas are merged into an uncalibrated area;

[0075] When the to-be-calibrated areas corresponding to two adjacent center change positions overlap, the two to-be-calibrated areas are merged, and all the initial areas corresponding to the to-be-calibrated areas are merged into a new to-be-calibrated area;

[0076] When the merging is completed, the target area to be inspected is traversed, the area offset index of the area to be calibrated is calculated, the size of the calibrated area is adjusted according to the area offset index, the blank area is filled as the uncalibrated area, and the adjacent uncalibrated areas are merged.

[0077] S300: Evaluate an online calibration requirement index of each area to be calibrated, where the online calibration requirement index is used to reflect the degree to which the inspection area needs online calibration;

[0078] S301: The steps of obtaining the online calibration demand index include:

[0079] Obtain the intensity difference, wind vibration coefficient, and height difference corresponding to the unit length in the area to be calibrated, and perform preprocessing and data standardization;

[0080] The weights of the light intensity difference, wind vibration coefficient, and height difference after data standardization on the online calibration demand index are obtained through an objective weighting method.

[0081] The online calibration requirement index is calculated based on the glare difference, wind vibration coefficient, height difference and their corresponding weights.

[0082] S302: The process of online calibration includes:

[0083] Extract reference elevation model and reference point cloud data from 3D terrain data;

[0084] Preprocess the real-time video data and real-time point cloud data;

[0085] Perform feature matching on the preprocessed real-time video data and the preprocessed real-time point cloud data;

[0086] A real-time elevation model is constructed based on the preprocessed real-time video data, and feature matching is performed between the real-time elevation model and the real-time point cloud data and the reference elevation model and the reference point cloud data respectively.

[0087] S400: determining whether to start online calibration of the area to be calibrated according to the online calibration demand index, when the online calibration demand index exceeds a set threshold, starting the online calibration, when the online calibration demand index does not exceed the set threshold, not starting the online calibration;

[0088] S500: When the drone starts flying inspection, real-time video data and real-time point cloud data are collected through the video sensor, and the video sensor is calibrated online and abnormal points are detected.

[0089] S501: The process of outlier detection includes:

[0090] Preprocess the real-time video data and real-time point cloud data;

[0091] Perform feature matching on the preprocessed real-time video data and the real-time point cloud data, and mark the position where the comparison difference result is greater than the difference threshold as the first abnormal point;

[0092] A real-time elevation model is constructed according to the preprocessed real-time video data, and the preprocessed real-time elevation model and real-time point cloud data are feature matched with the reference elevation model and reference point cloud data respectively, and the position that is abnormal to both the reference elevation model and the reference point cloud data is obtained and marked as the second abnormal point;

[0093] The positions of the first abnormal point and the second abnormal point are compared. When the positions are repeated, an acoustic signal and a visual annotation of a warning notification are issued. When the positions are not repeated, only a visual annotation is issued.

[0094] Furthermore, this embodiment also provides an online calibration system for a UAV video sensor based on three-dimensional terrain data, comprising:

[0095] A data acquisition module is used to acquire three-dimensional terrain data of the target area to be inspected, wherein the three-dimensional terrain data includes a reference elevation model and reference point cloud data;

[0096] The area division module divides the inspection area corresponding to the three-dimensional terrain data to obtain the uncalibrated area and the area to be calibrated;

[0097] The demand assessment module evaluates the online calibration demand index of each area to be calibrated. The online calibration demand index is used to reflect the degree to which online calibration is needed in the inspection area.

[0098] The online calibration module determines whether to start the online calibration of the area to be calibrated according to the online calibration demand index. When the online calibration demand index exceeds the set threshold, the online calibration is started. When the online calibration demand index does not exceed the set threshold, the online calibration is not started.

[0099] The anomaly detection module collects real-time video data and real-time point cloud data through video sensors when the drone starts flying inspections, and performs online calibration and anomaly point detection on the video sensors.

[0100] In summary, by evaluating the online calibration demand index of the inspection area, it is intelligently decided whether to start online calibration, thereby improving the accuracy and real-time performance of three-dimensional terrain data collection and processing by UAVs in dynamic environments, and effectively solving the problem of low accuracy of existing calibration methods when collecting and processing three-dimensional terrain data of UAVs in dynamic environments.

[0101] By matching the features of real-time video data with real-time point cloud data and comparing them with the reference elevation model and reference point cloud data, abnormal points can be accurately identified and warnings can be issued when the positions overlap, thereby achieving real-time and automated detection and annotation of abnormal points, effectively improving the accuracy and efficiency of UAV three-dimensional terrain data collection and processing in dynamic environments.

[0102] By extracting the central change position of the changed area and the boundary position of the transmission line, and combining the relationship between adjacent areas to merge and adjust the areas, while ensuring the accurate calibration of the area, the division of the area to be calibrated and the area not to be calibrated is optimized, thereby achieving more efficient and accurate regional calibration and improving the data processing and analysis accuracy in transmission line inspection.

[0103] Example 2

[0104] Reference Figure 1 - Figure 2 , which is the second embodiment of the present invention, provides an online calibration method for UAV video sensors based on three-dimensional terrain data. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0105] Before the UAV flies, the three-dimensional terrain data of the target area to be inspected is obtained and the inspection area is divided, which includes the uncalibrated area and the area to be calibrated; the online calibration demand index of each area to be calibrated is evaluated, and the online calibration demand index is used to reflect the degree to which the inspection area needs online calibration; when the online calibration demand index exceeds the set threshold, the online calibration is started; when the online calibration demand index does not exceed the set threshold, the online calibration is not started; when the UAV starts flying inspection, real-time video data and real-time point cloud data are collected in real time through the video sensor, and the video sensor is calibrated online and abnormal points are detected.

[0106] Furthermore, the specific steps of obtaining three-dimensional terrain data and dividing the inspection area are as follows: obtaining three-dimensional terrain data before flight, the three-dimensional terrain data includes a reference elevation model and reference point cloud data; dividing the inspection area according to the three-dimensional terrain data, and in the process of dividing the area: step 1, evenly dividing the flight area based on the grid division method to obtain the initial area; step 2, marking the change area in each initial area, the change area includes a strong light change area, a strong vibration area and a strong height difference area; step 3, according to the marked change area, adjusting the initial area to the area to be calibrated and the uncalibrated area, analyzing the area offset index of each area to be calibrated, the area offset index is used to reflect the distance that each area to be calibrated needs to be expanded or reduced, and readjusting the boundary of the area to be calibrated according to the area offset index. This makes the division of the inspection area more in line with the actual environmental conditions. The area to be calibrated needs further precise adjustment, while the uncalibrated area can be processed according to the standard process. This distinction is conducive to improving the efficiency and accuracy of inspections.

[0107] In this embodiment, the grid division method is a method of dividing a spatial area into several small areas (grids), which is commonly used in finite element analysis and spatial data management, so that each area can be processed and analyzed uniformly. The strong light change area refers to an area where the light conditions change dramatically, and the rate of change of its brightness is significant, such as the brightness change difference per second exceeds a certain threshold, such as 100cd / m 2 . Strong vibration area refers to the area where the building structure vibrates strongly due to the influence of wind speed. For common buildings or bridges, the wind vibration coefficient is generally between 0.1 and 2.0. If the wind vibration coefficient is large (such as greater than 1.5), it means that the structure is strongly vibrated by the wind and is considered a strong vibration area. Strong height difference area refers to the area where the terrain height changes dramatically, which may affect the visual system of the flight equipment. The height difference of more than 30 meters is considered a strong height difference area.

[0108] Furthermore, the specific steps of adjusting the initial area to the area to be calibrated and the area to be uncalibrated include: extracting the center change position of the change area and the boundary position of the transmission line direction; drawing a circle in the initial area with the center change position of the change area as the center and the farthest boundary position of the transmission line direction as the radius to obtain a circular area to be calibrated, recorded as the area to be calibrated; when there is no area to be calibrated in two adjacent initial areas, the two initial areas are merged into the area to be uncalibrated; when the areas to be calibrated corresponding to two adjacent center change positions overlap, the two areas to be calibrated are merged, and all the initial areas corresponding to the area to be calibrated are merged into a new area to be calibrated; when the areas to be calibrated corresponding to two adjacent center change positions do not overlap, it is determined whether the non-overlapping area and the two areas to be calibrated are in the same initial area: if in the same initial area Then the non-overlapping area and the two areas to be calibrated are merged and recorded as the area to be calibrated; if there is a non-overlapping area and the two areas to be calibrated do not exist in the same initial area, but the initial areas to which the two areas to be calibrated belong are adjacent, then the two areas to be calibrated are recorded as the areas to be calibrated, and the remaining areas in the two initial areas are recorded as uncalibrated areas; if the two areas to be calibrated do not overlap and do not exist in the same initial area, and the initial areas to which the two areas to be calibrated belong are not adjacent, then the two areas to be calibrated are recorded as the areas to be calibrated, and the remaining areas of the two initial areas corresponding to the two areas to be calibrated are recorded as uncalibrated areas; when the merger is completed, traverse the target area to be inspected, calculate the area offset index of the area to be calibrated, adjust the size of the calibration area according to the area offset index, fill the blank area as the uncalibrated area, and merge the adjacent uncalibrated areas.

[0109] In this embodiment, a circle is drawn with the center change position as the center and the farthest boundary position as the radius to obtain a circular area to be calibrated, so as to simplify the boundary definition of complex areas, make the division of the area to be calibrated more intuitive and easy to manage, and also facilitate subsequent area adjustment and optimization. Adjacent initial areas where there is no center change position are merged into uncalibrated areas to reduce unnecessary calibration work and improve inspection efficiency. At the same time, merging overlapping areas can avoid duplication of work, ensure the continuity and consistency of the inspection area, and improve the coverage and efficiency of the inspection. By dynamically optimizing the inspection area according to the regional offset index, the accuracy and effectiveness of the inspection can be ensured. Filling blank areas as uncalibrated areas and merging adjacent uncalibrated areas help to simplify the management of inspection areas.

[0110] Furthermore, the steps for obtaining the regional offset index are as follows: numbering the areas to be calibrated respectively; obtaining the maximum light intensity difference in the strong illumination change area, the maximum wind vibration coefficient in the strong vibration area, and the maximum height difference in the strong height difference area, preprocessing them, and performing data standardization; and obtaining the weights of the maximum light intensity difference in the strong illumination change area, the maximum wind vibration coefficient in the strong vibration area, and the maximum height difference in the strong height difference area after data standardization for the regional offset index respectively through an objective weighting method; obtaining the first boundary coordinates and the second boundary coordinates of the area to be calibrated in the direction of the transmission line, the third boundary coordinates and the fourth boundary coordinates of the circular area in the area to be calibrated, and the fifth center change position and the sixth center change position closest to the area to be calibrated; wherein the order of the coordinates in the direction of the transmission line is: the first boundary coordinates of the area to be calibrated, the third boundary coordinates of the circular area in the area to be calibrated, the fifth center change position closest to the area to be calibrated, the sixth center change position closest to the area to be calibrated, the fourth boundary coordinates of the circular area in the area to be calibrated, and the second boundary coordinates of the area to be calibrated; the regional offset index is obtained according to each boundary coordinate and the center change position through the regional offset index calculation formula.

[0111] In this embodiment, by obtaining the maximum light intensity difference, the maximum wind vibration coefficient, and the maximum height difference and standardizing them, the influence of different dimensions and numerical ranges can be eliminated, so that the data can be compared under the same standard, providing an accurate data basis for subsequent weight calculation and determination of the regional offset index.

[0112] Furthermore, the regional shift index calculation formula is:

[0113]

[0114] Where D i is the regional offset index of the i-th region to be calibrated, GQ max is the maximum light intensity difference in the strong light change area, QZ max is the maximum wind vibration coefficient in the strong vibration area, GD max is the maximum height difference in the strong height difference area, α1 is GQ max Right D i The weight of QZ max Right D i The weight of GD max Right D i The weight of , e is a natural constant, is the distance that the first boundary coordinate (x1, y1) of the area to be calibrated in the i-th area needs to be offset. is the distance that the second boundary coordinate (x2, y2) of the area to be calibrated in the i-th area needs to be offset. is the distance from the second boundary coordinate (x2, y2) of the area to be calibrated to the fourth boundary coordinate (x4, y4) of the circular area in the area to be calibrated. They are respectively the distance from (x1, y1) to (x3, y3) of the i-th area to be calibrated, the distance from (x4, y4) to (x6, y6), the distance from (x2, y2) to (x6, y6), the distance from (x3, y3) to (x5, y5), and the distance from (x1, y1) to (x5, y5); among which, (x3, y3) is the third boundary coordinate of the circular area in the area to be calibrated, (x5, y5) is the fifth center change position closest to the area to be calibrated, and (x6, y6) is the sixth center change position closest to the area to be calibrated.

[0115] In this embodiment,

[0116] The maximum light intensity difference in the strong light change area can be obtained by light intensity meter, photometer, etc. max =5cd / m 2 (candela per square meter), the maximum wind vibration coefficient in the strong vibration area can be obtained by QZ max = 1.5 (unitless), the maximum height difference in the strong height difference area can be measured by LiDAR by emitting laser pulses and receiving reflected light to obtain the distance, GD max =0.3km, α1=0.3, α2=0.3, α3 is 0.4, and D i =0.275km, that is, the second boundary coordinate (x2, y2) of the area to be calibrated is displaced by 0.275km in the direction of the fourth boundary coordinate (x4, y4) of the circular area in the area to be calibrated.

[0117] Furthermore, the steps for obtaining the online calibration requirement index are as follows: obtain the intensity difference, wind vibration coefficient, and height difference corresponding to the unit length in the area to be calibrated, preprocess them, and perform data standardization; and use an objective weighting method to obtain the weights of the intensity difference, wind vibration coefficient, and height difference after data standardization for the online calibration requirement index; and calculate the online calibration requirement index through the online calibration requirement index calculation formula.

[0118] Furthermore, the online calibration demand index calculation formula is:

[0119]

[0120] In the formula, ZB i is the online calibration demand index of the i-th area to be calibrated, is the intensity difference of the kth unit length in the i-th area to be calibrated, is the wind vibration coefficient of the kth unit length in the i-th area to be calibrated, is the height difference of the kth unit length in the i-th area to be calibrated, k is the number of the unit length in the area to be calibrated, k = 1, 2, K, β1 is About ZB i The weight of About ZB i The weight of About ZB i The weight of .

[0121] In this embodiment, when k=1, the threshold is set to 0.5. β1=0.3, β2=0.3, β3=0.4, we get ZB i =0.476, which does not exceed the set threshold, so the online calibration is not enabled for the area to be calibrated within this unit length.

[0122] Furthermore, the process of online calibration is as follows: when online calibration is turned on during flight, the video sensor carried by the drone is used to collect real-time video or point cloud data, and the real-time video data and real-time point cloud data are feature matched with the three-dimensional terrain data in real time, and the geometric calibration parameters of the video sensor are output.

[0123] In this embodiment, the three-dimensional terrain data is usually derived from laser radar (LIDAR), aerial photography or geographic information system (GIS). Through these data, the elevation, slope, obstacles and other information of the area can be obtained, so as to be divided into multiple small areas for evaluation. The terrain characteristics of different areas determine the degree of need for online calibration. Different inspection areas may face different flight environments and sensor accuracy requirements. Therefore, the evaluation of the online calibration demand index can dynamically adjust the calibration strategy according to the characteristics of the area. When the demand index exceeds the predetermined threshold, it means that the data quality of the sensor may be greatly affected and calibration adjustment is required; if the demand index is lower than the threshold, calibration can be temporarily not performed to save computing resources and time.

[0124] Furthermore, the process of matching the real-time video data and the real-time point cloud data with the three-dimensional terrain data in real time is as follows: extracting the reference elevation model and the reference point cloud data from the three-dimensional terrain data; preprocessing the real-time video data and the real-time point cloud data; matching the preprocessed real-time video data with the preprocessed real-time point cloud data; constructing a real-time elevation model based on the preprocessed real-time video data, and matching the real-time elevation model and the preprocessed real-time point cloud data with the reference elevation model and the reference point cloud data in the three-dimensional terrain data. By using the feature extraction and matching algorithms in computer vision and point cloud registration technology (such as ICP algorithm, Iterative Closest Point, point cloud precise registration algorithm), feature points are extracted from the real-time collected image or point cloud, and matched with the reference data to obtain the precise spatial relationship between the sensor and the environment. The real-time elevation model reflects the dynamic changes of the terrain during the flight of the UAV. By matching the real-time elevation model with the reference elevation model, the flight path and sensor position can be corrected more accurately.

[0125] Furthermore, the process of outlier detection is as follows: preprocessing the real-time video data and the real-time point cloud data; performing feature matching on the preprocessed real-time video data and the real-time point cloud data, and marking the position where the comparison difference result is greater than the difference threshold as the first outlier; constructing a real-time elevation model based on the preprocessed real-time video data, and performing feature matching on the preprocessed real-time elevation model and the real-time point cloud data with the reference elevation model and the reference point cloud data in the three-dimensional terrain data, respectively, to obtain a position that is abnormal to both the reference elevation model and the reference point cloud data, and marking the position as the second outlier; if the positions of the first outlier and the second outlier are repeated, a warning notification sound signal and a visually marked position are issued, otherwise only a visually marked position is issued.

[0126] In this embodiment, the first abnormal point may be caused by sensor error, environmental interference (such as illumination change, fog, reflection, etc.) or other factors. The second abnormal point may be an abnormality caused by sudden terrain change, equipment failure or environmental problems. Through two independent matching processes (video / point cloud and reference elevation / reference point cloud), the abnormal situation in the data can be judged more accurately. For example, changes in video data may be caused by changes in illumination, while changes in point cloud data may reflect changes in terrain or sensor errors. Multiple verifications are used to reduce the occurrence of false alarms and ensure the reliability of abnormal point detection; the system automatically detects and marks abnormal points, so that possible faults or environmental changes during the flight can be identified in a timely manner, and corresponding measures can be taken in a timely manner to ensure the smooth progress of the mission.

[0127] In summary, the embodiments of the present application significantly improve the accuracy, reliability and adaptability of drone inspections through comprehensive feature matching and abnormal point detection of real-time video data, point cloud data and three-dimensional terrain data.

[0128] Example 3

[0129] This embodiment also provides a computer device, which is applicable to an online calibration method of a drone video sensor based on three-dimensional terrain data, and includes a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement a forced oscillation detection and positioning method for a distribution network as proposed in the above embodiment.

[0130] This embodiment further provides a storage medium on which a computer program is stored. When the program is executed by a processor, a forced oscillation detection and positioning method for a distribution network is implemented as proposed in the above embodiment.

[0131] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0132] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0133] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0134] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0135] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0136] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An online calibration method for UAV video sensor based on three-dimensional terrain data, characterized by: The method comprises obtaining three-dimensional terrain data of a target area to be inspected, wherein the three-dimensional terrain data comprises a reference elevation model and reference point cloud data; Dividing the inspection area corresponding to the three-dimensional terrain data to obtain an uncalibrated area and an area to be calibrated; Evaluate the online calibration requirement index of each area to be calibrated, where the online calibration requirement index is used to reflect the degree to which the inspection area needs online calibration; Determining whether to start online calibration of the area to be calibrated according to the online calibration demand index, when the online calibration demand index exceeds a set threshold, starting online calibration, and when the online calibration demand index does not exceed the set threshold, not starting online calibration; When the UAV starts flying inspection, real-time video data and real-time point cloud data are collected in real time through the video sensor, and the video sensor is calibrated online and abnormal points are detected.

2. The online calibration method for UAV video sensor based on three-dimensional terrain data according to claim 1, characterized in that: The specific steps of dividing the inspection area include: uniformly dividing the flight area based on the grid division method to obtain an initial area; Marking the change area in each initial area, wherein the change area includes a strong illumination change area, a strong vibration area and a strong height difference area; According to the changed area, the initial area is adjusted into an area to be calibrated and an uncalibrated area, and an area shift index of each area to be calibrated is analyzed. The area shift index is used to reflect the distance that each area to be calibrated needs to be expanded or reduced.

3. The online calibration method for UAV video sensor based on three-dimensional terrain data according to claim 2, characterized in that: The step of obtaining the regional offset index comprises: Number the areas to be calibrated; Obtain the maximum light intensity difference in the strong light change area, the maximum wind vibration coefficient in the strong vibration area, and the maximum height difference in the strong height difference area, and perform preprocessing and data standardization on them; The objective weighting method is used to obtain the weights of the maximum light intensity difference in the strong light change area, the maximum wind vibration coefficient in the strong vibration area, and the maximum height difference in the strong height difference area on the regional deviation index after data standardization. Obtain the first boundary coordinates and the second boundary coordinates of the area to be calibrated in the direction of the transmission line, the third boundary coordinates and the fourth boundary coordinates of the circular area in the area to be calibrated, and the fifth center change position and the sixth center change position closest to the area to be calibrated; The regional offset index is calculated according to the boundary coordinates, the center change position, the maximum light intensity difference, the maximum wind vibration coefficient, the maximum height difference and their corresponding weights.

4. The online calibration method for UAV video sensor based on three-dimensional terrain data according to claim 3, characterized in that: The specific steps of adjusting the initial area into the area to be calibrated and the uncalibrated area include: Extract the central change position of the change area and the boundary position of the transmission line direction; In the initial area, a circle is drawn with the central change position of the change area as the center and the farthest boundary position of the transmission line as the radius to obtain a circular area to be calibrated, which is recorded as the area to be calibrated; When there is no area to be calibrated in two adjacent initial areas, the two initial areas are merged into an uncalibrated area; When the to-be-calibrated areas corresponding to two adjacent center change positions overlap, the two to-be-calibrated areas are merged, and all the initial areas corresponding to the to-be-calibrated areas are merged into a new to-be-calibrated area; When the merging is completed, the target area to be inspected is traversed, the area offset index of the area to be calibrated is calculated, the size of the calibrated area is adjusted according to the area offset index, the blank area is filled as the uncalibrated area, and the adjacent uncalibrated areas are merged.

5. The online calibration method for UAV video sensor based on three-dimensional terrain data according to claim 4, characterized in that: The step of obtaining the online calibration demand index includes: Obtain the intensity difference, wind vibration coefficient, and height difference corresponding to the unit length in the area to be calibrated, and perform preprocessing and data standardization; The weights of the light intensity difference, wind vibration coefficient, and height difference after data standardization on the online calibration demand index are obtained through an objective weighting method. An online calibration requirement index is calculated according to the strong light difference, the wind vibration coefficient, the height difference and their corresponding weights.

6. The online calibration method for UAV video sensor based on three-dimensional terrain data according to claim 5, characterized in that: The online calibration process includes: Extract reference elevation model and reference point cloud data from 3D terrain data; Preprocess the real-time video data and real-time point cloud data; Perform feature matching on the preprocessed real-time video data and the preprocessed real-time point cloud data; A real-time elevation model is constructed according to the preprocessed real-time video data, and feature matching is performed between the real-time elevation model and the real-time point cloud data and the reference elevation model and the reference point cloud data, respectively.

7. The online calibration method for UAV video sensor based on three-dimensional terrain data according to claim 6, characterized in that: The process of outlier detection includes: Preprocess the real-time video data and real-time point cloud data; Perform feature matching on the preprocessed real-time video data and the real-time point cloud data, and mark the position where the comparison difference result is greater than the difference threshold as the first abnormal point; A real-time elevation model is constructed according to the preprocessed real-time video data, and the preprocessed real-time elevation model and real-time point cloud data are feature matched with the reference elevation model and reference point cloud data respectively, and the position that is abnormal to both the reference elevation model and the reference point cloud data is obtained and marked as the second abnormal point; The positions of the first abnormal point and the second abnormal point are compared. When the positions are repeated, an acoustic signal of a warning notification and a visual annotation are issued. When the positions are not repeated, only a visual annotation is issued.

8. An online calibration system for a UAV video sensor based on three-dimensional terrain data, based on the online calibration method for a UAV video sensor based on three-dimensional terrain data according to any one of claims 1 to 7, characterized in that: Also includes, A data acquisition module, which acquires three-dimensional terrain data of a target area to be inspected, wherein the three-dimensional terrain data includes a reference elevation model and reference point cloud data; An area division module is used to divide the inspection area corresponding to the three-dimensional terrain data into an uncalibrated area and an area to be calibrated; A demand assessment module, which assesses the online calibration demand index of each area to be calibrated, wherein the online calibration demand index is used to reflect the degree to which the inspection area needs online calibration; An online calibration module determines whether to start online calibration of the area to be calibrated according to the online calibration demand index, and starts online calibration when the online calibration demand index exceeds a set threshold, and does not start online calibration when the online calibration demand index does not exceed the set threshold; The anomaly detection module collects real-time video data and real-time point cloud data through the video sensor when the UAV starts flying inspection, and performs online calibration and anomaly point detection on the video sensor.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the online calibration method of the UAV video sensor based on three-dimensional terrain data described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the online calibration method of a UAV video sensor based on three-dimensional terrain data described in any one of claims 1 to 7 are implemented.

Images