Unmanned aerial vehicle video sensor online calibration method and system based on three-dimensional terrain data

Abstract

The present application relates to the technical field of computer vision, and particularly relates to a UAV video sensor online calibration method and system based on three-dimensional terrain data. Three-dimensional terrain data of a target area to be inspected is acquired, and the three-dimensional terrain data comprises a reference elevation model and reference point cloud data; an inspection area corresponding to the three-dimensional terrain data is divided to obtain a non-calibration area and a calibration area to be calibrated; an online calibration demand index of each calibration area to be calibrated is evaluated; whether to start online calibration is determined according to the online calibration demand index; when a UAV starts to fly and inspect, real-time video data and real-time point cloud data are collected in real time through a video sensor, and online calibration and abnormal point detection are performed on the video sensor. The present application intelligently determines whether to start online calibration by evaluating the online calibration demand index of an inspection area, so that the accuracy and real-time performance of a UAV in collecting and processing three-dimensional terrain data in a dynamic environment are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a method and system for online calibration of a video sensor of an unmanned aerial vehicle based on three-dimensional terrain data. BACKGROUND

[0002] At present, image-based calibration methods have been widely used in sensor calibration, especially in binocular vision and multi-view vision calibration. Common calibration methods include: Zhang's calibration method: Zhang's calibration method is to obtain the internal and external parameters of the camera and the distortion parameters through the checkerboard pattern image. The advantages of this method are simple operation, high precision, and the ability to reduce errors through optimization algorithms. Epipolar geometry calibration: In a binocular vision system, epipolar geometry calibration is based on corresponding points in images to calculate the relative position and direction relationship between two cameras, thereby obtaining the internal and external parameters of the cameras and the epipolar constraint relationship. SIFT feature matching: In vision calibration, SIFT features (Scale-Invariant Feature Transform) are used to extract key points in images, and the geometric relationship between cameras is established by matching feature points at multiple viewing angles.

[0003] For example, the invention patent with the publication number CN110458947A discloses a three-dimensional reconstruction method and system for power transmission conductor sag, which includes: parameter calibration of a binocular camera, including: determining the internal parameters and distortion parameters of each camera based on the checkerboard image using Zhang's calibration method; wherein the internal parameters are used to represent the internal structure of the binocular camera, and the distortion parameters are used to represent radial distortion and tangential distortion; then, based on the images collected by each camera, the corner points with sudden changes in pixel values on the images are extracted, and the SIFT feature description values at these corner points are calculated; the corresponding control points in different images are matched using Hamming distance, and the pose relationship between the two cameras is established; wherein the control points are pixel points that are prominent in certain attributes in the image, and are isolated points, endpoints of line segments that have maximum or minimum intensity in certain attributes.

[0004] For example, the invention patent with the publication number CN115731303B discloses a three-dimensional reconstruction method for large-span power transmission conductor sag based on bidirectional binocular vision, which includes: the parameters of the binocular camera are calibrated using Zhang's calibration method, and epipolar rectification is performed, including: step 101, calibrating the internal, external, and distortion parameters of the binocular camera; step 102, performing epipolar rectification with reference to the internal, external, and distortion parameters; step 103, setting and determining the focal length f of the camera and the distance b between the centers of the left and right cameras; wherein the internal parameters are the internal structure parameters of the binocular camera; the external parameters are the rotation matrix and translation matrix of the binocular camera; and the distortion parameters are the radial distortion and tangential distortion. SUMMARY

[0005] In view of the problems in the prior art, the present application is proposed.

[0006] Therefore, the problem to be solved by the present application is how to in the prior art, although Zhang's calibration method and other image matching algorithms can provide relatively accurate calibration results in a static environment, in actual application, especially in the flight process of the unmanned aerial vehicle, due to environmental factors, dynamic motion, sensor offset and the like, the dynamic calibration accuracy is insufficient, and the existing calibration methods are mostly offline calibration, which cannot update the parameters of the camera or the sensor in real time in the flight process of the unmanned aerial vehicle. This limits the data acquisition capability of the unmanned aerial vehicle in a dynamic environment, especially for rapid change of the terrain or dynamic obstacle detection. In summary, the existing calibration method has the problem of low accuracy in three-dimensional terrain data acquisition and processing of the unmanned aerial vehicle in a dynamic environment.

[0007] To solve the above technical problems, the present application provides the following technical solutions:

[0008] In a first aspect, the present application provides an unmanned aerial vehicle video sensor online calibration method based on three-dimensional terrain data, which comprises: acquiring three-dimensional terrain data of a target area to be inspected, wherein the three-dimensional terrain data comprises a reference elevation model and reference point cloud data;

[0009] The three-dimensional terrain data corresponding to the inspection area is divided to obtain an uncalibrated area and a to-be-calibrated area;

[0010] An online calibration demand index of each to-be-calibrated area is evaluated, and the online calibration demand index is used to reflect the degree to which the inspection area needs online calibration;

[0011] It is determined whether to start online calibration of the to-be-calibrated area according to the online calibration demand index, when the online calibration demand index exceeds a set threshold, the online calibration is started, and when the online calibration demand index does not exceed the set threshold, the online calibration is not started;

[0012] When the unmanned aerial vehicle starts flight inspection, real-time video data and real-time point cloud data are collected by a video sensor in real time, and online calibration and abnormal point detection are performed on the video sensor.

[0013] As a preferred scheme of the unmanned aerial vehicle video sensor online calibration method based on three-dimensional terrain data, the specific steps of dividing the inspection area comprise: uniformly dividing the flight area based on a grid division method to obtain initial areas;

[0014] In each initial area, a change area is marked, and the change area comprises a strong light change area, a strong vibration area and a strong height difference area;

[0015] According to the change region, the initial region is adjusted into a region to be calibrated and a region without calibration, and a region offset index of each region to be calibrated is analyzed, the region offset index being used to reflect a distance that each region to be calibrated needs to be enlarged or reduced.

[0016] As a preferred scheme of the online calibration method for a UAV video sensor based on three-dimensional terrain data, the step of obtaining the region offset index comprises:

[0017] The regions to be calibrated are numbered.

[0018] The maximum light intensity difference of the strong light change region, the maximum wind vibration coefficient of the strong vibration region, and the maximum height difference of the strong height difference region are obtained, and are preprocessed and standardized.

[0019] The weights of the maximum light intensity difference of the strong light change region, the maximum wind vibration coefficient of the strong vibration region, and the maximum height difference of the strong height difference region to the region offset index after data standardization are obtained by an objective weighting method.

[0020] First boundary coordinates and second boundary coordinates of the region to be calibrated in the direction of the power transmission line, third boundary coordinates and fourth boundary coordinates of a circular region in the region to be calibrated, and fifth center change positions and sixth center change positions closest to the region to be calibrated are obtained.

[0021] The region offset index is calculated according to the boundary coordinates, the center change positions, the maximum light intensity difference, the maximum wind vibration coefficient, the maximum height difference, and the corresponding weights.

[0022] As a preferred scheme of the online calibration method for a UAV video sensor based on three-dimensional terrain data, the specific steps of adjusting the initial region into the region to be calibrated and the region without calibration comprise:

[0023] The center change positions of the change region and the boundary positions of the power transmission line are extracted.

[0024] In the initial region, a circle is drawn with the center change positions of the change region as the center and the farthest boundary positions of the power transmission line as the radius, to obtain a circular region to be calibrated, which is recorded as a region to be calibrated.

[0025] When there is no region to be calibrated in two adjacent initial regions, the two initial regions are merged into a region without calibration.

[0026] When there is an overlap between two regions to be calibrated corresponding to adjacent center change positions, the two regions to be calibrated are merged, and all initial regions corresponding to the regions to be calibrated are merged into a new region to be calibrated.

[0027] When the merging is completed, the target area to be inspected is traversed, the area offset index of the area to be calibrated is calculated, the size of the calibration area is adjusted according to the area offset index, and the blank area is filled as a non-calibration area, and adjacent non-calibration areas are merged.

[0028] As a preferred scheme of the online calibration method of the unmanned aerial vehicle video sensor based on three-dimensional terrain data, the acquisition of the online calibration demand index comprises:

[0029] The strong light difference value, the wind vibration coefficient and the height difference value corresponding to the unit length in the area to be calibrated are obtained, preprocessed and standardized;

[0030] The weights of the strong light difference value, the wind vibration coefficient and the height difference value after data standardization on the online calibration demand index are obtained by the objective weighting method;

[0031] The online calibration demand index is calculated according to the strong light difference value, the wind vibration coefficient, the height difference value and the corresponding weights.

[0032] As a preferred scheme of the online calibration method of the unmanned aerial vehicle video sensor based on three-dimensional terrain data, the process of the online calibration comprises:

[0033] The reference elevation model and the reference point cloud data are extracted from the three-dimensional terrain data;

[0034] The real-time video data and the real-time point cloud data are preprocessed;

[0035] The preprocessed real-time video data and the preprocessed real-time point cloud data are matched in features;

[0036] The real-time elevation model is constructed according to the preprocessed real-time video data, and the real-time elevation model and the real-time point cloud data are matched in features with the reference elevation model and the reference point cloud data respectively.

[0037] As a preferred scheme of the online calibration method of the unmanned aerial vehicle video sensor based on three-dimensional terrain data, the process of the abnormal point detection comprises:

[0038] The real-time video data and the real-time point cloud data are preprocessed;

[0039] The preprocessed real-time video data and the real-time point cloud data are matched in features, and the positions with a comparison difference greater than a difference threshold are marked as first abnormal points;

[0040] According to the pre-processed real-time video data, a real-time elevation model is constructed, the pre-processed real-time elevation model and real-time point cloud data are respectively matched with the reference elevation model and reference point cloud data, and positions simultaneously abnormal to the reference elevation model and reference point cloud data are obtained, and are marked as second abnormal points;

[0041] When the positions of the first abnormal points and the second abnormal points are compared, when the positions are repeated, an audible signal and a visual mark of a warning notice are issued, and when the positions are not repeated, only the visual mark is issued.

[0042] In a second aspect, an embodiment of the present application provides a UAV video sensor online calibration system based on three-dimensional terrain data, which comprises a data acquisition module, which acquires three-dimensional terrain data of a target area to be inspected, the three-dimensional terrain data comprising a reference elevation model and reference point cloud data;

[0043] A region division module divides the inspection region corresponding to the three-dimensional terrain data to obtain a non-calibration region and a calibration region to be calibrated.

[0044] A demand evaluation module evaluates an online calibration demand index of each calibration region to be calibrated, the online calibration demand index reflecting the degree to which the inspection region needs online calibration.

[0045] An online calibration module determines whether to start online calibration of the calibration region to be calibrated according to the online calibration demand index, starts online calibration when the online calibration demand index exceeds a set threshold, and does not start online calibration when the online calibration demand index does not exceed the set threshold.

[0046] An anomaly detection module acquires real-time video data and real-time point cloud data in real time through a video sensor when a UAV starts flying inspection, and performs online calibration and anomaly point detection on the video sensor.

[0047] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory and a processor, the memory storing a computer program, wherein the computer program instructions are executed by the processor to implement the steps of the UAV video sensor online calibration method based on three-dimensional terrain data according to the first aspect of the present application.

[0048] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, wherein the computer program instructions are executed by the processor to implement the steps of the UAV video sensor online calibration method based on three-dimensional terrain data according to the first aspect of the present application.

[0049] The present application has the beneficial effects that: by evaluating the online calibration demand index of the inspection area, it is intelligently determined whether to start online calibration, thereby improving the precision and real-time performance of the unmanned aerial vehicle in dynamic environment when collecting and processing three-dimensional terrain data, and effectively solving the problem of low precision of the existing calibration method when the unmanned aerial vehicle collects and processes three-dimensional terrain data in dynamic environment.

[0050] By feature matching real-time video data with real-time point cloud data, and comparing with reference elevation model and reference point cloud data, the abnormal points are accurately identified and a warning is issued when the positions overlap, thereby realizing real-time and automatic detection and labeling of abnormal points, and effectively improving the accuracy and efficiency of the unmanned aerial vehicle in dynamic environment when collecting and processing three-dimensional terrain data.

[0051] By extracting the center change position of the change area and the boundary position of the power line direction, and combining the relationship of adjacent areas for region merging and adjustment, the division of the region to be calibrated and the region without calibration is optimized while ensuring accurate calibration of the region, thereby realizing more efficient and accurate region calibration, and improving the data processing and analysis accuracy in power line inspection. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0053] Fig. 1 Flowchart of the unmanned aerial vehicle video sensor online calibration method based on three-dimensional terrain data;

[0054] Fig. 2 Computer device diagram of the unmanned aerial vehicle video sensor online calibration method based on three-dimensional terrain data. DETAILED DESCRIPTION

[0055] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings.

[0056] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from the description, and those skilled in the art can make similar generalizations without departing from the concept of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0057] Second, the "one embodiment" or "an embodiment" referred to herein means a specific feature, structure, characteristic, or combination of features and / or characteristics described herein that can be included in at least one implementation of the present application. The various appearances of "in one embodiment" or "an embodiment" in the specification do not all refer to the same embodiment, although they can.

[0058] Embodiment 1

[0059] Reference Figs. 1-2 For the first embodiment of the present application, the embodiment provides a method for online calibration of a UAV video sensor based on three-dimensional terrain data, comprising,

[0060] S100: Obtain three-dimensional terrain data of a target area to be inspected, the three-dimensional terrain data comprising a reference elevation model and reference point cloud data;

[0061] S200: Divide the inspection area corresponding to the three-dimensional terrain data to obtain a non-calibration area and a calibration area to be calibrated;

[0062] S201: The specific steps of dividing the inspection area include: uniformly dividing the flight area based on a grid division method to obtain an initial area;

[0063] Label the change area in each initial area, the change area including a strong light change area, a strong vibration area, and a strong height difference area;

[0064] According to the change area, the initial area is adjusted to the calibration area to be calibrated and the non-calibration area, and the area offset index of each calibration area to be calibrated is analyzed, the area offset index being used to reflect the distance that each calibration area to be calibrated needs to be expanded or reduced.

[0065] S202: The steps for obtaining the area offset index include:

[0066] Number the calibration area to be calibrated;

[0067] Obtain the maximum light intensity difference of the strong light change area, the maximum wind vibration coefficient of the strong vibration area, and the maximum height difference of the strong height difference area, and preprocess and standardize the data;

[0068] Obtain the weight of the maximum light intensity difference of the strong light change area, the maximum wind vibration coefficient of the strong vibration area, and the maximum height difference of the strong height difference area after data standardization on the area offset index by an objective weighting method;

[0069] Obtain the first boundary coordinate and the second boundary coordinate of the calibration area to be calibrated in the direction of the power transmission line, the third boundary coordinate and the fourth boundary coordinate of the circular area in the calibration area to be calibrated, and the fifth center change position and the sixth center change position closest to the calibration area to be calibrated;

[0070] According to the boundary coordinates, the center change position, the maximum light intensity difference, the maximum wind vibration coefficient, the maximum height difference, and the corresponding weight, a region offset index is calculated.

[0071] S203: The specific steps of adjusting the initial region into the to-be-calibrated region and the non-calibration region include:

[0072] The center change position of the change region and the boundary position of the power line direction are extracted;

[0073] In the initial region, a circle is drawn with the center change position of the change region as the center and the farthest boundary position of the power line direction as the radius, to obtain a circular region to be calibrated, i.e., the to-be-calibrated region;

[0074] When there is no to-be-calibrated region in the adjacent two initial regions, the two initial regions are merged into a non-calibration region;

[0075] When the to-be-calibrated regions corresponding to the adjacent two center change positions overlap, the two to-be-calibrated regions are merged, and all initial regions corresponding to the to-be-calibrated regions are merged into a new to-be-calibrated region;

[0076] After the merging is completed, the to-be-inspected target region is traversed, the region offset index of the to-be-calibrated region is calculated, the size of the calibration region is adjusted according to the region offset index, and the blank region is filled as a non-calibration region, and the adjacent non-calibration regions are merged.

[0077] S300: An online calibration demand index of each to-be-calibrated region is evaluated, and the online calibration demand index is used to reflect the degree to which the inspection region needs to use online calibration;

[0078] S301: The steps of obtaining the online calibration demand index include:

[0079] The strong light difference value, the wind vibration coefficient, and the height difference value corresponding to the unit length in the to-be-calibrated region are obtained, and are preprocessed and standardized;

[0080] The weights of the strong light difference value, the wind vibration coefficient, and the height difference value after data standardization on the online calibration demand index are obtained by an objective weighting method;

[0081] According to the strong light difference value, the wind vibration coefficient, the height difference value, and the corresponding weight, an online calibration demand index is calculated.

[0082] S302: The process of online calibration includes:

[0083] Reference elevation model and reference point cloud data are extracted from three-dimensional terrain data;

[0084] Real-time video data and real-time point cloud data are preprocessed;

[0085] The preprocessed real-time video data is matched with the preprocessed real-time point cloud data in features;

[0086] A real-time elevation model is constructed according to the preprocessed real-time video data, and the real-time elevation model and the real-time point cloud data are respectively matched with a reference elevation model and a reference point cloud data in features.

[0087] S400: Determine whether to start online calibration of the to-be-calibrated region according to the online calibration demand index. When the online calibration demand index exceeds a set threshold, start online calibration. When the online calibration demand index does not exceed the set threshold, do not start online calibration.

[0088] S500: When the unmanned aerial vehicle starts to fly for inspection, real-time video data and real-time point cloud data are collected in real time by a video sensor, and the video sensor is calibrated and detected for abnormal points in real time.

[0089] S501: The process of detecting abnormal points includes:

[0090] The real-time video data and the real-time point cloud data are preprocessed;

[0091] The preprocessed real-time video data is matched with the real-time point cloud data in features, and positions with a comparison difference greater than a difference threshold are marked as first abnormal points.

[0092] A real-time elevation model is constructed according to the preprocessed real-time video data, and the preprocessed real-time elevation model and the real-time point cloud data are respectively matched with a reference elevation model and a reference point cloud data in features, to obtain positions that are simultaneously abnormal to the reference elevation model and the reference point cloud data, which are marked as second abnormal points.

[0093] The positions of the first abnormal points and the second abnormal points are compared. When the positions are repeated, an audible warning signal and a visual mark are issued. When the positions are not repeated, only the visual mark is issued.

[0094] Further, the embodiment also provides an unmanned aerial vehicle video sensor online calibration system based on three-dimensional terrain data, comprising,

[0095] A data acquisition module acquires three-dimensional terrain data of a target region to be inspected, and the three-dimensional terrain data includes a reference elevation model and reference point cloud data.

[0096] A region division module divides an inspection region corresponding to the three-dimensional terrain data to obtain a calibration-free region and a to-be-calibrated region.

[0097] A demand evaluation module evaluates an online calibration demand index of each to-be-calibrated region, and the online calibration demand index is used to reflect a degree to which the inspection region needs online calibration.

[0098] an online calibration module determines whether to start online calibration of the to-be-calibrated region according to the online calibration demand index, and when the online calibration demand index exceeds a set threshold, the online calibration is started, and when the online calibration demand index does not exceed the set threshold, the online calibration is not started;

[0099] an anomaly detection module, when the unmanned aerial vehicle starts flight inspection, real-time video data and real-time point cloud data are collected in real time through the video sensor, online calibration and anomaly point detection are performed on the video sensor.

[0100] In summary, by evaluating the online calibration demand index of the inspection region, it is intelligently determined whether to start online calibration, thereby improving the accuracy and real-time performance of the unmanned aerial vehicle in collecting and processing three-dimensional terrain data in a dynamic environment, effectively solving the problem of low accuracy of existing calibration methods in collecting and processing three-dimensional terrain data of the unmanned aerial vehicle in a dynamic environment.

[0101] By feature matching real-time video data and real-time point cloud data, and comparing with reference elevation model and reference point cloud data, the anomaly points are accurately identified and a warning is issued when the positions overlap, thereby realizing real-time and automatic detection and labeling of anomaly points, effectively improving the accuracy and efficiency of the unmanned aerial vehicle in collecting and processing three-dimensional terrain data in a dynamic environment.

[0102] By extracting the center change position of the change region and the boundary position of the power line direction, and combining the relationship of adjacent regions for region merging and adjustment, the division of the to-be-calibrated region and the non-calibrated region is optimized while ensuring accurate calibration of the region, thereby realizing more efficient and accurate region calibration, and improving the data processing and analysis accuracy in power line inspection.

[0103] Embodiment 2

[0104] Reference Fig. 1 - Fig. 2 As a second embodiment of the present application, the embodiment provides a method for online calibration of a video sensor of an unmanned aerial vehicle based on three-dimensional terrain data. In order to verify the beneficial effects of the present application, economic benefit calculation and simulation experiments are used for scientific demonstration.

[0105] Before the unmanned aerial vehicle flies, three-dimensional terrain data of a target region to be inspected is obtained and the inspection region is divided into a non-calibrated region and a to-be-calibrated region; the online calibration demand index of each to-be-calibrated region is evaluated, the online calibration demand index is used to reflect the degree to which the online calibration is needed in the inspection region; when the online calibration demand index exceeds a set threshold, the online calibration is started; when the online calibration demand index does not exceed the set threshold, the online calibration is not started; when the unmanned aerial vehicle starts flight inspection, real-time video data and real-time point cloud data are collected in real time through the video sensor, and online calibration and anomaly point detection are performed on the video sensor.

[0106] Further, the specific steps of dividing the inspection area according to the three-dimensional terrain data are as follows: obtaining three-dimensional terrain data before flight, the three-dimensional terrain data including a reference elevation model and reference point cloud data; dividing the inspection area according to the three-dimensional terrain data, in the process of dividing the area: step 1, uniformly dividing the flight area based on a grid division method to obtain an initial area; step 2, marking out a change area in each initial area, the change area including a strong light change area, a strong vibration area, and a strong height difference area; step 3, adjusting the initial area to a region to be calibrated and a region without calibration according to the marked change area, analyzing a region offset index of each region to be calibrated, the region offset index being used to reflect a distance that each region to be calibrated needs to be expanded or reduced, and adjusting the boundary of the region to be calibrated according to the region offset index. This makes the division of the inspection area more in line with the actual environmental conditions, the region to be calibrated needs further accurate adjustment, and the region without calibration can be processed according to a standard process, which is beneficial to improve the inspection efficiency and accuracy.

[0107] In the embodiment, the grid division method is a method of dividing a space area into a plurality of small areas (grids), which is commonly used in finite element analysis and space data management, and can make each area be uniformly processed and analyzed. The strong light change area refers to an area with a dramatic change in light conditions, and the change rate of brightness is significant, such as a brightness change difference per second exceeding a certain threshold, for example, 100 cd / m 2 The strong vibration area refers to an area with strong vibration of a building structure due to wind speed, and for common buildings or bridges, the wind vibration coefficient is generally between 0.1 and 2.0. If the wind vibration coefficient is large (such as greater than 1.5), it means that the structure is subjected to strong wind vibration, and is regarded as a strong vibration area. The strong height difference area refers to an area with a dramatic change in terrain height, which may affect the visual system of the flight equipment, and a height difference exceeding 30 meters is regarded as a strong height difference area.

[0108] Further, the specific steps of adjusting the initial region into the to-be-calibrated region and the non-calibration region include: extracting the center change position of the change region and the boundary position of the power line direction; drawing a circle in the initial region with the center change position of the change region as the center and the farthest boundary position of the power line direction as the radius to obtain a circular region to be calibrated, denoted as the to-be-calibrated region; when there is no to-be-calibrated region in the adjacent two initial regions, merging the two initial regions into a non-calibration region; when the to-be-calibrated regions corresponding to the adjacent two center change positions overlap, merging the two to-be-calibrated regions, and simultaneously merging all initial regions corresponding to the to-be-calibrated regions into a new to-be-calibrated region; when the to-be-calibrated regions corresponding to the adjacent two center change positions do not overlap, judging whether the non-overlapping region and the two to-be-calibrated regions are in the same initial region: if they are in the same initial region, merging the non-overlapping region and the two to-be-calibrated regions and denoting them as the to-be-calibrated region; if the non-overlapping region and the two to-be-calibrated regions are not in the same initial region, but the initial regions to which the two to-be-calibrated regions belong are in an adjacent relationship, denoting the two to-be-calibrated regions as the to-be-calibrated region and the remaining regions in the two initial regions as the non-calibration region; if the two to-be-calibrated regions do not overlap and are not in the same initial region, and the initial regions to which the two to-be-calibrated regions belong are also not in an adjacent relationship, denoting the two to-be-calibrated regions as the to-be-calibrated region and the remaining regions in the two initial regions corresponding to the to-be-calibrated regions as the non-calibration region; after the merging is completed, traversing the to-be-inspected target region, calculating the region offset index of the to-be-calibrated region, adjusting the size of the calibration region according to the region offset index, filling the blank region as the non-calibration region, and merging the adjacent non-calibration regions.

[0109] In the embodiment, a circle is drawn with the center change position as the center and the farthest boundary position as the radius to obtain a circular region to be calibrated, so as to simplify the definition of the boundary of the complex region, make the division of the to-be-calibrated region more intuitive and easy to manage, and facilitate subsequent region adjustment and optimization. Merging the adjacent initial regions without the center change position into the non-calibration region can reduce unnecessary calibration work and improve the inspection efficiency. Meanwhile, merging the overlapping regions can avoid repeated work, ensure the continuity and consistency of the inspection region, and improve the coverage rate and efficiency of the inspection. Dynamically optimizing the inspection region according to the region offset index can ensure the accuracy and effectiveness of the inspection. Filling the blank region as the non-calibration region and merging the adjacent non-calibration regions can help to simplify the management of the inspection region.

[0110] Further, the step of obtaining the regional offset index comprises: numbering the to-be-calibrated regions respectively; obtaining the maximum light intensity difference of the strong light change region, the maximum wind vibration coefficient of the strong vibration region, and the maximum height difference of the strong height difference region, preprocessing the same, and performing data standardization; and obtaining the weight of the maximum light intensity difference of the strong light change region, the maximum wind vibration coefficient of the strong vibration region, and the maximum height difference of the strong height difference region after data standardization to the regional offset index through an objective weighting method; obtaining the first boundary coordinate and the second boundary coordinate of the to-be-calibrated region in the power transmission line direction, the third boundary coordinate and the fourth boundary coordinate of the to-be-calibrated region in the circle region, and the fifth center change position and the sixth center change position closest to the to-be-calibrated region; wherein the sequence of the coordinates in the power transmission line direction is: the first boundary coordinate of the to-be-calibrated region, the third boundary coordinate of the to-be-calibrated region in the circle region, the fifth center change position closest to the to-be-calibrated region, the sixth center change position closest to the to-be-calibrated region, the fourth boundary coordinate of the to-be-calibrated region in the circle region, and the second boundary coordinate of the to-be-calibrated region; and obtaining the regional offset index according to the boundary coordinates and the center change positions through a regional offset index calculation formula.

[0111] In the embodiment, by obtaining the maximum light intensity difference, the maximum wind vibration coefficient, and the maximum height difference and performing standardization, the influence of different dimensions and numerical ranges can be eliminated. The data is compared under the same standard, thereby providing an accurate data basis for subsequent weight calculation and determination of the regional offset index.

[0112] Further, the regional offset index calculation formula is:

[0113]

[0114] In the formula, D i is the regional offset index of the i-th to-be-calibrated region, GQ max is the maximum light intensity difference of the strong light change region, QZ max is the maximum wind vibration coefficient of the strong vibration region, and GD max is the maximum height difference of the strong height difference region, α1 is the weight of GQ max to D i , α2 is the weight of QZ max to D i , α3 is the weight of GD max to D i , e is a natural constant, is the distance that the first boundary coordinate (x1, y1) of the to-be-calibrated region of the i-th region needs to offset, is the distance that the second boundary coordinate (x2, y2) of the to-be-calibrated region of the i-th region needs to offset, is the distance from the second boundary coordinate (x2, y2) of the to-be-calibrated region to the fourth boundary coordinate (x4, y4) of the to-be-calibrated region in the circle region, respectively, the distance of the to-be-calibrated region (x1, y1) to (x3, y3), the distance of (x4, y4) to (x6, y6), the distance of (x2, y2) to (x6, y6), the distance of (x3, y3) to (x5, y5), the distance of (x1, y1) to (x5, y5) in the i-th region; wherein (x3, y3) is the third boundary coordinate of the circular region in the to-be-calibrated region, (x5, y5) is the fifth center change position closest to the to-be-calibrated region, and (x6, y6) is the sixth center change position closest to the to-be-calibrated region.

[0115] In the embodiment,

[0116] The maximum light intensity difference of the strong light change area can be obtained by a light intensity meter, a photometer, etc. max = 5 cd / m 2 (Candela per square meter), the maximum wind vibration coefficient of the strong vibration area can be obtained by QZ max = 1.5 (unitless), the maximum height difference of the strong height difference area can be obtained by a laser radar (LiDAR) by emitting a laser pulse and receiving reflected light to measure the distance, GD max = 0.3 km, α1 = 0.3, α2 = 0.3, and α3 = 0.4, so that D i = 0.275 km, that is, the second boundary coordinate (x2, y2) of the to-be-calibrated region is displaced by 0.275 km in the direction of the fourth boundary coordinate (x4, y4) of the circular region in the to-be-calibrated region.

[0117] Further, the obtaining step of the online calibration demand index is: obtaining the strong light difference value, the wind vibration coefficient, and the height difference value corresponding to a unit length in the to-be-calibrated region, preprocessing the same, and performing data standardization; and obtaining the weight of the strong light difference value, the wind vibration coefficient, and the height difference value after data standardization with respect to the online calibration demand index through an objective weighting method; and calculating the online calibration demand index through an online calibration demand index calculation formula.

[0118] Further, the online calibration demand index calculation formula is:

[0119]

[0120] In the formula, ZB i is the online calibration demand index of the i-th to-be-calibrated region, is the strong light difference value of the k-th unit length in the i-th to-be-calibrated region, is the wind vibration coefficient of the k-th unit length in the i-th to-be-calibrated region, is the height difference value of the kth unit length in the ith region to be calibrated, k is the number of the unit length in the region to be calibrated, k = 1, 2, K, β1is the weight of ZB to ZB i , β2is the weight of ZB to ZB i , and β3is the weight of ZB to ZB i .

[0121] In this embodiment, when k = 1, the threshold is set to 0.5, β1= 0.3, β2= 0.3, β3= 0.4, and ZB i = 0.476, which does not exceed the set threshold, so the region to be calibrated in the unit length does not start online calibration.

[0122] Further, the process of online calibration is as follows: when online calibration is started during flight, the real-time video or point cloud data is collected by the video sensor carried by the unmanned aerial vehicle, and the real-time video data and real-time point cloud data are matched with the three-dimensional terrain data in real time to output the geometric calibration parameters of the video sensor.

[0123] In this embodiment, the three-dimensional terrain data is usually obtained from laser radar (LIDAR), aerial photography or geographic information system (GIS). Through these data, the elevation, slope and obstacles of the region can be obtained, so as to divide the region into multiple small regions for evaluation. The terrain characteristics of different regions determine the degree of demand for online calibration. Different inspection regions may face different flight environments and sensor accuracy requirements, therefore, the evaluation of online calibration demand index can dynamically adjust the calibration strategy according to the characteristics of the region. When the demand index exceeds the predetermined threshold, it means that the data quality of the sensor may be greatly affected, and calibration adjustment is needed; if the demand index is lower than the threshold, calibration can be temporarily suspended to save computing resources and time.

[0124] Further, the process of feature matching real-time video data and real-time point cloud data with three-dimensional terrain data is: extracting reference elevation model and reference point cloud data from three-dimensional terrain data; preprocessing real-time video data and real-time point cloud data; feature matching preprocessed real-time video data with preprocessed real-time point cloud data; constructing real-time elevation model according to preprocessed real-time video data, and feature matching real-time elevation model and preprocessed real-time point cloud data with reference elevation model and reference point cloud data in three-dimensional terrain data. Through feature extraction and matching algorithm in computer vision and point cloud registration technology (such as ICP algorithm, Iterative Closest Point, point cloud precise registration algorithm), feature points are extracted from real-time collected images or point clouds, and matched with reference data to obtain accurate spatial relationship between sensor and environment. Real-time elevation model reflects dynamic changes of terrain during UAV flight, and by matching real-time elevation model with reference elevation model, flight path and sensor position can be corrected more accurately.

[0125] Further, the process of anomaly point detection is: preprocessing real-time video data and real-time point cloud data; feature matching preprocessed real-time video data with real-time point cloud data, and marking positions with difference greater than difference threshold as first anomaly points; constructing real-time elevation model according to preprocessed real-time video data, and feature matching preprocessed real-time elevation model and real-time point cloud data with reference elevation model and reference point cloud data in three-dimensional terrain data to obtain positions simultaneously abnormal to reference elevation model and reference point cloud data, and marking the positions as second anomaly points; if first anomaly point and second anomaly point positions are repeated, issuing warning notification sound signal and visualizing marking position, otherwise only issuing visualized marking position.

[0126] In this embodiment, the first anomaly point may be caused by sensor error, environmental interference (such as light change, cloud, reflection, etc.) or other factors. The second anomaly point may be caused by terrain mutation, equipment failure or environmental problems. Through two independent matching processes (video / point cloud and reference elevation / reference point cloud), the abnormal situation in the data can be more accurately judged. For example, changes in video data may be caused by light changes, while changes in point cloud data may reflect changes in terrain or sensor errors. Through multiple verification, false positives are reduced, ensuring the reliability of anomaly point detection; through automatic detection and marking of anomaly points by the system, possible faults or environmental changes during flight can be identified in time, and appropriate measures can be taken in time to ensure the smooth progress of the task.

[0127] In summary, the embodiment of the application significantly improves the accuracy, reliability and self-adaptive ability of the unmanned aerial vehicle inspection by comprehensive feature matching and abnormal point detection of real-time video data, point cloud data and three-dimensional terrain data.

[0128] Embodiment 3

[0129] The embodiment also provides a computer device suitable for the case of the online calibration method for the unmanned aerial vehicle video sensor based on three-dimensional terrain data, including a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions to realize the forced oscillation detection and positioning method for the power distribution network as proposed in the above embodiment.

[0130] The embodiment also provides a storage medium having a computer program stored thereon, and the program is executed by a processor to realize the forced oscillation detection and positioning method for the power distribution network as proposed in the above embodiment.

[0131] The computer device can be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0132] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the application or the parts of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, etc.

[0133] The logic and / or steps represented in the flow diagrams, or otherwise described herein, for example, can be considered as a sequence of executable instructions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber (optical), and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example via an optical scanner, then compiled, interpreted, or otherwise processed, and stored in a computer memory in a form that can be later executed by a computer. In this context, a "computer-readable medium" can be any means that can store the program for use by or in connection with the instruction execution system, apparatus, or device.

[0134] The foregoing description, for purposes of explanation, only is specific to the preferred embodiment setting forth the novel aspects of the application. There can be many variations made to the preferred embodiment, without departing from the spirit and scope of the application. For example, the steps can be performed in a different order or other steps can be added or omitted. Specific dimensions can be varied. The application's technical features can be implemented in hardware, software, firmware, or a combination thereof. The application's technical features can be implemented in software or firmware, stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, the technical features can be implemented with any or a combination of the following, which are known in the art: a discrete logic circuit(s) having logic gates for implementing logic functions upon an application of data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array(s) (PGA), a field programmable gate array(s) (FPGA), etc.

[0135] It is to be understood that the technical features of the present application can be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, a plurality of steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, the technical features can be implemented with any or a combination of the following, which are known in the art: a discrete logic circuit(s) having logic gates for implementing logic functions upon an application of data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array(s) (PGA), a field programmable gate array(s) (FPGA), etc.

[0136] It should be noted that the above examples are only used to illustrate the technical solutions of the present application and not limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the present application, and all of them should be covered in the scope of the claims of the present application.

Claims

1. A method for online calibration of UAV video sensors based on three-dimensional terrain data, characterized in that: This includes acquiring three-dimensional terrain data of the target area to be inspected, wherein the three-dimensional terrain data includes a reference elevation model and reference point cloud data; The inspection area corresponding to the three-dimensional terrain data is divided to obtain uncalibrated areas and areas to be calibrated. Assess the online calibration demand index for each area to be calibrated, the online calibration demand index being used to reflect the degree to which the inspection area needs online calibration; Whether to enable online calibration of the area to be calibrated is determined based on the online calibration demand index. When the online calibration demand index exceeds a set threshold, online calibration is enabled; when the online calibration demand index does not exceed the set threshold, online calibration is not enabled. When the UAV begins its flight inspection, real-time video data and real-time point cloud data are collected in real time through video sensors, and the video sensors are calibrated online and anomaly detection is performed.

2. The online calibration method for UAV video sensors based on three-dimensional terrain data as described in claim 1, characterized in that: The process of dividing the inspection area corresponding to the three-dimensional terrain data includes: uniformly dividing the flight area based on the grid division method to obtain the initial area; Within each initial region, the changing regions are marked, including areas of strong light change, areas of strong vibration, and areas of strong height difference. Based on the changed region, the initial region is adjusted into a region to be calibrated and an uncalibrated region. The region offset index of each region to be calibrated is analyzed. The region offset index is used to reflect the distance that each region to be calibrated needs to be expanded or shrunk.

3. The online calibration method for UAV video sensors based on three-dimensional terrain data as described in claim 2, characterized in that: The steps for obtaining the region offset index include: Number the areas to be calibrated; The maximum light intensity difference in areas of strong light variation, the maximum wind vibration coefficient in areas of strong vibration, and the maximum height difference in areas of strong height difference are obtained, and then preprocessed and standardized. The weights of the maximum light intensity difference in the strong light variation zone, the maximum wind vibration coefficient in the strong vibration zone, and the maximum height difference in the strong height difference zone on the regional offset index were obtained by using the objective weighting method. Obtain the coordinates of the first and second boundaries of the area to be calibrated along the direction of the transmission line, the coordinates of the third and fourth boundaries of the circular area in the area to be calibrated, and the positions of the fifth and sixth centers that are closest to the area to be calibrated. The regional offset index is calculated based on the boundary coordinates, the center change position, the maximum light intensity difference, the maximum wind vibration coefficient, the maximum height difference, and their corresponding weights.

4. The online calibration method for UAV video sensors based on three-dimensional terrain data as described in claim 3, characterized in that: The specific steps for adjusting the initial region into the region to be calibrated and the uncalibrated region include: Extract the center location of the change area and the boundary location of the transmission line direction; In the initial region, a circle is drawn with the center of the changing region as the center and the farthest boundary of the transmission line as the radius. This circle is the area to be calibrated and is denoted as the area to be calibrated. If neither of the two adjacent initial regions has a region to be calibrated, then the two initial regions are merged into an uncalibrated region. If the regions to be calibrated corresponding to two adjacent center change positions overlap, the two regions to be calibrated will be merged, and all the initial regions corresponding to the regions to be calibrated will be merged into a new region to be calibrated. After the merging is completed, the target area to be inspected is traversed, the area offset index of the area to be calibrated is calculated, the size of the calibrated area is adjusted according to the area offset index, and blank areas are filled into uncalibrated areas, and adjacent uncalibrated areas are merged.

5. The online calibration method for UAV video sensors based on three-dimensional terrain data as described in claim 4, characterized in that: The steps for obtaining the online calibration demand index include: Obtain the intensity light difference, wind vibration coefficient, and height difference per unit length in the area to be calibrated, and then preprocess and standardize the data. The weights of the intensity light difference, wind vibration coefficient, and height difference on the online calibration demand index were obtained by using the objective weighting method. The online calibration demand index is calculated based on the intensity light difference, the wind vibration coefficient, the height difference, and their corresponding weights.

6. The online calibration method for UAV video sensors based on three-dimensional terrain data as described in claim 5, characterized in that: The online calibration process includes: Extract reference elevation models and reference point cloud data from 3D terrain data; Preprocess real-time video data and real-time point cloud data; Feature matching is performed between the preprocessed real-time video data and the preprocessed real-time point cloud data; A real-time elevation model is constructed based on the preprocessed real-time video data, and the real-time elevation model and real-time point cloud data are respectively matched with the reference elevation model and reference point cloud data.

7. The online calibration method for UAV video sensors based on three-dimensional terrain data as described in claim 6, characterized in that: The anomaly detection process includes: Preprocess real-time video data and real-time point cloud data; The preprocessed real-time video data and real-time point cloud data are matched for features, and the locations where the difference results are greater than the difference threshold are marked as the first anomaly. A real-time elevation model is constructed based on the preprocessed real-time video data. The preprocessed real-time elevation model and real-time point cloud data are then matched with the reference elevation model and reference point cloud data to identify the locations that are simultaneously abnormal in both the reference elevation model and reference point cloud data, and these locations are marked as the second anomaly points. Compare the positions of the first anomaly point and the second anomaly point. If the positions are the same, issue an audible warning signal and a visual label. If the positions are different, only issue a visual label.

8. An online calibration system for UAV video sensors based on three-dimensional terrain data, based on the online calibration method for UAV video sensors based on three-dimensional terrain data according to any one of claims 1 to 7, characterized in that: It also includes, The data acquisition module acquires three-dimensional terrain data of the target area to be inspected, including a reference elevation model and reference point cloud data. The region division module divides the inspection area corresponding to the three-dimensional terrain data to obtain uncalibrated areas and areas to be calibrated. The demand assessment module evaluates the online calibration demand index for each area to be calibrated. The online calibration demand index is used to reflect the degree to which the inspection area needs online calibration. The online calibration module determines whether to enable online calibration of the area to be calibrated based on the online calibration demand index. When the online calibration demand index exceeds a set threshold, online calibration is enabled; when the online calibration demand index does not exceed the set threshold, online calibration is not enabled. The anomaly detection module collects real-time video data and real-time point cloud data through video sensors when the UAV begins its flight inspection, and performs online calibration and anomaly detection on the video sensors.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the online calibration method for UAV video sensors based on three-dimensional terrain data as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the online calibration method for UAV video sensors based on three-dimensional terrain data as described in any one of claims 1 to 7.

Images