Binocular camera measuring platform and image measuring system

Abstract

The invention relates to the technical field of computer vision, in particular to a binocular camera measuring platform and an image measuring system, which comprise a vision measuring system, a three-axis stable holder control system, a vision measuring system and an image measuring system, the vision measurement system is used for enabling a target to reflect or emit a light beam to reach a vision sensor through an optical system, an electronic element completes real-time high-speed collection of collected information and preprocesses collected grayscale image or color image data, one part is transmitted to the display device, and the other part is transmitted to the display device. And the other part is transmitted to an image processing system to complete effective processing of the image, and target extraction and pose measurement tasks are completed. An unmanned aerial vehicle binocular stereoscopic measurement technology is adopted to measure the arcing horn and the like, a stereoscopic measurement binocular camera is carried by the unmanned aerial vehicle to carry out one-time imaging on a part needing to be measured, point-to-point and point-line distance measurement and the like are supported based on an image measurement system, and the method can be widely applied to various measurement requirements of power equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and specifically to a binocular camera measurement platform and an image measurement system. Background Art

[0002] The routine inspection of transmission lines by unmanned aerial vehicles is developing in a normal way. However, at present, it is only limited to the naked-eye recognition of visible light photos or the analysis of spatial distances through certain image processing. The analysis of the spatial distances of some power grid equipment requires a measurement accuracy of up to centimeter level to meet the requirements, such as the measurement of arcing horns, ice coating thickness, vibration damper displacement, conductor clamp offset discharge gap, lightning strike position, tower member size, jumper-tower distance measurement, etc. For example, before the annual ice melting test in the plateau area, the discharge gap needs to be measured, and during the annual power outage maintenance, each tower needs to be measured. During daily operation, the overhead ground wire is affected by factors such as unbalanced tension of the ground wire, its own technical process, and ice coating, etc., which is likely to cause the loosening of the discharge gap plate electrode or rod electrode, resulting in the deformation, increase or decrease of the discharge gap. When the discharge gap is too small, it is easy to cause the gap to be broken down by the ice melting current, resulting in the failure of ice melting; when the discharge gap is too large, it will affect the lightning protection effect of the line. The same situation also exists for arcing horns during daily operation. To ensure that the discharge gap and arcing horns meet the requirements of regulations and specifications, it is necessary for operators to repeatedly climb the tower for inspection and measurement every year.

[0003] The current operation methods have the following problems:

[0004] Traditional measurement methods using steel tape measures are affected by factors such as the skill level of personnel and measurement methods, and there are errors to varying degrees. For some key positions, during the measurement process, the measurement personnel need to go down to the conductor for measurement. Due to the human body weight and balance conditions, the measurement error is increased. When personnel climb the tower for measurement, in addition to being affected by the environment in the plateau area, there are also operation risks such as induced electricity. Usually, each person can complete the measurement of 1-2 towers per day, which is time-consuming, laborious, and inefficient. Moreover, the high-frequency use of tower climbing high-altitude operations greatly increases the risks such as personal injury and death. Traditional photogrammetry has many errors in data acquisition, data processing, result measurement, etc., and cannot meet the requirement that the mean square error is better than 1 cm. Airborne lidar technology mainly obtains decimeter-level laser point clouds over a large range, and also cannot meet the requirement that the mean square error in a small range is better than 1 cm. For the three-dimensional modeling based on traditional photogrammetry or airborne lidar technology, according to the principle of error accumulation, the accuracy of its model is lower than the data accuracy and cannot meet the requirements either. High-precision ground-based three-dimensional laser scanners can theoretically meet the measurement accuracy of millimeters, but have high requirements for measurement distance and visibility, and the operation efficiency is low.

[0005] Under this background, there is an urgent need for an indirect measurement device and technical solution for power grid equipment that can be fast, efficient, safe, highly accurate, standardized, and popularized. Summary of the Invention

[0006] The object of the present invention is to provide a binocular camera measurement platform and an image measurement system to solve the problems raised in the above-mentioned background technology.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A binocular camera measurement platform and an image measurement system, comprising:

[0009] A vision measurement system and a three-axis stabilized gimbal control system for stabilizing the vision measurement system. The vision measurement system and the image measurement system. The vision measurement system is used for the target reflected or emitted light beam to reach the vision sensor through the optical system. The electronic components complete the real-time high-speed acquisition of the collected information, preprocess the collected grayscale image or color image data, transmit a part to the display device, and transmit the other part to the image processing system to complete the effective processing of the image, and complete the target extraction and pose measurement tasks.

[0010] Further, the binocular vision measurement system specifically includes two cameras that simultaneously acquire digital images of the target within the overlapping field of view, and processes the images through relevant theories such as projective transformation to reconstruct the target position and pose.

[0011] Further, the vision measurement system includes a camera mounted on a drone. Based on the SDK software development kit of the drone and the open API interface of the drone, the payload communicates with the drone flight control, GPS, and video transmission internal systems to achieve control and data acquisition functions.

[0012] Further, the three-axis stabilized gimbal control system includes a mathematical model of the gimbal motor. The mathematical model of the gimbal motor infers the accurate position and pose information of the target based on the images acquired by the camera.

[0013] Further, the mathematical model of the gimbal motor infers the accurate position and pose information of the target based on the images acquired by the camera specifically by using the collinearity condition equation solution method. The collinearity condition equation solution method includes the image point coordinate error equation method based on the collinearity condition equation, the multi-image space resection method, and the space resection method. The image plane coordinates of the control points in the space resection method based on the collinearity equation are used as the observations to solve for the internal and external orientation elements and distortion variations of the camera.

[0014] Further, the equation of the collinearity condition equation solution method is expressed as:

[0015]

[0016] In the formula: x and y are the image plane coordinates;

[0017] x0, y0, and f are the interior orientation elements of the photograph;

[0018] X s , Y s , Z s , ψ, ω, and κ are the exterior orientation elements of the photograph;

[0019] X, Y, and Z are the object space coordinates;

[0020] a i , b i , c i (i = 1, 2, 3) are the direction cosines formed by the exterior orientation angular elements of the photograph.

[0021] Furthermore, the distortion correction includes that the radial lens distortion difference △r can be expressed by the following odd polynomial:

[0022] △r = k1r 3 + k2r 5 + k3r 7 + L Equation 1

[0023]

[0024] In the equation: r — the radial distance of the image point;

[0025] x, y — the coordinates of the image point;

[0026] x0, y0 — the coordinates of the principal point of the image;

[0027] k i (i = 1, 2, 3L) — the radial distortion parameters.

[0028] Furthermore, in close-range photogrammetry, the decentering distortion varies with the change of the focal length D, and there is also a change in the decentering distortion of the ground objects at different focal lengths D. Considering the decentering distortion correction at the close-range condition with the focal length D is shown in the following equation:

[0029]

[0030] In the equation: △x D , △y D — the components of the decentering distortion difference at the focal length D;

[0031] f — the principal distance at the focal length D;

[0032] D — the focal length;

[0033] p1, p2 — the decentering distortion parameters.

[0034] Furthermore, spatial resection:

[0035] The work of calculating the coordinates of model points or ground points by using the interior orientation elements of two images in a stereo pair, the coordinates of homologous image points, and the relative orientation elements or exterior orientation elements of the stereo pair.

[0036] Furthermore, space resection:

[0037] A method for determining the exterior orientation elements of an image by using ground control points and their image points on the image. This method requires at least three ground control points that are not collinear.

[0038] Pyramid method: Solving the exterior orientation elements of an image by using the collinearity condition equation

[0039]

[0040] Compared with the prior art, the beneficial effects of the present invention are:

[0041] The present invention uses the unmanned aerial vehicle binocular stereo measurement technology to measure spatial distances such as the arcing angle and discharge gap. The unmanned aerial vehicle is equipped with a stereo measurement binocular camera to take a one-time image of the part to be measured according to the specified flight path, and the data is transmitted back in real time through the 4G / 5G network. After obtaining the data, the PC-side image measurement system is used for intelligent data analysis to complete the analysis work of the spatial distance in real time. The measurement accuracy can reach a medium error of better than 1 cm, solving the problems of efficiency, safety, reliability, etc. of the traditional measurement method. Based on the image measurement system, it supports point-to-point, point-to-line distance measurement, line-line angle measurement, line-plane angle measurement, and three-point area measurement. The project results can be widely applied to various measurement requirements of power equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a diagram showing the relationship between world coordinates, image coordinates, and camera coordinates of the Zhang Zhengyou algorithm.

[0043] Figure 2 It is the overall framework design diagram of the present invention.

[0044] Figure 3 It is the implementation scheme diagram of the present invention.

[0045] Figure 4 It is the three-dimensional flight path planning flow chart.

[0046] Figure 5 It is the schematic diagram of the tower flight path planning.

[0047] Figure 6 It is the research roadmap of the binocular camera measurement platform and the image measurement system.

[0048] Figure 7 It is the image correction diagram.

[0049] Figure 8 It is the structure diagram of the three-axis stabilized cloud platform.

[0050] Figure 9 It is a forward intersection diagram in space.

[0051] Figure 10 It is a resection diagram in space.

[0052] Figure 11 It is a schematic diagram of global optimization. Specific implementation manners

[0053] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0054] Please refer to Figures 1 to 11 , the present invention provides a technical solution:

[0055] The present invention (project) uses the unmanned aerial vehicle (UAV) binocular stereo measurement technology to measure spatial distances such as the arcing angle and the discharge gap. The UAV is equipped with a stereo measurement binocular camera to perform a one-time imaging of the part to be measured according to the specified flight path, and the data is transmitted back in real time through the 4G / 5G network. After obtaining the data, the PC-side image measurement system is used to perform intelligent data analysis to complete the analysis work of the spatial distance in real time. The measurement accuracy can reach a medium error better than 1 cm, solving the problems of efficiency, safety, reliability, etc. of the traditional measurement method. Based on the image measurement system, it supports point-to-point, point-to-line distance measurement, line-line angle measurement, line-plane angle measurement, and three-point area measurement. The project results can be widely applied to various measurement requirements of power equipment.

[0056] This project realizes long-distance and high-precision intelligent measurement of power grid equipment. Through the UAV stereo measurement technology, the operation efficiency of measuring key positions of power grid equipment can be greatly improved, the measurement accuracy can be improved, the operation risks such as induced electricity can be reduced, the operation cost can be saved, and intelligent and standardized operation can be realized. It is applicable to working scenarios where it is difficult for manpower to reach, such as plateaus. Taking the measurement of the arcing angle and the discharge gap as an example: the measurement operation that originally had to be carried out during power outage maintenance can be integrated into daily live patrol. Moreover, the efficiency of measuring 6 towers per day by 2 people (one team) can be improved to measuring 20 towers per day by 2 people, and the efficiency is increased by 2.3 times, which has a significant effect on reducing power grid failures, reducing failure losses, and comprehensively eliminating potential safety hazards in the power system.

[0057] Traditional measurement methods, such as using a steel tape measure, are affected by factors such as the skill level of personnel and measurement methods, resulting in errors of varying degrees. For some key positions, during the measurement process, the surveyors need to measure under the conductor. Due to the body weight and balance conditions, the measurement error is increased. When personnel climb the tower for measurement, there are operation risks such as induced electricity. Usually, each person can complete the measurement of 1-2 towers per day, which is time-consuming, laborious, and inefficient. Moreover, the high-frequency use of high-altitude tower climbing operations greatly increases the risks such as personal injury and death. Traditional photogrammetry methods have many errors in data acquisition, data processing, and result measurement, and cannot meet the requirement of an absolute error of centimeter level (1-2 cm). Airborne lidar technology mainly obtains decimeter-level laser point clouds over a large range and also cannot meet the requirement of an absolute error of centimeter level (1-2 cm) for a small range. For 3D modeling based on traditional photogrammetry or airborne lidar technology, according to the principle of error accumulation, the model accuracy is lower than the data accuracy and cannot meet the requirements either. High-precision ground-based 3D laser scanners can theoretically meet the measurement accuracy of millimeter level, but have high requirements for the measurement distance and visibility, and the operation efficiency is low.

[0058] The binocular stereo vision measurement method has the advantages of high efficiency, non-contact measurement, simple system structure, and low cost.

[0059] Research on Intelligent Route Planning Technology Based on Laser Point Cloud or Tower Model

[0060] The research content mainly includes point cloud classification technology, key component identification technology of the tower, and refined inspection route planning of the tower, etc., to realize controlling the unmanned aerial vehicle to perform high-precision and intelligent refined inspection of the tower according to the planned route.

[0061] Research on High-Precision Measurement Technology Based on Stereo Vision

[0062] Using image processing technology and machine vision theory to study the long-distance measurement method based on the binocular stereo vision system, designing a long-distance ranging scheme with high measurement accuracy, studying the basic models and theories of the existing binocular vision systems, studying the binocular convergent ranging model and the binocular parallel ranging model, and selecting the ranging model of this paper according to the actual application scenario. Secondly, analyze the influence of the system structure parameters and image coordinates on the error, study the influence trend of the system structure parameters on the error, so as to improve the measurement accuracy of stereo vision. By studying the relevant theories of the binocular system, analyzing the structural characteristics to determine the optimal system structure design, analyzing the most fundamental factor restricting the vision measurement system, the discretization of the detector, determining the theoretical accuracy limit of the system, analyzing the influence mechanism of the model parameter calibration error on the measurement accuracy, and determining the way to improve the accuracy.

[0063] Research on the Integration of Unmanned Aerial Vehicle and High-Precision Measurement Platform

[0064] Integrate the binocular camera and the drone through the SDK to achieve the integration of the drone and the high-precision measurement platform, and study the stability of the integrated system, the coordination of the camera and the video transmission signal, the control of the shooting angle of the binocular camera, etc.

[0065] Research on the Stability Technology of High-Precision Measurement Platform

[0066] Study the stability of the binocular camera mounted on the drone, reasonably design the mounting structure, which is easy to install, improve the stability of the entire measurement system, and enable the structure of the binocular camera to rotate during operation to facilitate finding the most suitable shooting angle and capturing clear images.

[0067] Research on the High-Precision Internal Parameter Solution of the Measurement System

[0068] The distance of the spatial target is hundreds or thousands of times the effective focal length of the camera. Small deviations in the internal and external parameters of the vision measurement system or the image imaging points will cause the measurement error to increase exponentially. Therefore, it is crucial to improve the calibration accuracy. Improve the calibration accuracy of the binocular system through reasonable algorithm design and reduce the influence of factors such as feature point extraction error on the calibration accuracy of the binocular vision system. Study the calibration method of the binocular vision measurement system, realize the estimation of system parameters, and then realize the corresponding relationship between the three-dimensional space and the matching image pairs. Only in this way can the binocular vision measurement system reverse the accurate position and attitude information of the target based on the acquired images.

[0069] Research on the Optimization of Dimension Measurement of the Measurement System

[0070] This solution uses the external parameters of the camera to establish a measurement coordinate system. In order to further eliminate the systematic error introduced by parameter calibration error, a model-based optimization method that uses the external parameters of the camera to compensate for the parameter calibration error is used to achieve this.

[0071] Research on High-Precision Image Distortion Calibration Technology

[0072] For a given system, the idea of improving the measurement accuracy is to improve the fitting accuracy of the model parameters to the system. The fitting accuracy is divided into two parts. One is the fitting accuracy of the model to the system itself, and the other is the model parameter estimation accuracy. The fitting accuracy mainly refers to the fitting accuracy of the single-camera model. For an optical system, the alignment error destroys the optical axis consistency of the optical system and the ideal imaging characteristics. On the one hand, it causes instability in the calibration of the internal parameters of the camera, and may converge to a local optimum during the iterative optimization process; on the other hand, it affects the fitting accuracy of the camera model. The camera model also includes a distortion model, and the factors causing camera distortion are complex and diverse. Therefore, almost all distortions are only approximately fitted. By analyzing the distortion characteristics of the optical system based on aberration theory, study the distortion model of the binocular camera, and thus determine the high-precision distortion calibration technology.

[0073] Research on High-Precision Measurement Management System

[0074] This solution is based on the principle of stereo vision measurement, and a high-precision measurement management system is designed and developed. The design work of the system software mainly includes the integrated development of functional modules such as structured light grating projection, image display, stereo calibration and operation of cameras, 3D reconstruction, visualization of tower information, 2D detection, result display, saving and loading of configuration files, and setting of binocular camera parameters.

[0075] UAV intelligent route planning and design

[0076] To improve the efficiency and intelligence of the refined route planning for transmission line towers, find the best shooting points to capture clear images of the objects to be measured, it is necessary to reasonably set parameters such as shooting distance and angle. The system automatically generates the photographing points of the key components for inspection and the UAV flight assistance points, and generates a high-precision and optimal flight route for this flight. The route planning needs to comprehensively consider conditions such as the characteristics of the UAV model, endurance time, operation efficiency, quality of inspection data collection, and tower type, so as to ensure the quality and efficiency of shooting to the greatest extent under the condition of automatic refined inspection of towers.

[0077] Parameter solution inside the measurement platform

[0078] In 1999, Dr. Zhang Zhengyou from Microsoft Research Institute innovatively proposed a camera calibration algorithm based on a two-dimensional checkerboard image calibration board. This method combines the advantages of good robustness of traditional calibration algorithms and low dependence on calibration objects of self-calibration algorithms. It only requires the movement of the calibration template in two directions without the parameters of the calibration board movement to perform calibration. This method has high calibration accuracy and low cost, and has been widely used in desktop vision systems.

[0079] High-precision distortion correction of images

[0080] The solution based on the collinearity condition equation is the most important and widely used method, which includes the image point coordinate error equation method based on the collinearity condition equation, the multi-image space resection method, the space resection method, and various bundle adjustment methods, etc.

[0081] The space resection method based on the collinearity equation is a calibration method that takes the image coordinates of control points (including object coordinates if necessary) as observations to solve the internal and external orientation elements, distortion differences, and other additional parameters of the camera. It can be further divided into the single-image space resection method and the multi-image space resection method.

[0082] Automatic or semi-automatic measurement technology for specific components

[0083] Analyze and solve the shape of specific components, construct a physical model; enable the measurement software to perform deep learning and recognition through a specific physical model; establish a database to provide underlying support for automation technology.

[0084] Project Technical Route and Implementation Plan

[0085] Overall framework design of this project:

[0086] The project plan consists of three major parts: 3D flight path planning software, binocular camera measurement platform, and image measurement system. First, the flight path is planned through the 3D flight path planning software. Then, the drone is equipped with a binocular camera measurement platform to perform the image acquisition task according to the track file. During the operation, the on-board computer on the binocular camera measurement platform can transmit the captured data to the server in real time through the 4 / 5G network. The ground operation personnel can analyze and measure the images downloaded from the cloud through the PC-side image measurement system and output the measurement report. The project implementation plan is as follows:

[0087] Research on Intelligent Flight Path Planning Technology

[0088] To improve the efficiency and intelligence of the refined flight path planning for transmission line towers, a method for identifying typical components based on Kinect depth images and point cloud edge features is proposed. First, the obtained depth image is processed using a non-linear filtering optimization algorithm to obtain the optimized target point cloud, and the octal neighborhood depth difference algorithm is proposed to extract the point cloud edge. Then, the random sample consensus (RANSAC) algorithm is used to detect the segmented point cloud edge, and the defined edge features are extracted to identify the components. According to the set parameters such as camera focal length, shooting distance, and angle, the system automatically generates the photographing points of the key components for inspection and the flight assistance points of the drone, and generates a high-precision and optimal flight path for this flight. The flight path planning needs to comprehensively consider conditions such as the characteristics of the drone model, endurance time, operation efficiency, inspection data acquisition quality, and tower type to ensure the quality and efficiency of the inspection to the greatest extent under the condition of automatic refined inspection of the tower.

[0089] Research on High-Precision Measurement Technology Based on Stereo Vision

[0090] The research route of the binocular camera measurement platform and the image measurement system shows that the high-precision measurement technology based on stereo vision, the integration of the drone and the high-precision measurement platform, and the high-precision measurement platform stability technology are research carried out for the hardware of the binocular camera measurement platform; the high-precision internal parameter solution of the measurement system, the optimization of the measurement system size measurement, the high-precision image distortion calibration technology, and the high-precision measurement management system are research carried out for the image measurement system (software).

[0091] Visual measurement technology has become the preferred measurement method for close-range space operations due to its outstanding advantages such as high precision, strong concealment, low cost, simple equipment, small size, and low power consumption. Visual measurement technology can not only provide geometric information and color information of the target spacecraft, but also the collected target image information is more in line with human information acquisition needs. It can be seen that the study of measurement methods, related technologies and theories involved in visual measurement systems is of great significance for promoting the progress of space operation technology. According to different visual measurement methods, the targets to be operated can be generally divided into cooperative targets and non-cooperative targets. Cooperative targets refer to targets with some known pattern information that can be used as prior knowledge for the visual system, such as cooperative targets with specific shape markers installed on the target spacecraft. When the spacecraft pattern shape and other information are known, the information collection of cooperative targets is easier to achieve. Non-cooperative targets refer to target spacecrafts whose own information is unknown and cannot provide strong information support for the executing spacecraft, such as scrapped spacecrafts and space debris. At present, various aerospace activities are frequent, and it is particularly important to achieve on-orbit maintenance and the removal of space debris such as faulty aircraft, which can not only improve the efficiency of aerospace development but also save the cost of aerospace activities. Aerospace technology research still faces many problems that need to be solved. The solution of such problems involves key technologies such as high-precision posture measurement of non-cooperative targets.

[0092] The main components of the visual measurement system are cameras. The target reflects or emits light beams through the optical system to reach the visual sensor. The electronic components complete the real-time and high-speed acquisition of the collected information, pre-process the collected grayscale image or color image data, transmit part of it to the display device, and transmit the other part to the image processing system to complete the effective processing of the image, complete the tasks of target extraction and posture measurement. The visual measurement methods are complex and diverse, corresponding to different visual system models, among which the two main visual systems are monocular model and binocular (multi-eye) model. The monocular model is simple, so the monocular method technology is the most, this type of method only uses the image sequence obtained by one camera, combined with the prior knowledge of the target to solve the posture parameters. Due to the accuracy of the monocular measurement method and the trouble of three-dimensional measurement, these factors limit some applications of monocular in the measurement field or bring some difficulties, especially for the case where the target geometric pattern or geometric constraint characteristics are completely unknown. For non-cooperative targets, the three-dimensional reconstruction method based on binocular (multi-eye) shows its superiority. Binocular vision measurement system is divided into two forms. One is that two cameras simultaneously obtain digital images of the target within the overlapping field of view, and the other is that a single mobile camera obtains digital images of the target at different positions in time, and processes the image through related theories such as projection transformation to reconstruct the target position and posture. Binocular vision has many advantages. Compared with the monocular measurement system, it is the simplest visual model that can achieve three-dimensional reconstruction.

[0093] Points in the objective world are three-dimensional points, while points in the image plane are two-dimensional points. The transformation from the objective world to the image plane is accomplished through the camera imaging model. Camera calibration is to solve the parameters of the imaging model, mainly including the internal parameters of the camera (camera focal length, image principal point, radial distortion coefficient, and tangential distortion coefficient) and external parameters (rotation matrix, translation matrix). Solving these parameters lays a solid foundation for the next step of image correction.

[0094] After completing camera calibration, the next task is image correction. This module has two functions. First, it uses the distortion coefficients in the internal parameters obtained from camera calibration to remove the radial and tangential distortions of the image, obtaining an undistorted image. Second, it makes the two cameras that are not in the same plane strictly correspond through the geometric transformation relationship for the undistorted image, so that the epipolar lines of the two images can be on the same horizontal line, and only one-dimensional search in a certain row is required for stereo matching to find the corresponding points.

[0095] Stereo matching is to find the matching corresponding points from images at different viewpoints. Therefore, the matching problem in stereo matching can be regarded as a process of finding the correlation degree between two sets of data. Stereo matching is to calculate the matching cost of the left and right images through one or more matching primitives, then cluster the matching cost to obtain an initial disparity map, and then perform post-processing such as interpolation and fitting on the initial disparity map to obtain the final disparity map. The sixth step is to calculate the depth information of the objects in the scene using the principle of triangulation. The above are the five steps of a complete binocular stereo vision system.

[0096] The most basic goal of stereo vision research is to obtain the three-dimensional information of the scene. To achieve this goal, a complete stereo vision process usually includes six parts: image acquisition, camera calibration, feature extraction, stereo matching, three-dimensional coordinate calculation, and post-processing. Image acquisition is the basis of stereo vision technology research. There are various ways of image acquisition, which are determined by the specific application occasions and purposes. The acquisition of stereo images should not only meet the application requirements but also consider the influence of viewpoint differences, lighting conditions, camera performance, and scene characteristics, etc. After the acquired images are subjected to feature extraction and target point matching, combined with the internal and external parameters obtained from camera calibration, the coordinates of the target points in the world coordinate system can be calculated to achieve the purpose of measurement.

[0097] Research on the Integration of UAV and High-Precision Measurement Platform

[0098] Since high-precision stereo vision measurement requires multi-camera shooting from multiple angles, the camera needs to read and store the drone's own GPS information, calibrate the photo shooting position, and the camera needs to shoot at different angles. Therefore, how to offset the influence of environmental factors on the camera shooting quality and take measures to reduce or eliminate such errors is the key technology for the integration of the drone and the high-precision measurement platform.

[0099] Adopt a commercial binocular camera. If the test accuracy does not meet the requirements, then consider customizing a binocular camera. Based on the drone's SDK software development kit, expand the application fields of the drone, realize the integrated control of software and hardware, and expand the flight operation functions. Based on the open API interface of the drone, enable the custom payload to communicate with the internal systems such as the drone flight control, GPS, and video transmission, and realize functions such as control and data acquisition.

[0100] Research on the Stability Technology of the High-Precision Measurement Platform

[0101] The purpose of the design of the three-axis stabilized gimbal control system is to ensure that the gimbal can still shoot and search for the target when there is wind resistance disturbance outside the system, can quickly respond to the system, and can maintain a high control accuracy, so that the shooting picture can still be kept stable when the optoelectronic device shoots the target. In order to achieve these purposes, the control system of the three-axis stabilized gimbal consists of three parts: the hardware circuit drive part, the software program design part, and the neural network algorithm control part.

[0102] First, study the overall control system of the three-axis stabilized gimbal, design the overall process of the entire control system and the functions that each module should achieve, then analyze the overall control model block diagram of the gimbal, establish the mathematical model of the three-axis stabilized gimbal, and finally obtain the transfer function of the gimbal control system, analyze the factors affecting the stable control of the gimbal, and give solutions. Study the hardware of the gimbal control system, select the type of motor, select the sensors required for the hardware system, ensure that the gimbal control system can obtain accurate attitude information, and lay the foundation for the subsequent stability control. Then study each main module of the entire hardware system of the three-axis stabilized gimbal to ensure the rationality of the design of each hardware module.

[0103] Study the algorithm part of the gimbal control system, study the learning and training methods of the RBF artificial neural network, combine the three-axis stabilized gimbal control system and the RBF artificial neural network algorithm control structure of the three-axis stabilized gimbal, and finally through simulation experiments, verify that compared with traditional control, the RBF artificial neural network control has more advantages in terms of response and anti-interference.

[0104] Research the software of the pan-tilt control system, including the overall process of the software framework, serial port, communication module, attitude calculation module, and neural network algorithm implementation module. Build the entire three-axis stable pan-tilt control system for debugging and experiments, and analyze the experimental results to ensure that each hardware module can work properly, meet the design requirements, the software can achieve the established functions, and the algorithm part can also achieve a certain anti-interference control.

[0105] The purpose of the design of the three-axis stable pan-tilt control system is to ensure that the pan-tilt can still shoot and search for the target when there is wind resistance disturbance outside the system, can quickly respond to the system, and can maintain a high control accuracy, so that the captured image can still be stable when the optoelectronic device shoots the target. To achieve these goals, the control system of the three-axis stable pan-tilt consists of three parts: the hardware circuit drive part, the software program design part, and the neural network algorithm control part. First, give an overall introduction to the two-axis stable pan-tilt control system, then select the main devices of the entire control system, and finally establish the mathematical model of the pan-tilt motor.

[0106] Research on the high-precision internal parameter solution of the measurement system

[0107] Research the calibration algorithm of the binocular vision measurement system, estimate the system parameters, and then realize the corresponding relationship between the three-dimensional space and the matching image pairs. Only in this way can the binocular vision measurement system reverse the accurate position and attitude information of the target according to the acquired images.

[0108] Direct linear transformation solution: The camera calibration uses the direct linear transformation solution, which is a solution that establishes a direct linear relationship between the image coordinates and the corresponding object point coordinates in the object space. Here, the image coordinates refer to the image coordinates that do not take the principal point of the image as the coordinate origin. This method does not require the initial values of the internal and external orientation elements, so it is particularly suitable for solving the calibration parameters of non-metric cameras. However, this method is a single-model solution and has high requirements for the number and distribution of control points. Otherwise, it is easy to appear ill-conditioned during the solution process.

[0109] Collinearity condition equation solution: The solution based on the collinearity condition equation is the most important and widely used method, which includes the image point coordinate error equation method based on the collinearity condition equation, the multi-image space resection method, the space resection method, and various bundle adjustment methods. The space resection method based on the collinearity equation is a calibration method that uses the image coordinates of the control points (including object coordinates if necessary) as observations to solve the internal and external orientation elements, distortion differences, and other additional parameters of the camera. It can be further divided into the single-image space resection method and the multi-image space resection method.

[0110] In principle, the direct linear transformation solution is evolved from the collinearity condition equation. The commonly used collinearity condition equation is expressed as:

[0111]

[0112] where: x and y are image plane coordinates;

[0113] x0, y0, and f are the interior orientation elements of the photograph;

[0114] X s , Y s , Z s , ψ, ω, and κ are the exterior orientation elements of the photograph;

[0115] X, Y, and Z are object space coordinates;

[0116] a i , b i , c i (i = 1, 2, 3) are the direction cosines formed by the exterior orientation angular elements of the photograph.

[0117] Distortion correction: The geometric distortion of an image refers to the error in the measured geometric position of the image points on the image plane relative to the target points, resulting in the imaging system being unable to strictly satisfy the pinhole imaging model (or central projection relationship) between the image and the actual scene over the entire field of view, causing the central projection rays to bend. Distortion can be divided into two types: radial distortion and tangential distortion. Radial distortion error refers to the distortion where the image point deviates from its ideal position along the radial direction and is only related to the distance of the image point from the principal point of the image. Radial distortion error has symmetry, but its center of symmetry does not exactly coincide with the principal point of the image. For simplicity, the principal point of the image is usually regarded as the center of symmetry. The positive and negative of radial distortion are related to the direction of its deviation. The deviation away from the principal point of the image is positive, and the deviation towards the principal point of the image is negative.

[0118] According to geometric optics theory, the radial distortion error △r of the lens can be expressed by the following odd polynomial:

[0119] △r = k1r 3 + k2r 5 + k3r 7 +... Equation 1

[0120]

[0121] where: r — the radial distance of the image point;

[0122] x, y — the coordinates of the image point;

[0123] x0, y0 — the coordinates of the principal point of the image;

[0124] k i (i = 1, 2, 3...) — the radial distortion parameters;

[0125] Eccentric distortion is the distortion that occurs when each unit of the lens deviates from the lens axis or tilts due to lens manufacturing, installation, and vibration, causing the image point to deviate from its theoretical position both radially and tangentially. Under normal circumstances, the eccentric distortion is much smaller than the radial distortion, and the total error caused by non-radial distortion is approximately 1 / 7 to 1 / 8 of the error caused by radial distortion. Similar to radial distortion, in close-range photogrammetry, the eccentric distortion varies with the focal length D, and there are also variations in eccentric distortion for ground objects (at a distance of D') at different focal lengths D. The correction for eccentric distortion considering the focal length D under close-range conditions is shown by the following formula:

[0126]

[0127] In the formula: Δx D , Δy D —— Components of eccentric distortion when the focal length is D;

[0128] f—— Principal distance when the focal length is D;

[0129] D—— Focal length;

[0130] p1, p2—— Eccentric distortion parameters.

[0131] Spatial resection:

[0132] The work of calculating the model point coordinates (or ground point coordinates) by using the interior orientation elements of two images in a stereo pair, the coordinates of homologous image points, and the relative orientation elements (or exterior orientation elements) of the stereo pair.

[0133] 1. Calculate the three-dimensional coordinates of the model points by using the relative orientation elements of two images in a stereo pair;

[0134] 2. Calculate the ground coordinates of the ground points by using the exterior orientation elements of two images in a stereo pair.

[0135] Spatial resection:

[0136] A method of determining the exterior orientation elements of an image by using ground control points and their image points on the image. This method requires at least three ground control points that are not on a straight line.

[0137] Pyramid method: Solve the exterior orientation elements of the image by using the collinearity condition equation

[0138]

[0139] Research on Optimization of Measuring System Dimension Measurement

[0140] The coupling of the coaxial error of the optical system and the camera model parameters may add position changes to the camera extrinsic parameters, which is contrary to the camera model theory. Secondly, the position transformation will also be transferred to the structural parameters, resulting in structural parameter errors. In the calibration process, factors such as checkerboard error, feature point extraction error, and image noise determine the inevitable errors in the calibration parameter results, so it is very necessary to further optimize the system. Binocular system calibration refers to the calibration of the system's intrinsic parameters, mainly the left and right camera intrinsic parameter matrices, distortion parameters, and system structural parameters. Usually, the position relationship between the camera and the target is unknown. In the calibration process, the camera's extrinsic parameters are used as intermediate quantities and work together with the system's intrinsic parameters to achieve camera fitting. Camera extrinsic parameters are also very important parameters, but they are usually ignored. For example, the camera extrinsic parameters are discarded and a camera coordinate system in the binocular system is directly set as the measurement coordinate system. If there is no position change component in the camera extrinsic parameters, this processing method is not wrong, otherwise it is likely to introduce system errors.

[0141] Based on the above analysis, this scheme uses camera extrinsic parameters to establish the measurement coordinate system. In order to further eliminate the system error introduced by parameter calibration error, a model-based optimization method is proposed to compensate the system parameter calibration error using camera extrinsic parameters. In theory, this optimization process can be directly performed together with calibration, but the following factors determine the necessity of separate optimization:

[0142] (1) During the calibration process, a target coordinate system is established for each target, so it is not convenient to unify the coordinate system.

[0143] (2) The instability of the calibration process causes differences in the results between different groups. The usual way to deal with this is to take the average of multiple calibration results as the final calibration result, which destroys the results of each calibration process.

[0144] coordination.

[0145] (3) Parameter calibration is for the entire measurement space, but the system has local errors, such as distortion, which are difficult to eliminate through calibration alone.

[0146] Research on high-precision image distortion correction technology

[0147] For a given system, the idea of improving the measurement accuracy is to enhance the fitting accuracy of the model parameters to the system. The fitting accuracy is divided into two parts. One is the fitting accuracy of the model to the system itself, and the other is the accuracy of model parameter estimation. The fitting accuracy mainly refers to the fitting accuracy of the single-camera model. For an optical system, the alignment error destroys the optical axis consistency of the optical system and its ideal imaging characteristics. On the one hand, it causes instability in the calibration of the internal parameters of the camera and may converge to a local optimum during the iterative optimization process. On the other hand, it affects the fitting accuracy of the camera model. The camera model also includes a distortion model. The factors causing camera distortion are complex and diverse, so almost all distortions are only approximately fitted. To improve the fitting accuracy of the single-camera model parameters, first, the aberration theory of the optical system is studied. Then, relevant research is carried out in three aspects, including the coupling of the internal and external parameters of the camera model, the non-coaxiality of the optical system, and distortion, and corresponding solutions are proposed or improved.

[0148] Image distortion refers to the errors in spectral characteristics and geometric characteristics between a remote sensing image and the true surface scene it reflects, that is, radiation error and geometric error. The former is manifested as the distortion of the image in terms of gray level, and the latter is manifested as the deformation in terms of geometric relationship. Image distortion is an important indicator for testing the interpretation and mapping performance of remote sensing images. Using uncorrected distorted remote sensing images without understanding the nature and degree of image distortion will not achieve the correct application effect and may even be counterproductive. Therefore, the original data of remote sensing images usually need to be radiometrically corrected and geometrically corrected.

[0149] The pose between two coordinate systems is determined by the representation of a set of points in the two coordinate systems. The main error source of the three-dimensional point cloud pose measurement algorithm based on the binocular measurement system is the reconstruction error. Since the target coordinate system is self-built, the coordinates of the points in the two coordinate systems are both observed values, which further increases the influence of the point cloud reconstruction error on the pose solution accuracy. Here, the reconstruction error refers to the random error caused by image noise, and the influence of reducing the random error on the accuracy of the pose estimation algorithm is reduced.

[0150] The binocular vision system takes pictures at M different orientations, obtains the coordinates of the point cloud in the target coordinate system at each position, and obtains the pose transformation matrix of the binocular vision system relative to the target coordinate system at the last position M according to the mutual constraints of multiple positions.

[0151] The camera captures images by using the principle of pinhole imaging to convert the external scene into an image, and in the imaging process, pinhole imaging is achieved through a lens. The external scene passes through the lens and forms a reduced inverted image in the camera's photosensitive component. In this process, light forms an image through the refraction of the lens. However, the refraction of the lens is related to the crystal structure of the lens. Its physical properties cause light to travel in a straight line at the optical center, and as the distance between the light and the optical axis of the optical center increases, the refraction effect caused by the lens becomes more obvious, which leads to the light farther from the optical axis deviating more from the optical center. This forms lens distortion. Generally speaking, lens distortion is actually the general term for the inherent perspective distortion effect of optical lenses, which makes the imaging system unable to strictly satisfy the pinhole imaging model between the actual scene and the image in the full field of view and causes the central projection rays to bend. Generally speaking, lens distortion is mainly non-linear distortion, including radial distortion and tangential distortion. Radial distortion refers to the length change that occurs along the ray direction at the vector end, that is, the change in the radius vector. Common barrel distortion and pillow distortion are exactly caused by radial distortion. Tangential distortion is caused by the change of the ideal point along the tangent direction, which can also be understood as a change in the angular direction. Ideal lens distortion is caused by the inherent distortion of an ideal lens. In fact, the reasons for lens distortion are not only due to the inherent distortion of the lens. The accuracy and quality of the lens also have a certain impact on lens distortion. This is manifested in that the lenses provided by the lens manufacturer are not ideal smooth lens models, and there are inevitably some differences between two lenses of the same type processed by the same manufacturer due to processing reasons. In addition, during the installation process of the lens, due to various objective mechanical position reasons and artificial installation operation reasons, it is impossible for the camera to perform visual detection in a completely ideal environment, all of which add more random influencing factors to lens distortion. In general, the impact of lens distortion on images is minimal. However, in machine vision detection, especially in high-precision machine vision detection, lens distortion is a major factor affecting accuracy. Therefore, in the currently used visual calibration system, this solution adds distortion correction for the camera lens, greatly reducing the impact of lens distortion on images and thus improving the detection accuracy.

[0152] 4.1.8 Research on High-Precision Measurement Management System

[0153] During measurement, the camera performs rapid acquisition and conducts steps such as feature extraction and recognition, stereo matching, 3D reconstruction, and motion analysis on the data stream in the buffer. Parallel computing is planned to be used to accelerate the entire algorithm process and ensure the measurement efficiency. After the measurement is completed, the deformation field, velocity field, and acceleration field of the measured surface within different time periods are visually displayed on the software interface, and an analysis report is generated for archiving. Therefore, the software design mainly includes top-level design and bottom-level design. The top-level design includes UI interface design, software management, and data management, while the bottom-level design includes system calibration, data acquisition, feature extraction and recognition, stereo matching, 3D reconstruction, and deformation analysis. It is necessary to design the overall framework of the software and reasonably divide each functional module. Formulate software code specifications and design test methods to ensure that the entire software project is easy to modify and maintain.

[0154] 4.2 Innovation Points of the Project

[0155] (1) Develop an airborne binocular camera measurement platform and image measurement system for unmanned aerial vehicles (UAVs) to achieve high-precision three-dimensional spatial measurement of power transmission lines based on digital optics, with a measurement error of less than 1 cm.

[0156] (2) Develop a three-dimensional flight path planning software to realize the automated operation of the airborne binocular camera measurement platform for UAVs.

[0157] (3) Improve the operation efficiency of measuring key positions of power grid equipment, improve measurement accuracy, reduce operation risks such as induced electricity, save operation costs, and achieve intelligent and standardized operation.

[0158] Functions Achieved by the Project Results

[0159] (1) Functions of the Intelligent Flight Path Planning and Design System

[0160] Design the flight path according to requirements to achieve the automatic flight operation of the airborne binocular camera measurement platform for UAVs.

[0161] (2) Functions of the Binocular Camera Measurement Platform

[0162] ① The captured image pictures are transmitted in real time, supporting real-time data backhaul via 4G / 5G wireless networks and real-time data analysis;

[0163] ② Support the horizontal rotation of the binocular camera from 0° to 360° and the vertical rotation from 0° to 90°;

[0164] ③ During operation, the inspection screen is transmitted by an intelligent camera, facilitating point selection and shooting.

[0165] (3) The image measurement system is used to analyze the image pictures taken by the "binocular camera measurement platform" and has the following functions:

[0166] ①Support point-to-point, point-to-line distance measurement, line-line angle measurement, line-plane angle measurement, and three-point area measurement;

[0167] ②Support image distortion correction;

[0168] ③Support one-key output of measurement reports;

[0169] ④Support free import of camera parameters;

[0170] ⑤Support automatic image classification;

[0171] ⑥Support selection of homologous points;

[0172] ⑦Support the corresponding matching of binocular photos and tower coordinates, and can display the line and tower number at the corresponding position of the photo.

[0173] Product technical parameters

[0174]

[0175]

[0176] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A binocular camera measurement platform and an image measurement system, characterized in that, Including: A visual measurement system and a three-axis stabilized gimbal control system for stabilizing the visual measurement system, a visual measurement system and an image measurement system. The visual measurement system is used for the target reflected or emitted light beam to reach the visual sensor through the optical system. The electronic components perform real-time high-speed acquisition on the collected information, preprocess the collected grayscale image or color image data, transmit a part to the display device, and transmit another part to the image processing system to complete the effective processing of the image, and complete the target extraction and pose measurement tasks.

2. A binocular camera measurement platform and an image measurement system according to claim 1, characterized in that, The binocular visual measurement system specifically includes two cameras to simultaneously obtain digital images of the target within the overlapping field of view, and processes the images through relevant theories such as projective transformation to reconstruct the target position and pose.

3. A binocular camera measurement platform and an image measurement system according to claim 1, characterized in that, The visual measurement system includes a camera mounted on a drone. Based on the drone's SDK software development kit and the open API interface of the drone, the payload communicates with the drone flight control, GPS, and video transmission internal systems to achieve control and data acquisition functions.

4. A binocular camera measurement platform and an image measurement system according to claim 2, characterized in that, The three-axis stabilized gimbal control system includes the mathematical model of the gimbal motor. The mathematical model of the gimbal motor infers the accurate position and pose information of the target based on the images obtained by the camera.

5. A binocular camera measurement platform and an image measurement system according to claim 4, characterized in that, The mathematical model of the gimbal motor infers the accurate position and pose information of the target based on the images obtained by the camera, specifically using the collinearity condition equation solution method. The collinearity condition equation solution method includes the image point coordinate error equation method based on the collinearity condition equation, the multi-image space resection method, and the space resection method. The image coordinates of the control points in the space resection method based on the collinearity equation are used as observations to solve for the internal and external orientation elements and distortion of the camera.

6. A binocular camera measurement platform and an image measurement system according to claim 5, characterized in that, The equation of the collinearity condition equation solution method is expressed as: In the formula: x, y are the image plane coordinates; x0, y0, f are the internal orientation elements of the image; X s , Y s , Z s , ψ, ω, κ are the exterior orientation elements of the photograph; X, Y, Z are the object space coordinates; a i ,b i ,c i (i = 1, 2, 3) are the direction cosines formed by the exterior orientation angular elements of the photograph.

7. The binocular camera measurement platform and image measurement system according to claim 5, characterized in that, Distortion correction: The radial lens distortion △r can be expressed by the following odd polynomial: △r = k1r 3 + k2r 5 + k3r 7 + L Equation 1 In the formula: r—the radial distance of the image point; x, y—the image point coordinates; x0, y0—the coordinates of the principal point of the image; k i (i = 1, 2, 3L) - Radial distortion parameter.

8. The binocular camera measurement platform and the image measurement system according to claim 7, characterized in that, In close-range photogrammetry, the decentering distortion changes with the change of the focal length D, and there are also changes in the decentering distortion of the ground objects at different focal lengths D. Considering the decentering distortion correction at the close-range condition with the focal length D is shown in the following formula: Where: Δx D , Δy D —— Eccentric aberration difference components when the focusing distance is D; f—the principal distance when the focal length is D; D—the focal length; p1, p2—the decentering distortion parameters.

9. A binocular camera measurement platform and an image measurement system according to claim 5, characterized in that, Space resection: The work of solving the model point coordinates or ground point coordinates by using the internal orientation elements of two images of a stereo pair, the homologous image point coordinates, and the relative orientation elements or external orientation elements of the image pair.

10. A binocular camera measurement platform and an image measurement system according to claim 5, characterized in that the space Resection: A method for determining the external orientation elements of an image by using the ground control points and their image points on the image. This method requires at least three ground control points not on a straight line. Pyramid method: Using the collinearity condition equation to solve the external orientation elements of the image

Images