A visual servoing based aerial camera pose control system

Abstract

The application relates to the field of aircraft visual servo control and machine vision technology, in particular to a flight camera attitude control system based on visual servo, which comprises a flight platform, a monocular camera, a left light source group, a right light source group, a synchronous controller, an inertial sensor, an image processing unit and a flight controller; the synchronous controller controls only one side of the light source group to be lighted during each frame exposure, and alternately acquires left and right illumination images; the image processing unit carries out compensation processing, feature point classification and brightness balance on the images, and outputs a brightness adjusting parameter according to the comparison between the average brightness and the reference brightness; the flight controller calculates a data reliability value according to the number of texture feature points, the number of highlight feature points and the proportion, and adjusts an attitude control proportional gain matrix and a damping gain matrix according to the data reliability value, and outputs an attitude control instruction.

Description

Technical Field

[0001] This invention relates to the field of aircraft visual servo control and machine vision technology, specifically to a flight camera attitude control system based on visual servoing. Background Technology

[0002] In the case of inspecting the inner walls of enclosed spaces such as large industrial metal storage tanks and pipe rack containers, the working environment usually lacks natural light, and the inspected surfaces are mostly curved metal surfaces, which are prone to obvious specular reflection. In order to ensure the stability and safety of the flight platform when working close to the inner wall, it is usually necessary to use airborne vision sensors to estimate the position and pose of the camera relative to the inspected surface in real time, and to complete attitude adjustment and hovering control accordingly. However, in the attitude control of flight cameras based on visual servoing, adjacent images are not only affected by the high-frequency displacement disturbances and rotational motion of the flight platform itself, but also by the interference of specular reflection jumps under alternating illumination conditions. Existing methods, when dealing with highly reflective and weakly textured metallic surfaces, usually directly remove the specular areas as noise, or rely solely on ordinary texture features for pose calculation. This approach can easily lead to an insufficient number of available feature points in areas with sparse texture, resulting in unstable distance judgment, surface normal perception, and attitude control inputs. At the same time, if the control gain remains fixed for a long time, it is difficult to balance the fast tracking requirements of texture-rich areas with the jitter suppression requirements of specular-dominant areas, thus affecting the control accuracy and operational reliability of the flight platform during near-wall inspection. Summary of the Invention

[0003] The purpose of this invention is to provide a visual servo-based attitude control system for a flight camera, addressing the following technical problems: Existing visual servo-based attitude control technologies suffer from insufficient effective feature point extraction, unstable pose perception, and difficulty in simultaneously achieving fast tracking and jitter suppression with fixed control gains when facing highly reflective metallic surfaces with weak textures. Therefore, there is an urgent need for a control system that can effectively decouple highlight and texture features and dynamically adjust the attitude control gain accordingly. This invention's objective can be achieved through the following technical solutions: A visual servo-based flight camera attitude control system includes a flight platform, a monocular camera mounted on the front center of the flight platform for acquiring images of the inspected surface, a left light source group and a right light source group located on the left and right sides of the front of the flight platform, a synchronization controller, an inertial sensor, an image processing unit, and a flight controller; the intrinsic parameters of the monocular camera and the relative pose of the monocular camera with the left and right light source groups are pre-calibrated and known. The synchronization controller controls the lighting of only one side of the light source group to emit light pulses within the single-frame exposure time of the monocular camera, so as to acquire adjacent left and right illumination images; the image processing unit receives the left illumination image, right illumination image and angular velocity information, performs compensation processing on adjacent images and classifies feature points, determines the visual pose data of the monocular camera relative to the inspected surface based on texture feature points, and determines the distance estimate and normal estimate based on specular feature points and in combination with the pre-established distance mapping table and normal mapping table; The flight controller calculates a data reliability value based on the number of texture feature points and the number of highlight feature points. The data reliability value is the ratio of the number of texture feature points to the total number of valid feature points, where the total number of valid feature points is the sum of the number of texture feature points and the number of highlight feature points. The flight controller has a pre-calibrated fixed value first confidence threshold and second confidence threshold, and the first confidence threshold is greater than the second confidence threshold. The flight controller adjusts the attitude control proportional gain matrix and damping gain matrix, both of which are 3×3, based on the comparison result between the data confidence value and the first confidence threshold and the second confidence threshold, and outputs attitude control commands.

[0004] Preferably, the monocular camera is a global shutter camera; the synchronization controller controls the left light source group and the right light source group to light up alternately according to the frame number, wherein odd-numbered frames correspond to the left illumination image and even-numbered frames correspond to the right illumination image; the emission pulses of the left light source group and the right light source group are located within the single-frame exposure time of the monocular camera, and only one side of the light source group emits light during each exposure.

[0005] Preferably, the image processing unit is further configured to perform brightness balancing on the left light source group and the right light source group, wherein the brightness balancing includes: calculating the average brightness of the left illumination image and the right illumination image within a preset detection area after removing dark and saturated areas at the image edges; and comparing the average brightness with a preset reference brightness. The image processing unit outputs brightness adjustment parameters to the synchronization controller based on the comparison result. The synchronization controller then adjusts the driving current of the left light source group and the right light source group based on the brightness adjustment parameters, and limits the driving current between a preset upper current limit and a preset lower current limit.

[0006] Preferably, the image processing unit's compensation process for adjacent images includes: receiving angular velocity information output by the inertial sensor, and obtaining a three-axis angular increment vector based on the angular velocity information between the exposure center timestamps of adjacent left and right illumination images; calculating the pixel offset caused by the rotation of the flight platform according to the three-axis angular increment vector, the pixel coordinates of the feature points in the left illumination image, and the camera intrinsic parameter matrix; and subtracting the pixel offset from the feature point pixel movement between the right and left illumination images to obtain the feature point displacement residual used for subsequent classification.

[0007] Preferably, the image processing unit's classification process for feature points includes: extracting feature points with gray-level gradient magnitudes greater than a local gradient threshold from the left and right illumination images, and performing neighborhood tracking based on an image matching algorithm; determining the direction offset ratio based on the component of the feature point displacement residual in the projection direction of the line connecting the emission centers of the left and right light source groups and the component in its vertical direction on the image plane, and based on the component of the feature point displacement residual in the projection direction and the component in the vertical direction; and determining the proportion of pixels with gray-level values ​​higher than a gray-level threshold within a preset window centered on the feature point. Feature points whose displacement residual magnitude is not greater than the first threshold are marked as texture feature points. Feature points whose displacement residual magnitude is not less than the second threshold, whose direction offset ratio is not less than the direction threshold, and whose bright pixel ratio is not less than the bright pixel ratio threshold are marked as specular feature points. The remaining feature points are marked as invalid feature points. Among them, feature points whose displacement residual is between the first threshold and the second threshold are classified as invalid feature points.

[0008] Preferably, the first threshold and the second threshold are dynamically adjusted thresholds; wherein, the first threshold is adaptively determined based on the variance of the feature point displacement residual and the angular increment modulus between adjacent images; the second threshold is greater than the first threshold and not lower than a preset minimum specular offset threshold, wherein the minimum specular offset threshold is the displacement residual threshold along the projection direction of the line connecting the emission centers on the image plane.

[0009] Preferably, the visual pose data of the monocular camera relative to the inspected surface includes lateral displacement, longitudinal displacement, distance direction displacement, and yaw angle change, and serves as the attitude control input data of the flight controller.

[0010] Preferably, the image processing unit is further configured to calculate distance estimates and normal estimates based on the highlight feature points. The calculation process includes: determining the corresponding weights based on the image contrast of each highlight feature point; calculating the average offset of the highlight feature points in the horizontal and vertical directions of the image based on the weights, and calculating the offset dispersion; using the average offset and the offset dispersion as query parameters, querying a preset distance mapping table and normal mapping table to obtain the distance estimates and the normal vector of the inspected surface in the camera coordinate system, wherein the distance mapping table and normal mapping table are pre-established mapping tables based on the relative pose calibration results of the monocular camera and the left and right light source groups.

[0011] Preferably, the flight controller is configured to: increase the proportional gain matrix of attitude control when the data confidence value is greater than or equal to the first confidence threshold; decrease the proportional gain matrix and increase the damping gain matrix when the data confidence value is less than or equal to the second confidence threshold; maintain the preset proportional gain matrix and damping gain matrix when the data confidence value is between the first confidence threshold and the second confidence threshold; receive the desired attitude and preset target normal preset according to the inspection task, and output attitude control commands based on the attitude error between the current attitude of the flight platform and the desired attitude, and the angle error between the normal estimate and the preset target normal.

[0012] Preferably, the distance between the light emission centers of the left light source group and the right light source group is 40 mm to 120 mm; the acquisition frame rate of the monocular camera is 50 frames per second to 120 frames per second; the single frame exposure time is 0.5 milliseconds to 3 milliseconds; the light emission pulse width of the left light source group and the right light source group is 0.6 to 0.8 times the single frame exposure time; and the synchronization error between the exposure timing of the monocular camera and the trigger timing of the left and right light source groups is no greater than 5 microseconds.

[0013] Compared with the prior art, the present invention has the following beneficial effects: 1. This system effectively classifies feature points into texture feature points and specular feature points by alternately lighting the left and right light sources and performing compensation processing on adjacent images. The system uses specular feature points in combination with a preset mapping table to obtain distance and normal estimation values, transforming traditional specular reflection interference into effective physical measurement data, thus solving the problem of unstable perception input caused by insufficient available feature points on weakly textured metal surfaces. 2. This system calculates the data confidence value based on the ratio of the number of texture and specular feature points to the total number of effective feature points, and dynamically adjusts the proportional gain matrix and damping gain matrix of attitude control based on the comparison result with the preset threshold. The above control configuration enables the attitude control strategy to adaptively adjust according to the actual texture conditions of the surface, solving the problem that fixed gain is difficult to balance fast tracking in texture-rich areas and jitter suppression in specular-dominant areas. Attached Figure Description

[0014] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the functional modules of a flight camera attitude control system based on visual servoing according to this application. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0016] like Figure 1 As shown, a flight camera attitude control system based on visual servoing includes a flight platform, a monocular camera installed in the front center of the flight platform for acquiring images of the inspected surface, a left light source group and a right light source group set on the left and right sides of the front of the flight platform, a synchronization controller, an inertial sensor, an image processing unit and a flight controller. The intrinsic parameters of the monocular camera and the relative pose of the monocular camera with the left and right light source groups are known in advance; The synchronization controller controls the illumination of only one side of the light source group to emit light pulses during the single-frame exposure time of the monocular camera, so as to obtain adjacent left and right illumination images; The image processing unit receives the left illumination image, the right illumination image, and angular velocity information, performs compensation processing on adjacent images and classifies feature points, determines the visual pose data of the monocular camera relative to the inspected surface based on texture feature points, and determines the distance estimate and normal estimate based on the specular feature points and in combination with the pre-established distance mapping table and normal mapping table. The flight controller calculates the data confidence value based on the number of texture feature points and the number of highlight feature points. The data confidence value is the ratio of the number of texture feature points to the total number of valid feature points, where the total number of valid feature points is the sum of the number of texture feature points and the number of highlight feature points. The flight controller has a pre-calibrated fixed value, a first confidence threshold and a second confidence threshold, and the first confidence threshold is greater than the second confidence threshold. Based on the comparison results between the data confidence value and the first confidence threshold and the second confidence threshold, the flight controller adjusts the attitude control proportional gain matrix and damping gain matrix, both of which are 3×3, and outputs attitude control commands.

[0017] A monocular camera is a global shutter camera; The synchronous controller controls the left and right light source groups to light up alternately according to the frame number, where odd-numbered frames correspond to the left illumination image and even-numbered frames correspond to the right illumination image. The emission pulses from the left and right light source groups are within the single-frame exposure time of the monocular camera, and only one light source group emits light during each exposure.

[0018] The image processing unit is also used to perform brightness balancing on the left and right light source groups, including: Remove dark areas at the image edges and overexposed / saturated areas directly exposed to highlights from the left and right illumination images. Use the remaining effective pixel areas as preset detection areas and calculate the average brightness within each preset detection area. Compare the average brightness with the preset reference brightness; The image processing unit outputs brightness adjustment parameters to the synchronization controller based on the comparison results. The synchronization controller then adjusts the drive current of the left and right light source groups based on the brightness adjustment parameters, and limits the drive current between a preset upper current limit and a preset lower current limit.

[0019] The flight camera attitude control system provides raw physical observation data for subsequent image feature classification through asymmetric spatiotemporal alternating illumination. In practical applications, the system is used for close-range inspection of the interior of large industrial metal storage tanks. There is no natural light inside the tanks, and the inner walls are mostly curved metal surfaces with significant specular reflections. The flight platform is equipped with a monocular camera that hovers or moves within a preset distance from the inner wall. During system operation, the synchronization controller provides the clock reference for the entire acquisition link; the synchronization controller sends a trigger signal to the monocular camera to control the opening and closing of its global shutter, and at the same time, according to the frame number of the current image, it alternately sends lighting pulses to the left light source group and the right light source group; When the frame number is odd, the synchronization controller only activates the drive circuit of the left light source group, ensuring that the emission pulse falls strictly within the exposure time window of that frame, thereby acquiring the left illumination image; when the frame number is even, only the right light source group is activated to acquire the right illumination image; this alternating timing is written into the hardware register to ensure that in adjacent frames, only the illumination position changes laterally within the adjacent frame period, while the camera's physical viewing angle does not deflect by more than a preset angle threshold; the acquired left and right illumination images are stored in the memory buffer of the image processing unit for subsequent differential processing; In a highly reflective environment on a metal inner wall, due to the temperature rise difference or installation tolerance of the left and right light sources, there may be an overall brightness shift between the left and right illumination images, which will affect the pixel matching and displacement judgment of subsequent feature points. After the image processing unit extracts the image from the buffer, it removes dark areas at the edges and saturated areas directly illuminated by highlights. It then counts the pixel grayscale values ​​within a preset detection area and calculates the average brightness. This average brightness is then compared with a preset reference brightness. Based on the comparison result, the image processing unit calculates the brightness adjustment parameters required for compensation and sends them to the synchronization controller. During the calculation process, the image processing unit uses discrete proportional-integral control logic, taking the current deviation of the average brightness from the preset reference brightness and its historical cumulative deviation as input, to calculate the required driving current adjustment increment as the brightness adjustment parameter. The specific control logic is as follows: the adjustment increment consists of two parts: a proportional term and an integral term. The proportional term is the product of the current deviation and the preset proportional coefficient, which is used to quickly respond to sudden changes in local reflectivity. The integral term is the product of the historical cumulative deviation and the preset integral coefficient, which is used to eliminate long-term steady-state deviations caused by differences in the temperature rise of the light source. The preset proportional coefficient and the preset integral coefficient are empirical values ​​that are pre-tuned and solidified in the system memory based on the step response test results of the light source brightness control closed loop. The required drive current adjustment increment can be obtained by adding the calculated proportional term and integral term; the synchronous controller updates the drive current of the left light source group and the right light source group respectively, and performs a limiting operation to clamp the current between the preset upper current limit and the preset lower current limit. The preset upper current limit is set as the rated maximum safe operating current of the light source, and the preset lower current limit is set as the minimum conduction sustaining current that enables the light source to emit light stably. If, during a certain sampling period, the local specular reflectivity of the inner wall of the metal exceeds the preset reflectivity threshold, resulting in an overexposed pixel area exceeding the preset area threshold, and thus an insufficient number of effective pixels within the preset detection area, the image processing unit stops updating the brightness adjustment parameters for the current period, simultaneously freezes the integral term accumulation of the discrete proportional-integral controller, and instructs the synchronous controller to maintain the drive current value of the previous period, thereby avoiding sudden changes in the light source drive current due to environmental reflection. In this embodiment, the compensation process for adjacent images by the image processing unit includes: It receives angular velocity information output by the inertial sensor and obtains a three-axis angular increment vector based on the angular velocity information between the exposure center timestamps of adjacent left and right illumination images; The pixel offset caused by the rotation of the flight platform is determined based on the three-axis angular increment vector and the intrinsic parameters of the monocular camera; the feature point displacement residuals used for subsequent classification are determined based on the feature point pixel movement and pixel offset between the right illumination image and the left illumination image.

[0020] The image processing unit's classification process for feature points includes: Feature points with gray-level gradient magnitudes greater than the local gradient threshold are extracted from the left and right illumination images. A pixel template of a preset size is constructed with the feature points as the center. An image matching algorithm is used to perform neighborhood tracking and obtain pixel coordinates. Based on the component of the feature point displacement residual in the projection direction of the line connecting the emission centers of the left and right light source groups on the image plane and its vertical component, the direction offset ratio is determined; the proportion of pixels with gray values ​​higher than the gray value threshold within the preset window centered on the feature point is determined. Feature points whose displacement residual modulus is not greater than the first threshold are marked as texture feature points. Feature points whose displacement residual modulus is not less than the second threshold, whose orientation offset ratio is not less than the orientation threshold, and whose bright pixel ratio is not less than the bright pixel ratio threshold are marked as highlight feature points. The remaining feature points are marked as invalid feature points. Feature points whose displacement residual is between the first threshold and the second threshold are classified as invalid feature points. Among them, the local gradient threshold and grayscale threshold are constants preset based on the prior conditions of ambient lighting and the background noise of the image sensor; the orientation threshold is preset based on the extreme angle offset deviation caused by the system hardware installation tolerance; and the high-brightness pixel ratio threshold is preset based on the ratio of the typical spot area of ​​the high-brightness reflection on the image to the preset window area.

[0021] The first and second thresholds are dynamically adjusted thresholds; The first threshold is adaptively determined based on the variance of the feature point displacement residual and the angular increment modulus between adjacent images. The second threshold is greater than the first threshold and not less than the preset minimum highlight offset threshold, which is the displacement residual threshold along the projection direction of the line connecting the emission centers on the image plane.

[0022] The visual pose data of the monocular camera relative to the inspected surface includes lateral displacement, longitudinal displacement, distance direction displacement, and yaw angle change, which are used as attitude control input data for the flight controller.

[0023] The image processing unit separates real surface texture from specular highlight reflection by eliminating the image background movement caused by the flight platform's own motion and by utilizing the displacement characteristics caused by lighting switching. In the case of inspecting the interior of large industrial metal storage tanks, due to the high-frequency slight attitude jitter of the flight platform itself, there is not only a difference in the position of the illumination between two adjacent alternating illumination images, but also the deflection of the camera's viewpoint. If the movement of feature points in the image is directly calculated, the displacement caused by the rotation of the flight platform will be regarded as the highlight displacement caused by the switching of the light source. To this end, the image processing unit receives the angular velocity information output by the inertial sensor in real time through the data bus; for the extracted adjacent left illumination image and right illumination image, the image processing unit extracts the timestamp of the midpoint between the two exposures, integrates the angular velocity within the time interval, and generates a three-axis angular increment vector with a dimension of 3×1. To align in the matrix dimension, the image processing unit uses the Rodriguez formula to transform the 3×1 triaxial angular increment vector into a 3×3 relative rotation matrix; at the same time, it extracts the initial two-dimensional pixel coordinates of the feature points in the left illumination image and augments them into a 3×1 homogeneous coordinate vector. The image processing unit takes the homogeneous coordinate vector as input, multiplies it sequentially by the inverse of the camera intrinsic matrix, the aforementioned 3×3 relative rotation matrix, and then multiplies it by the camera intrinsic matrix again to complete the pure rotation reprojection calculation of the image point under the assumption of the infinite plane; by normalizing the new homogeneous pixel coordinates obtained after reprojection by homogeneous division and subtracting them from the initial two-dimensional pixel coordinates, the pixel offset caused by the rotation of the flight platform is calculated. The image processing unit subtracts the pixel offset from the total pixel movement obtained by the image matching algorithm to remove the motion factor of the flight platform and obtain the feature point displacement residual. If the inertial sensor data is lost during the integration interval, the system will automatically call the historical average of the angular velocity of the previous three effective cycles as the alternative input for the current integration window to ensure the continuity of the residual calculation link. After obtaining the displacement residual, the system enters the feature classification stage. Since there may be real welds, scratches, and reflective points generated by the left and right light source groups on the inner wall of the tank, the image processing unit filters out smooth area pixels with grayscale gradient amplitudes lower than the preset amplitude lower limit. Based on the projection direction of the pre-calibrated line connecting the light emission centers of the left and right light sources on the image plane, the displacement residual of each feature point is decomposed into a component along the projection direction and a component in the vertical direction, and the ratio of their absolute values ​​is calculated to obtain the direction offset ratio. At the same time, the proportion of pixels with grayscale values ​​exceeding the grayscale threshold within the preset window around the feature point is statistically analyzed to obtain the proportion of bright pixels. If a fixed distance discrimination threshold is used, the tracking noise caused by the vibration of the flight platform under different rotation speeds or different airflow disturbances will interfere with the classification accuracy. Therefore, the image processing unit extracts the variance of the displacement residuals of all feature points in the current frame and, combined with the previously calculated angular increment modulus, dynamically calculates the first threshold. The system sets a basic static threshold to characterize background pixel noise under ideal static conditions. The image processing unit adds the product of the displacement residual variance and the first weighting coefficient to the basic static threshold, and then combines it with the product of the angular increment modulus and the second weighting coefficient to obtain the dynamic first threshold. The first and second weighting coefficients are obtained by statistically fitting the image background displacement data under typical disturbance conditions of the flight platform. To ensure the dimensional homogeneity of the dynamic threshold equation, the first weighting coefficient has [pixel...] The second weighting coefficient has the dimension of [pixels / radians]; Furthermore, the preset minimum specular offset threshold is a theoretical minimum specular movement pixel equivalent calculated based on optical triangulation relationships, using hardware parameters such as the camera's minimum working distance, the distance between the centers of the left and right light sources, and the physical size of the pixels. For example, if the basic static threshold is 0.5 pixels, when airflow disturbances cause an increase in the variance of the displacement residual and a larger increase in the jitter angle of the flight platform, the first threshold will automatically increase to 1.2 pixels, thereby avoiding misjudging invalid feature points with large displacement residuals caused by severe vibrations of the flight platform as specular feature points. The system sets a second threshold that is greater than the first threshold, while ensuring that it is not lower than the preset minimum specular offset threshold. Based on the above indicators, the image processing unit executes classification logic: if the displacement residual magnitude of a feature point is not greater than the first threshold, it is determined that it meets the preset stability condition and is written into the texture feature point set; if the displacement residual magnitude of a feature point is not less than the second threshold, and the displacement direction is concentrated in the direction of the line connecting the left and right light sources, the direction offset ratio is not less than the direction threshold, and the local brightness is extremely high, and the proportion of bright pixels is not less than the proportion of bright pixels threshold, then it is classified into the highlight feature point set; other feature points that do not meet the clear directionality are discarded. After classification, the extracted texture feature point set is sent to the visual odometry module to calculate the lateral displacement, longitudinal displacement, distance direction displacement and yaw angle change of the camera relative to the inner wall of the tank. The specific calculation process is as follows: the image processing unit uses epipolar geometry constraints to calculate the fundamental matrix based on the matching pixel coordinates of texture feature points in two adjacent frames, and further recovers the essential matrix by combining it with the pre-calibrated camera intrinsic parameter matrix. The system performs singular value decomposition on the essential matrix to extract the initial values ​​of the camera's relative rotation matrix and translation vector. Further, it constructs a local 3D point cloud using pure texture feature points accumulated over historical cycles, and matches the currently extracted texture feature points with this 3D point cloud. The system then initializes and corrects the local 3D point cloud's absolute scale using distance estimates obtained from specular feature point inversion. A perspective multi-point localization algorithm is employed to perform iterative minimization of reprojection error, thereby obtaining visual pose data containing the true physical scaling. From this data, the lateral displacement, longitudinal displacement, distance direction displacement, and yaw angle change relative to the tank's inner wall are decoupled and extracted. This visual pose data is then pushed to the flight controller as the basic state feedback for maintaining the flight platform's hovering position. In this embodiment, the image processing unit is further configured to calculate distance estimates and normal estimates based on the specular feature points. The calculation process includes: Offset parameters are extracted based on the image features of each highlight feature point; based on the offset parameters, distance estimates and normal estimates are determined through a preset distance mapping table and normal mapping table.

[0024] The flight controller is used for: When the data confidence value is greater than or equal to the first confidence threshold, increase the proportional gain matrix of the attitude control; when the data confidence value is less than or equal to the second confidence threshold, decrease the proportional gain matrix and increase the damping gain matrix. When the data confidence value is greater than the second confidence threshold and less than the first confidence threshold, the preset proportional gain matrix and damping gain matrix are maintained. It receives the desired attitude and preset target normal based on the inspection mission, and outputs attitude control commands based on the attitude error between the current attitude of the flight platform and the desired attitude, as well as the angle error between the normal estimate and the preset target normal.

[0025] The distance between the light-emitting centers of the left and right light-emitting groups is 40 mm to 120 mm. The frame rate of a monocular camera is 50 to 120 frames per second; The single-frame exposure time is 0.5 milliseconds to 3 milliseconds; the pulse width of the light emission from the left and right light source groups is 0.6 to 0.8 times the single-frame exposure time. The synchronization error between the exposure timing of the monocular camera and the trigger timing of the left and right light source groups is no greater than 5 microseconds.

[0026] The system uses the separated highlight feature points to perform surface geometry inversion and dynamically adjusts the attitude control strategy of the flight platform accordingly. In the case of inspecting the inside of large industrial metal storage tanks, when in the area of ​​the tank wall lacking texture, the jump amplitude and dispersion of the highlight point when switching between left and right light sources have a stable physical mapping relationship with the distance between the camera and the metal surface and the local normal of the surface. The image processing unit reads the set of highlight feature points generated in the previous step, extracts the image contrast of the neighborhood of each highlight feature point and normalizes it as a weight. The specific process of normalization is as follows: obtain the maximum and minimum values ​​of all contrast data in the current feature point set, subtract the minimum value from the contrast of each highlight feature point, and divide by the difference between the maximum and minimum values, thereby mapping the weight to the interval between 0 and 1. If the maximum value is equal to the minimum value, then set equal weights for all points. Based on this weight, the image processing unit performs a weighted summation of the displacement residuals of all highlight feature points in the set in the horizontal and vertical directions of the image to obtain the average offset, and calculates the variance of each point relative to the average offset to generate the offset dispersion. The image processing unit uses the average offset and the offset dispersion as indexes to retrieve the pre-calibrated distance mapping table and normal mapping table from the system memory for interpolation lookup. The distance estimate and the normal vector of the inspected surface in the camera coordinate system output by the lookup are transmitted to the flight controller. If no highlight feature points that meet the conditions are extracted in a certain local area, the image processing unit will retain the distance and normal estimates from the previous cycle and add a history expiration mark or a failure mark to prevent the flight controller from causing command jumps due to input discontinuity. After receiving visual pose data and specular inversion data, the flight controller begins to construct control commands. The flight controller counts the number of texture feature points and specular feature points in the current period, calculates the sum of the two as the total number of valid feature points, calculates the proportion of texture feature points in the total number, and defines it as the data reliability value. In the attitude calculation link, the flight controller continuously compares the data confidence value with the preset first confidence threshold and second confidence threshold. When the flight platform is in a textured weld seam area, if the data confidence value is greater than or equal to the first confidence threshold, the flight controller increases the proportional gain matrix of attitude control to ensure rapid tracking of the desired attitude. When the flight platform moves to a textureless surface with a surface roughness lower than a preset roughness threshold, the number of highlight feature points exceeds the preset upper limit, and the data confidence value is less than or equal to the second confidence threshold. If a high proportional gain matrix is ​​maintained at this time, the system is prone to misinterpreting the displacement of the highlight points as the actual displacement of the flight platform and making incorrect compensations. Therefore, the flight controller actively reduces the proportional gain while increasing the damping gain to suppress high-frequency attitude jitter. If the data confidence value is between the two thresholds, the current gain parameters are kept unchanged. The flight controller reads the desired attitude and preset target normal from the inspection mission, calculates the three-dimensional attitude error vector containing angular error components of three orthogonal axes (roll, pitch, and yaw) between the current attitude and the desired attitude, and extracts the normal estimate obtained from the previous query, calculates the normal deviation vector between it and the preset target normal, and projects the normal deviation vector onto the roll and pitch axes of the body coordinate system. When merging attitude error and normal deviation, the system avoids direct scalar superposition, but instead uses a dynamic weight allocation mechanism based on data reliability values ​​to independently allocate weights for each axis in three-dimensional space. When the data confidence value is lower than the second confidence threshold, i.e. the surface is extremely smooth and dominated by highlights, the flight controller will increase the weight of the normal deviation vector on the corresponding axis and decrease the weight of the attitude error vector on the corresponding axis, so that the flight camera attitude control system will prioritize normal alignment based on the normal estimate to ensure safe obstacle avoidance; conversely, when the data confidence value is higher than the first confidence threshold, the weight of the attitude error vector on the corresponding axis will be increased to prioritize the hovering accuracy of the desired path. To quantify the aforementioned qualitative rules, the system stipulates that the sum of the attitude error weight and the normal deviation weight on the same control axis is always 1. When the data confidence value is lower than the second confidence threshold, the normal deviation weight and attitude error weight are preset to, for example, 0.8 and 0.2, respectively. When it is higher than the first confidence threshold, the weight ratios of the two are swapped. When it is between the two, linear interpolation is performed based on the offset ratio of the confidence value relative to the two thresholds to smoothly obtain the real-time weight. The flight controller weights and merges the two on each axis of the body coordinate system according to the dynamically allocated weights to obtain a three-dimensional comprehensive error vector containing three orthogonal axis components. Based on the three-dimensional integrated error vector, the flight controller, combined with the dynamically adjusted proportional gain matrix and damping gain matrix (both 3×3 diagonal matrices), generates attitude control commands containing three-axis torques. The specific attitude control execution steps include: the three-dimensional desired output correction torque vector of the attitude control command is composed of two superimposed parts. The first part is the product of the adjusted 3×3 proportional gain matrix and the three-dimensional integrated error vector at the current moment, providing the three-axis basic restoring torque that causes the flight platform to tend toward the desired attitude of the mission. The second part is the product of the current 3×3 damping gain matrix and the current frame rate of change of the three-dimensional integrated error vector, which provides a three-axis reverse damping torque to suppress motion overshoot and attitude oscillation. The flight controller combines these two three-dimensional torque vectors into the desired three-axis torque vector. And multiply by the dimension corresponding to the rotor configuration of the flight platform. Preset control allocation matrix ,in, The number of rotors of the flight platform, through Solve for the contents Thrust control vectors for each rotor power actuator in each thrust dimension This signal is converted into a pulse width modulation drive signal and mapped to each rotor power actuator of the flight platform, thereby guiding the flight platform to hover stably close to the inner wall of the storage tank at a safe incident angle. in, , , These represent the torque components of the desired three-axis torque vector along the three orthogonal axes of roll, pitch, and yaw in the body coordinate system, respectively. Represents the transpose of a vector or matrix; This represents matrix multiplication. To ensure that the alternating illumination and feature decoupling occur reliably within a preset time period, the system's hardware selection and timing configuration are constrained within specific physical boundaries; the distance between the light emission centers of the left and right light source groups is set between 40 mm and 120 mm to ensure that sufficient highlight offset can be generated at close range and resolved by the pixel matrix. The acquisition frame rate of the monocular camera is set to 50 to 120 frames per second, and the exposure time of a single frame is limited to 0.5 to 3 milliseconds, thereby minimizing the movement of the flight platform between frames. In addition, the pulse width of the light source is controlled to 0.6 to 0.8 times the exposure time of a single frame, and the synchronization error of the hardware trigger signal is strictly controlled within 5 microseconds to ensure that the light source has sufficient energy and no cross-frame crosstalk occurs within each exposure window.

[0027] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A flight camera attitude control system based on visual servoing, characterized in that, It includes a flight platform, a monocular camera installed at the front center of the flight platform for acquiring images of the inspected surface, a left light source group and a right light source group set on the left and right sides of the front of the flight platform, a synchronization controller, an inertial sensor, an image processing unit, and a flight controller. The intrinsic parameters of the monocular camera and the relative pose of the monocular camera with the left and right light source groups are pre-calibrated and known. The synchronization controller controls the lighting of only one side of the light source group to emit light pulses during the single-frame exposure time of the monocular camera, so as to obtain adjacent left and right illumination images. The image processing unit receives the left illumination image, the right illumination image, and angular velocity information, performs compensation processing on adjacent images and classifies feature points, determines the visual pose data of the monocular camera relative to the inspected surface based on texture feature points, and determines the distance estimate and normal estimate based on specular feature points and in combination with a pre-established distance mapping table and normal mapping table. The flight controller calculates a data reliability value based on the number of texture feature points and the number of highlight feature points. The data reliability value is the ratio of the number of texture feature points to the total number of valid feature points, where the total number of valid feature points is the sum of the number of texture feature points and the number of highlight feature points. The flight controller has a pre-calibrated fixed value, a first confidence threshold and a second confidence threshold, and the first confidence threshold is greater than the second confidence threshold. The flight controller adjusts the attitude control proportional gain matrix and damping gain matrix, both of which are 3×3, based on the comparison result between the data confidence value and the first confidence threshold and the second confidence threshold, and outputs attitude control commands.

2. The flight camera attitude control system based on visual servoing according to claim 1, characterized in that, The monocular camera is a global shutter camera; The synchronization controller controls the left light source group and the right light source group to light up alternately according to the frame number, wherein odd-numbered frames correspond to the left illumination image and even-numbered frames correspond to the right illumination image; The emission pulses of the left and right light source groups are located within the single-frame exposure time of the monocular camera, and only one light source group emits light during each exposure.

3. The flight camera attitude control system based on visual servoing according to claim 2, characterized in that, The image processing unit is further configured to perform brightness balancing on the left light source group and the right light source group, the brightness balancing including: Remove dark areas at the image edges and overexposed / saturated areas directly exposed by local highlights from the left and right illumination images. Use the remaining effective pixel area as the preset detection area and calculate the average brightness within the preset detection area. The average brightness is compared with a preset reference brightness; The image processing unit outputs brightness adjustment parameters to the synchronization controller based on the comparison result. The synchronization controller then adjusts the driving current of the left light source group and the right light source group based on the brightness adjustment parameters, and limits the driving current between a preset upper current limit and a preset lower current limit.

4. The visual servo-based flight camera attitude control system according to claim 1, characterized in that, The image processing unit's compensation process for adjacent images includes: The system receives the angular velocity information output by the inertial sensor and obtains a three-axis angular increment vector based on the angular velocity information between the exposure center timestamps of adjacent left and right illumination images. The pixel offset caused by the flight platform rotation is determined based on the triaxial angular increment vector and the monocular camera intrinsic parameters; the feature point displacement residual used for subsequent classification is determined based on the feature point pixel movement between the right illumination image and the left illumination image and the pixel offset.

5. The flight camera attitude control system based on visual servoing according to claim 4, characterized in that, The image processing unit's classification process for feature points includes: Feature points with gray-level gradient magnitudes greater than the local gradient threshold are extracted from the left and right illumination images. A pixel template of a preset size is constructed with the feature points as the center. An image matching algorithm is used to perform neighborhood tracking and obtain pixel coordinates. Based on the component of the feature point displacement residual in the projection direction of the line connecting the emission centers of the left and right light source groups on the image plane and its vertical component, the direction offset ratio is determined based on the component of the feature point displacement residual in the projection direction and its vertical component; the proportion of pixels with gray values ​​higher than the gray value threshold within the preset window centered on the feature point is determined. Feature points with a displacement residual magnitude not greater than a first threshold are marked as texture feature points. Feature points with a displacement residual magnitude not less than a second threshold, a direction offset ratio not less than a direction threshold, and a bright pixel ratio not less than a bright pixel ratio threshold are marked as specular feature points. All other feature points are marked as invalid feature points. Feature points with displacement residuals between the first and second thresholds are classified as invalid feature points. The local gradient threshold and the grayscale threshold are constants preset based on prior ambient lighting conditions and the image sensor's background noise. The direction threshold is preset based on the limit angle offset deviation caused by system hardware installation tolerances. The bright pixel ratio threshold is preset based on the ratio of the typical spot area of ​​specular reflection on the image to the area of ​​a preset window.

6. The visual servo-based flight camera attitude control system according to claim 5, characterized in that, The first threshold and the second threshold are dynamically adjusted thresholds; The first threshold is adaptively determined based on the variance of the feature point displacement residual and the angular increment modulus between adjacent images. The second threshold is greater than the first threshold and not less than a preset minimum highlight offset threshold, wherein the minimum highlight offset threshold is the displacement residual threshold along the projection direction of the line connecting the light emission centers on the image plane.

7. The flight camera attitude control system based on visual servoing according to claim 1, characterized in that, The visual pose data of the monocular camera relative to the inspected surface includes lateral displacement, longitudinal displacement, distance direction displacement, and yaw angle change, and serves as the attitude control input data for the flight controller.

8. The flight camera attitude control system based on visual servoing according to claim 5, characterized in that, The image processing unit is further configured to calculate distance estimates and normal estimates based on the specular feature points, the calculation process including: Offset parameters are extracted based on the image features of each highlight feature point; based on the offset parameters, distance estimates and normal estimates are determined using a preset distance mapping table and normal mapping table.

9. The visual servo-based flight camera attitude control system according to claim 1, characterized in that, The flight controller is used for: When the data confidence value is greater than or equal to the first confidence threshold, the proportional gain matrix of attitude control is increased; when the data confidence value is less than or equal to the second confidence threshold, the proportional gain matrix is ​​decreased and the damping gain matrix is ​​increased. When the data confidence value is greater than the second confidence threshold and less than the first confidence threshold, the preset proportional gain matrix and damping gain matrix are maintained. The system receives the desired attitude and preset target normal based on the inspection task, and outputs attitude control commands based on the attitude error between the current attitude of the flight platform and the desired attitude, as well as the angle error between the estimated normal value and the preset target normal.

10. A flight camera attitude control system based on visual servoing according to claim 1, characterized in that, The distance between the light-emitting centers of the left light source group and the right light source group is 40 mm to 120 mm; The monocular camera has a frame rate of 50 to 120 frames per second. The single-frame exposure time is 0.5 milliseconds to 3 milliseconds; the pulse width of the light emission from the left light source group and the right light source group is 0.6 to 0.8 times the single-frame exposure time; The synchronization error between the exposure timing of the monocular camera and the trigger timing of the left and right light source groups is no greater than 5 microseconds.

Images