An unmanned aerial vehicle autonomous inspection method, system, device and medium for a valve hall

Abstract

The application discloses a kind of unmanned aerial vehicle autonomous inspection method, system, equipment and medium of valve hall equipment, characterized in that, including: obtain the motion data of unmanned aerial vehicle in the process of inspection flight and the video sequence collected in valve hall;Each image frame of video sequence is input to target detection model, and the detection frame of the detection frame of converter valve, bushing and other equipment is obtained, based on each prediction frame and each detection frame determined comprehensive matching cost function by Kalman filtering algorithm prediction, with the minimum total cost as target Hungarian algorithm is used to solve comprehensive matching cost function, obtain the continuous trajectory information of valve hall equipment surface;Based on continuous trajectory information and unmanned aerial vehicle pose, calculate inspection attitude error vector, and generate motor driving signal to realize the autonomous inspection of unmanned aerial vehicle to valve hall equipment.The application can improve the control accuracy of unmanned aerial vehicle in the valve hall environment to implement autonomous inspection.

Description

Technical Field

[0001] This invention relates to the field of intelligent inspection technology for power systems, and in particular to a method, system, equipment, and medium for unmanned aerial vehicle (UAV) autonomous inspection of valve hall equipment. Background Technology

[0002] With the rapid development of ultra-high voltage direct current (UHVDC) transmission projects, the valve hall of the converter station, as the core hub of DC transmission, directly affects the safe and stable operation of the entire power system due to the health status of its internal equipment (such as converter valves, bushings, and busbars). Valve halls are characterized by complex spatial structures, dense equipment, high voltage levels, and strong electromagnetic environments, and are in a closed state during operation, making personnel inaccessible. Currently, the inspection of valve hall equipment mainly relies on fixed-point cameras or manual inspections after periodic power outages. Fixed cameras have limitations such as blind spots and the inability to observe equipment details up close; while power outage inspections affect system availability and cannot detect abnormalities such as overheating and corona discharge in operating equipment. To improve the status perception capabilities and intelligent inspection level of valve hall equipment, the use of drones equipped with visual sensors and inertial measurement units to automatically identify key equipment within the valve hall and conduct autonomous fly-around inspections has become an important development direction for ensuring uninterrupted power supply and refined operation and maintenance of UHVDC transmission projects.

[0003] Therefore, it is necessary to propose a technical solution that can stably identify valve hall equipment targets in complex environments and achieve continuous tracking and autonomous inspection flight control based on high-precision pose calculation, so as to improve the stability and control accuracy of UAV inspection. However, existing technologies still have shortcomings in terms of the stability of continuous identification of valve hall equipment targets by UAVs and the accuracy of inspection flight control. On the one hand, existing visual tracking methods mostly adopt simple inter-frame feature matching or local association based on a single motion model. In the case of short-term target occlusion, fluctuations in detection confidence, or interference from adjacent structures, mismatches or trajectory breaks are easily generated, making it difficult to form stable and continuous target trajectory information, thus affecting the reliability of the inspection process. On the other hand, existing autonomous flight control mostly relies on the loose coupling fusion of global positioning system and inertial data, or only on the single visual pose estimation result for control. It lacks a pose estimation mechanism that jointly optimizes and solves the visual feature reprojection error and inertial pre-integration error, which limits the accuracy of UAV degree-of-freedom pose calculation, thus affecting the accuracy of its spatial offset calculation relative to the valve hall equipment, making it difficult to achieve stable and accurate autonomous inspection flight control. Summary of the Invention

[0004] This invention provides a method, system, equipment, and medium for autonomous inspection of valve hall equipment by unmanned aerial vehicles (UAVs), which can improve the control accuracy of UAVs performing autonomous inspections in valve hall environments.

[0005] In a first aspect, embodiments of the present invention provide a method for autonomous inspection of valve hall equipment using unmanned aerial vehicles (UAVs), comprising: Acquire motion data and video sequences of valve hall equipment during the inspection flight of the UAV; Each image frame of the video sequence of the valve hall equipment is input into a preset target detection model to obtain several detection boxes of the valve hall equipment. Based on each predicted box and each of the detection boxes, a comprehensive matching cost function is determined. The Hungarian algorithm is used to solve the comprehensive matching cost function with the goal of minimizing the total cost to obtain continuous trajectory information on the surface of the valve hall equipment. Each predicted box is obtained by extrapolating the historical detection boxes of the valve hall equipment through the Kalman filter algorithm. The valve hall equipment includes a converter valve, a bushing, and a busbar. Based on the continuous trajectory information and the UAV's degree of freedom pose, the inspection attitude error vector of the UAV relative to the valve hall equipment is calculated, and a motor drive signal is generated based on the inspection attitude error vector to realize the UAV's autonomous inspection of the valve hall equipment. The UAV's degree of freedom pose is determined by solving a joint optimization objective function constructed based on preset camera feature points and the motion data.

[0006] This invention achieves synchronous perception of external targets and its own motion state by simultaneously acquiring video sequences of valve hall equipment and UAV motion data, providing a multi-source data foundation for subsequent trajectory construction and pose estimation, thereby improving the control accuracy of UAV autonomous inspection of valve hall equipment targets. A target detection model identifies and locates valve hall equipment targets in each image frame, obtaining structured detection box information to ensure the accuracy of target spatial location acquisition and provide reliable input for continuous trajectory generation, thus improving the control accuracy of UAV autonomous inspection of valve hall equipment targets. A Kalman filter algorithm is used for time extrapolation to generate prediction boxes, and a comprehensive matching cost function is constructed. A Hungarian algorithm is then used to achieve global matching. Optimal matching ensures the continuity and stability of the target trajectory, reducing the impact of mismatches on the control system, thereby improving the control accuracy of the UAV's autonomous inspection of valve hall equipment targets. By calculating the inspection attitude error vector based on continuous trajectory information and the UAV's degree of freedom pose, and generating motor drive signals accordingly, closed-loop control adjustment is achieved, reducing control lag and oscillation, thus improving the control accuracy of the UAV's autonomous inspection of valve hall equipment targets. By jointly optimizing the visual reprojection error and inertial pre-integration error, the accuracy and stability of the UAV's six-degree-of-freedom pose estimation are improved, providing an accurate benchmark for error vector calculation, thereby improving the control accuracy of the UAV's autonomous inspection of valve hall equipment targets.

[0007] Furthermore, determining the comprehensive matching cost function based on each predicted bounding box and each detected bounding box includes: Calculate the Mahalanobis distance between each of the predicted boxes and each of the detected boxes, and determine the motion cost based on each of the Mahalanobis distances; Appearance features are extracted from each of the predicted boxes and each of the detected boxes to obtain first appearance features and second appearance features, respectively. The cosine distance between each of the first appearance features and each of the second appearance features is calculated to determine the appearance cost based on each of the cosine distances. Feature point matching is performed on each of the predicted boxes and each of the detected boxes to obtain matching point pairs. The structural similarity score is calculated based on the spatial distribution consistency of each matching point pair, and the structural cost is determined based on the structural similarity score. The motion cost, appearance cost, and structural cost are weighted and fused according to preset weights to obtain the comprehensive matching cost function between each predicted box and each detected box.

[0008] This invention improves the accuracy and robustness of matching predicted and detected boxes by integrating motion cost, appearance cost, and structural cost to construct a comprehensive matching cost function. This enhances the continuous and stable tracking capability in complex environments, thereby providing UAVs with more accurate and continuous equipment location information and further improving the control precision of UAVs performing autonomous inspections in valve hall environments.

[0009] Furthermore, the step of inputting each image frame of the video sequence of the valve hall equipment into a preset target detection model to obtain several detection boxes of the valve hall equipment includes: Each image frame of the video sequence of the valve hall equipment is input into a preset target detection model, and the backbone network in the target detection model is used to extract features from each image frame at several levels to obtain a backbone feature map for enhancing small-scale equipment components in the valve hall equipment, wherein the small-scale equipment components include equalizing rings and bolts. Based on the channel attention mechanism and the spatial attention mechanism, the backbone feature map is jointly weighted and calculated to obtain the target area enhancement feature map for enhancing the valve plate of the converter valve, the bushing skirt and the busbar in the valve hall equipment. Based on the feature pyramid structure in the target detection model, the enhanced feature map of the target region is subjected to multi-scale fusion processing to obtain a multi-scale fused feature map. Boundary box regression calculation is performed on the multi-scale fused feature map to obtain several detection boxes containing category information and location parameters.

[0010] The embodiments of the present invention improve the detection accuracy and environmental adaptability of small-scale equipment components (such as equalizing rings and bolts) in the valve hall by multi-level feature extraction, attention enhancement and multi-scale fusion processing, thereby providing more accurate and reliable target location information for subsequent stable tracking and UAV autonomous inspection in the valve hall environment.

[0011] Furthermore, acquiring the motion data of the UAV during the inspection flight and the video sequence of the valve hall equipment includes: The initial motion data is calibrated with zero bias to obtain the motion data; Multi-scale wavelet decomposition is performed on the acquired initial valve hall equipment video sequence to calculate the high-frequency subband energy of each image frame in the initial valve hall equipment video sequence and compare it with a preset energy threshold to obtain the comparison result, and the blurred region mask image is determined based on the comparison result. The blurred region mask image is input into a preset deblurring neural network model, so that the blurred region mask image is processed by the deformation convolution and channel attention mechanism in the deblurring neural network model to extract features and output a deblurred and enhanced image sequence. Calculate the perspective transformation matrix between each image frame in the deblurred and enhanced image sequence, and perform inverse perspective transformation processing on each image frame based on each perspective transformation matrix to obtain the valve hall equipment video sequence.

[0012] This invention improves the clarity and geometric consistency of video data from valve hall equipment by performing zero-bias calibration on motion data, identifying and deblurring video sequences, and correcting image distortion using perspective transformation. This provides more stable and accurate input information for subsequent target detection, tracking, and autonomous inspection control of unmanned aerial vehicles.

[0013] Furthermore, each of the predicted frames is obtained by extrapolating the historical detection frames of the valve hall equipment using a Kalman filter algorithm, including: Extract the center position coordinates, width, height, and corresponding velocity components of each historical detection box, and combine the center position coordinates, width, height, and velocity components to form an initial state vector; The state prediction formula based on the Kalman filter algorithm is used to perform a linear state transition operation on the initial state vector to obtain the predicted state vector. The predicted state vector is extracted to reconstruct the boundary box geometry of the valve hall equipment, thus obtaining each predicted box.

[0014] This invention constructs a predicted frame by performing Kalman filtering time extrapolation prediction on historical detection frames to compensate for information loss caused by detection gaps and short-term occlusions. This improves the continuity and stability of the target trajectory, thereby providing smoother and more reliable position information support for UAVs to perform autonomous inspections in valve hall environments and enhancing overall control accuracy.

[0015] Furthermore, the step of calculating the inspection attitude error vector of the UAV relative to the valve hall equipment based on the continuous trajectory information and the UAV's degrees of freedom pose includes: Extract the reference position vector and reference velocity vector of the continuous trajectory information in the world coordinate system; By performing difference operations on the reference position vector and the target position vector, and on the reference velocity vector and the target velocity vector, respectively, the position error vector and the velocity error vector are obtained. The attitude deviation matrix is ​​obtained by performing relative rotation calculation between the reference rotation matrix and the attitude rotation matrix, and the attitude deviation matrix is ​​decomposed into attitude angles to obtain the attitude error vector. The target position vector, the target velocity vector, and the attitude rotation matrix are determined based on the UAV's degree of freedom pose, and the reference rotation matrix is ​​determined based on the reference velocity vector. The position error vector, the velocity error vector, and the attitude error vector are combined to obtain the inspection attitude error vector of the UAV relative to the valve hall equipment.

[0016] This invention, through error modeling of continuous trajectory information and UAV six-degree-of-freedom pose in a unified coordinate system, constructs a comprehensive inspection attitude error vector of position, velocity, and attitude, thereby realizing a precise relative state description between the UAV and the valve hall equipment. This provides a high-precision error signal for subsequent closed-loop control, further improving the stability and control accuracy of the UAV's autonomous inspection in the valve hall environment.

[0017] Furthermore, the step of generating motor drive signals based on the inspection attitude error vector to achieve autonomous inspection flight of the UAV over the valve hall equipment includes: The inspection attitude error vector is subjected to proportional, integral, and differential operations to obtain the corresponding proportional, integral, and differential operation results, respectively. The proportional calculation result, the integral calculation result, and the differential calculation result are weighted and summed according to a preset proportional gain matrix, integral gain matrix, and differential gain matrix to obtain the attitude correction control quantity; Based on a preset attitude assignment matrix, the attitude correction control quantity is mapped to the pulse width modulation signal corresponding to each motor in the UAV, so as to realize the UAV's autonomous inspection flight of the valve hall equipment based on the pulse width modulation signal.

[0018] This invention achieves a rapid closed-loop response from error to actuator by comprehensively adjusting the inspection attitude error vector using proportional-integral-derivative methods and combining it with the attitude allocation matrix to accurately map the correction control quantity to the pulse width modulation signal of each motor. This effectively improves the dynamic stability, anti-disturbance capability, and control accuracy of the UAV during autonomous inspection in the valve hall environment.

[0019] Secondly, embodiments of the present invention provide an unmanned aerial vehicle (UAV) autonomous inspection system for valve hall equipment, the system comprising: an acquisition module, a solution module, and a control module; The acquisition module is used to acquire motion data of the UAV during the inspection flight and the video sequence of the valve hall equipment collected. The solution module is used to input each image frame of the video sequence of the valve hall equipment into a preset target detection model to obtain several detection boxes of the valve hall equipment. Based on each predicted box and each of the detection boxes, a comprehensive matching cost function is determined. The Hungarian algorithm is used to solve the comprehensive matching cost function with the goal of minimizing the total cost to obtain continuous trajectory information on the surface of the valve hall equipment. Each predicted box is obtained by extrapolating the historical detection boxes of the valve hall equipment through the Kalman filter algorithm. The valve hall equipment includes a converter valve, a bushing, and a busbar. The control module is used to calculate the inspection attitude error vector of the UAV relative to the valve hall equipment based on the continuous trajectory information and the UAV's degree of freedom pose, and to generate a motor drive signal based on the inspection attitude error vector to realize the UAV's autonomous inspection of the valve hall equipment. The UAV's degree of freedom pose is determined by solving a joint optimization objective function constructed based on preset camera feature points and the motion data.

[0020] This invention, through the construction of a system architecture in which the acquisition module, the solution module, and the control module work together, achieves integrated processing of target detection in valve hall equipment, continuous trajectory tracking, and closed-loop attitude control of UAVs. It deeply integrates visual perception results with six-degree-of-freedom pose calculation results, forming a complete closed-loop link from environmental perception to motion execution, thereby significantly improving the continuity, stability, and control accuracy of UAVs in autonomous inspection within the valve hall environment.

[0021] Thirdly, embodiments of the present invention provide a terminal device, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; The memory is used to store at least one executable instruction that causes the processor to perform operations as described in this application of an unmanned aerial vehicle (UAV) autonomous inspection method for valve hall equipment.

[0022] Fourthly, embodiments of the present invention provide a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device or system where the computer-readable storage medium is located to perform an unmanned aerial vehicle (UAV) autonomous inspection method for valve hall equipment as described in this application.

[0023] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0024] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating an embodiment of the unmanned aerial vehicle (UAV) autonomous inspection method for valve hall equipment provided in this application; Figure 2 This is a flowchart illustrating steps S201 to S204 provided in this application; Figure 3 This is a flowchart illustrating steps S301 to S304 provided in this application; Figure 4 This is a schematic diagram of an embodiment of an unmanned aerial vehicle (UAV) autonomous inspection method for valve hall equipment provided in this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0028] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0029] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0030] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0031] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0032] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0033] With the expansion of power grid scale and the increasingly complex distribution environment of valve hall equipment, traditional manual inspection methods suffer from low efficiency, high risk, and poor real-time performance in mountainous canyons, densely packed tower areas, and post-disaster environments, prompting UAV autonomous inspection technology to become an important development direction. However, existing technologies still have shortcomings in continuous target recognition and flight control accuracy: on the one hand, visual tracking methods based on a single motion model or simple inter-frame matching are prone to mismatches and trajectory interruptions under conditions of occlusion, confidence fluctuations, or structural interference, making it difficult to obtain stable and continuous equipment surface trajectories; on the other hand, existing pose estimation methods mostly adopt loosely coupled fusion or single visual control methods, lacking a joint optimization mechanism for visual reprojection error and inertial pre-integration error, resulting in limited accuracy of UAV six-degree-of-freedom pose calculation, which in turn affects the stability and control accuracy of autonomous inspection in valve hall environments.

[0034] See Figure 1 In order to improve the control accuracy of UAVs performing autonomous inspections in valve hall environments, an embodiment of the present invention provides a method for autonomous inspection of valve hall equipment by UAVs, including steps S101 to S103. Step S101: Acquire motion data of the UAV during the inspection flight and collect video sequences of the valve hall equipment; In some embodiments, the valve hall equipment is the core equipment in the converter station, mainly including converter valves, cooling systems, surge arresters, etc., used to realize AC and DC conversion. The valve hall typically features a high-voltage electric field, compact space, high-value equipment, and extremely high environmental cleanliness requirements, making it a closed equipment area with significant inspection challenges. The UAV is equipped with a visual sensor module and an inertial measurement module. The visual sensor module is used to collect video image data of the valve hall equipment inspection area, and the inertial measurement module is used to collect motion data of the UAV during flight. The visual sensor module can be an industrial-grade visible light camera, mounted on a three-axis gimbal structure under the UAV fuselage to ensure stable alignment of the image acquisition direction with the valve hall equipment during flight. The inertial measurement module includes a three-axis accelerometer and a three-axis gyroscope, used to output the UAV's linear acceleration and angular velocity data in real time. Before takeoff, the inertial measurement module undergoes zero-bias calibration, specifically including: continuously acquiring raw acceleration and angular velocity data within a preset time period while the drone is stationary; calculating the average value of each axis as a zero-bias estimate; and storing the zero-bias estimate in the control system for real-time compensation during subsequent flight, thereby obtaining calibrated motion data. During the inspection flight, the vision sensor module continuously acquires video image frames of the valve hall equipment at a preset frame rate, forming a raw video sequence; simultaneously, the inertial measurement module continuously outputs linear acceleration and angular velocity data at a sampling frequency higher than the video frame rate. To ensure time consistency, the visual and inertial data undergoes timestamp synchronization processing, specifically: assigning a corresponding timestamp to each frame of image data, and interpolating and aligning the inertial data at the same or adjacent times according to the timestamps to generate a time-aligned motion data sequence.

[0035] In some embodiments, the motion data further includes UAV attitude angle data, velocity estimation data, or altitude data output by the flight controller. The attitude angle data includes roll, pitch, and yaw angles, used to characterize the UAV's attitude state relative to the ground coordinate system; the velocity estimation data is used to characterize the UAV's three-axis velocity components in the world coordinate system. Regarding data transmission, the visual data and the motion data are cached and preprocessed by an onboard processing unit, which can be an embedded computing platform. The visual data is stored in a video buffer as an image frame cache, and the motion data is stored in a state buffer as a time series, arranged chronologically for subsequent use by image enhancement, target detection, and trajectory estimation modules.

[0036] Through the above steps, high-precision acquisition, zero-bias calibration, and time synchronization processing of visual and motion data are achieved, ensuring accurate alignment and stable output of multi-source sensor information under a unified time reference. This provides a reliable data foundation for subsequent target detection, trajectory estimation, and attitude control, thereby improving the stability and control accuracy of UAVs in autonomous inspection within the valve hall environment.

[0037] Step S102: Input each image frame of the video sequence of the valve hall equipment into a preset target detection model to obtain several detection boxes of the valve hall equipment. Determine a comprehensive matching cost function based on each predicted box and each of the detection boxes. Use the Hungarian algorithm to solve the comprehensive matching cost function with the goal of minimizing the total cost to obtain continuous trajectory information on the surface of the valve hall equipment. Each predicted box is obtained by extrapolating the historical detection boxes of the valve hall equipment using the Kalman filter algorithm. The valve hall equipment includes a converter valve, a bushing, and a busbar. In some embodiments, acquiring motion data of the UAV during inspection flight and the collected video sequence of valve hall equipment includes: performing zero-bias calibration on the collected initial motion data to obtain motion data; performing multi-scale wavelet decomposition on the collected initial valve hall equipment video sequence to calculate the high-frequency subband energy of each image frame in the initial valve hall equipment video sequence and compare it with a preset energy threshold to obtain a comparison result, and determining a blurred region mask image based on the comparison result; inputting the blurred region mask image into a preset deblurring neural network model to perform feature extraction processing on the blurred region mask image through deformation convolution and channel attention mechanisms in the deblurring neural network model, and outputting a deblurred enhanced image sequence; calculating the perspective transformation matrix between each image frame in the deblurred enhanced image sequence, and performing inverse perspective transformation processing on each image frame based on each perspective transformation matrix to obtain the valve hall equipment video sequence.

[0038] In some embodiments, the initial motion data collected is calibrated to obtain the motion data. Specifically, the UAV is equipped with an inertial measurement unit (IMU) to output triaxial acceleration and triaxial angular velocity data. Due to manufacturing errors and temperature drift in the sensor, its raw output contains a bias error, therefore, bias calibration is required before flight. Let the initial measurement value of the triaxial accelerometer be: ,in, This represents the original measurement vector of the accelerometer. Represents the true acceleration vector. Indicates zero bias acceleration. This represents the measurement noise. Let the initial measurement value of the gyroscope be: ,in, Represents the original measurement vector of angular velocity. Represents the true angular velocity. Indicates zero angular velocity bias. This indicates measurement noise. Continuous data was collected while the UAV was stationary. The data sets are averaged over time for each axis measurement to obtain a zero-bias estimate. ; in, Indicates the number of sampling points; Indicates the first The acceleration measurement value; Indicates the first Secondary angular velocity measurement. Zero-bias compensation is performed on real-time data during flight. ; in, The calibrated acceleration; This represents the calibrated angular velocity. The calibrated motion data is used for subsequent vision-inertial fusion algorithms for attitude estimation.

[0039] In some embodiments, multi-scale wavelet decomposition is performed on the acquired initial valve hall equipment video sequence to calculate the high-frequency subband energy of each image frame in the initial valve hall equipment video sequence and compare it with a preset energy threshold to obtain a comparison result. Based on the comparison result, a mask image for blurred regions is determined. Specifically, since the high-speed flight of the UAV causes motion blur at the edges of the valve hall equipment, it is first necessary to identify the blurred regions. For each frame of the video sequence... Perform multi-scale Haar wavelet decomposition to obtain the first... Three high-frequency sub-bands at the layer scale: Horizontal high-frequency sub-band; Vertical high-frequency sub-band; : Diagonal high-frequency sub-band. At pixel position The high-frequency energy is defined as: ; in, Indicates the wavelet decomposition level; Indicates the first Layered zone In pixels The grayscale value at that location; This represents the high-frequency energy of that pixel. The average high-frequency energy was calculated by statistically analyzing 1000 sets of clear valve hall equipment image samples. Set a fuzzy judgment threshold: ,in, The proportionality constant is set to 0.6. 0.7 is the midpoint of the interval; This is the threshold for fuzzy judgment. When... When a pixel is identified as blurred, a binary mask image is constructed. Then, the following method is used: 3 3 cross-shaped structural elements are etched, and then through 5 The five square structural elements are expanded; morphological optimization is performed on blurred areas, with a positioning error of less than 2 pixels. When the ambient light is below 50 lux, the totem pole-type fill light system is automatically triggered to improve image brightness.

[0040] In some embodiments, the blurred region mask image is input into a preset deblurring neural network model to perform feature extraction processing on the blurred region mask image through deformation convolution and channel attention mechanisms in the deblurring neural network model, outputting a deblurred and enhanced image sequence. Specifically, this embodiment uses an improved DeblurGANv2 model for image restoration. To enhance the recovery capability of valve hall equipment edges, deformation convolution is introduced into the generator network, and its calculation form is as follows: ; in, Indicates the position of the output pixel; This is the set of convolution sampling locations; These are the convolution weights; For learnable offsets; Input feature map; This is to output a feature map. A channel attention mechanism is also introduced, with the kernel size depending on the number of input channels. Adaptive determination: ,in, This refers to the number of channels in the feature map. The size of a one-dimensional convolution kernel; This indicates rounding down. The loss function uses the joint loss form: ,in, To combat the losses; For perceptual loss based on VGG features; This represents the total variational loss; The training data contains 5000 sets of blurred images of valve hall equipment. The output is a sequence of deblurred and enhanced images.

[0041] In some embodiments, the perspective transformation matrix between each image frame in the deblurred and enhanced image sequence is calculated, and the inverse perspective transformation processing is performed on each image frame based on the perspective transformation matrix to obtain the valve hall equipment video sequence. Specifically, to eliminate the camera motion effect caused by the drone's flight, global motion compensation based on the homography matrix is ​​adopted. First, for adjacent two frames of images... and Extract SIFT feature points and establish matching relationships. Estimate the homography matrix using the RANSAC algorithm. Its mathematical expression is: ; in, The coordinates are in the reference frame; The corresponding coordinates in the current frame; 3 3. Homography matrix. Perspective transformation is explicitly expressed as: ; in, These are the elements of the homography matrix. To eliminate camera motion, an inverse transform is performed on the current frame. The pixel grayscale values ​​are calculated using bilinear interpolation. After compensation, the static background achieves stable alignment, and the motion of the target device in the image mainly reflects the relative motion between the UAV and the target device. The compensated image is then cropped to an effective region, with an alignment error of less than 1.5 pixels. The result is stored in a 10-frame buffer, forming a closed loop of "preprocessing-compensation-buffering".

[0042] Please refer to Figure 2 In some embodiments, the step of inputting each image frame of the video sequence of the valve hall equipment into a preset target detection model to obtain several detection boxes of the valve hall equipment includes: steps S201 to S204. Step S201: Input each image frame of the video sequence of the valve hall equipment into a preset target detection model, so as to perform feature extraction of each image frame at several levels through the backbone network in the target detection model, and obtain the backbone feature map for enhancing the small-scale equipment components in the valve hall equipment, wherein the small-scale equipment components include equalizing rings and bolts; In some embodiments, the target detection model employs an improved YOLOv5 network structure, with the backbone network using the CSPDarknet53 structure. This structure divides the feature map into two branches for parallel computation through a cross-stage partial connection mechanism; one branch participates in residual calculation, while the other performs cross-layer connections, thereby improving the multi-granularity feature representation capability while ensuring gradient flow stability. Considering the characteristics of the sleeve in the valve hall equipment scenario—a slender linear structure—and the valve tower of the converter valve—a multi-layered repeating structure—a dilated convolution module is introduced in stages 3 to 5 of CSPDarknet53 to expand the receptive field and enhance the global modeling capability for slender structures. The formula for calculating the receptive field of dilated convolution is: ; in, This indicates the size of the receptive field of the dilated convolution; This represents the size of the receptive field in a standard convolution; Indicates the kernel size; The dilation rate is represented by the number of holes. By setting different dilation rates, the backbone network expands its receptive range while maintaining the same resolution, thereby enhancing its ability to capture slender targets in the valve hall. After multiple convolutions and downsampling by the backbone network, a multi-scale backbone feature map is output, which serves as the input for subsequent attention enhancement and multi-scale fusion.

[0043] Step S202: Perform joint weighted calculation on the backbone feature map based on the channel attention mechanism and the spatial attention mechanism to obtain the target area enhancement feature map for enhancing the converter valve or bushing in the valve hall equipment; In some embodiments, the feature map output by the backbone network The CBAM attention module is introduced, including a channel attention submodule and a spatial attention submodule. Channel attention is used to enhance the channel characteristics of key targets such as converter valves and bushings, while suppressing background interference from valve hall walls, steel beams, and adjacent equipment. Its calculation form is as follows: ; in, Input feature map; To perform global average pooling on the feature map; To perform global max pooling on the feature map; It is a multilayer perceptron; Use the Sigmoid activation function; The channel attention weights are multiplied channel by channel by channel of the input feature map to obtain the enhanced channel feature map. Spatial attention is used to enhance the linear distribution region of the bushing and the local structural region of the converter valve tower. Its calculation form is: ,in, Indicates channel splicing; express Convolution operation; This is a spatial attention map. The final output enhanced feature map is: ,in, The feature map after channel attention processing; This represents element-wise multiplication. After double attention weighting, the enhanced feature map of the target region is obtained.

[0044] Step S203: Perform multi-scale fusion processing on the enhanced feature map of the target region based on the feature pyramid structure in the target detection model to obtain a multi-scale fused feature map; In some embodiments, the feature pyramid structure adopts a combination of FPN and PAN. The FPN structure fuses high-level semantic information with low-level spatial information through top-down feature propagation; the PAN structure further enhances the semantic expressive power of low-level features through bottom-up path enhancement. Simultaneously, an SPPF module is introduced into the neck network to improve the global receptive power of features through multi-scale pooling operations. Furthermore, a multi-scale weighted fusion mechanism is introduced to assign different weights to features at different scales. ,in, Indicates the first Each scale feature map; This represents the fusion weights at the corresponding scale; Indicates the number of scales; This represents the fused feature map. During the training phase, a phase grouping labeling strategy is adopted, which labels the converter valve tower and its auxiliary equipment in the same phase as a whole target, thereby enhancing the model's ability to learn the overall structural relationship of the valve hall equipment and improving the detection stability in complex environments.

[0045] Step S204: Perform bounding box regression calculation on the multi-scale fused feature map to obtain several detection boxes containing category information and location parameters.

[0046] In some embodiments, the detection head performs convolutional prediction on the fused feature map, outputting the class probability, confidence score, and location parameters for each candidate box. Bounding box regression uses the following parameter representation: ,in, Indicates the coordinates of the center point of the prediction box; This represents the width and height of the predicted bounding box. The location regression process is as follows: ; in, These are the regression parameters output by the network; This is the grid offset; The prior frame size; The function used is the Sigmoid function. The final output consists of several bounding boxes, each containing: a category label; a confidence score; and location information parameters. These detection results are then passed to the target tracking module for data association and trajectory generation.

[0047] Please refer to Figure 3 In some embodiments, determining the comprehensive matching cost function based on each predicted box and each detected box includes: steps S301 to S304; Step S301: Calculate the Mahalanobis distance between each of the predicted boxes and each of the detected boxes, so as to determine the motion cost based on each of the Mahalanobis distances; In some embodiments, the target tracking module receives the current frame detection box information (including bounding box position, category label, and confidence level) output by the target detection module and associates it with the trackers established in the previous frame. In this embodiment, each tracker has a built-in Kalman filter. Considering the characteristic that valve hall equipment targets (converter valves, bushings) exhibit approximately uniform linear motion over a short period, the state vector is defined as: ,in, The coordinates of the center of the target bounding box; The velocity component at the target center; Define the width and height of the target bounding box; The ratio of width to height; The current frame time. The Kalman filter prediction formula is: ; in, For the predicted state; To predict the covariance matrix; This is the state transition matrix; The process noise covariance matrix is ​​adaptively set based on the UAV flight speed and camera frame rate. After prediction, the predicted bounding box of the current frame is obtained. Let the current frame be the first... The observations for each detection frame are: The predicted observations are: ,in, The observation matrix; For the first The first tracker. Then the second... The detection box and the first The Mahalanobis distance between the predicted boxes is: ,in, , To predict the mapping of covariance in the observation space; To observe the noise covariance matrix, the Mahalanobis distance is used as the motion cost: ,when If the threshold is exceeded, it is directly determined as unmatchable, thereby improving matching efficiency and avoiding false associations.

[0048] Step S302: Extract appearance features from each of the predicted boxes and each of the detected boxes to obtain first appearance features and second appearance features, and calculate the cosine distance between each of the first appearance features and each of the second appearance features to determine the appearance cost based on each of the cosine distances. In some embodiments, considering the periodic repetitive texture of the converter valve disc and the umbrella-skirt structure on the sleeve surface, this embodiment introduces a deep convolutional network to extract ReID features. The detection box region is cropped and input into a pre-trained convolutional neural network; the output is a 128-dimensional or 256-dimensional normalized feature vector. Let the... Each detection box has the following features: ;No. The historical appearance characteristics of each tracker maintenance are as follows: The tracker's historical features are updated using an exponential moving average: ,in, To update the weight coefficients, appearance similarity is represented by cosine distance: ,in," " represents the vector dot product; Represents the vector norm. Defines the appearance cost: This metric is robust under conditions of brief shading or changes in light intensity.

[0049] Step S303: Perform feature point matching on each of the predicted boxes and each of the detection boxes to obtain matching point pairs, and calculate the structural similarity score based on the spatial distribution consistency in each of the matching point pairs, so as to determine the structural cost based on the structural similarity score. In some embodiments, considering the characteristics of bushings having obvious linear texture features and converter valve towers having multi-layered repeating structures, this embodiment introduces an ORB feature point matching mechanism. ORB feature points are extracted within the detection box region; descriptor matching is performed with ORB feature points in the tracker's historical region; and nearest neighbor matching is performed using Hamming distance. Let the number of matched point pairs be... The total number of feature points is Spatial distribution stability is assessed using the consistency of affine transformations of matched point pairs. The structural similarity score is defined as: ,in, This represents the number of interior points verified by RANSAC. The structural cost is defined as: When there is ambiguity in the matching results between motion and appearance, structural cost serves as the basis for decision arbitration.

[0050] Step S304: The motion cost, appearance cost, and structural cost are weighted and fused according to preset weights to obtain the comprehensive matching cost function between each predicted box and each detected box.

[0051] In some embodiments, to achieve the fusion of multi-level matching mechanisms, this embodiment constructs a comprehensive matching cost function: ,in, The value of sports; Value for appearance; Value for the structure; Confidence of the detection box; The coefficient for balancing motion and appearance is preferably 0.5; These are the structural weighting coefficients; The confidence penalty coefficient can be set to 0.2. When the detection confidence level... When a "conservative update" mode is triggered, only Kalman trajectories are used to update the trajectory, without incorporating bounding box position information. After constructing the comprehensive cost matrix, the Hungarian algorithm is used to solve for the optimal allocation scheme, achieving a one-to-one match between the bounding boxes and the trackers. After matching: if a match is successful, a Kalman update is performed; if no bounding box is matched, a new tracker is initialized; if no tracker is matched, the occlusion prediction state is entered; if there are multiple consecutive frames without a match, the tracker is deleted.

[0052] In some embodiments, the Hungarian algorithm is used to solve the comprehensive matching cost function with the objective of minimizing the total cost, thereby obtaining continuous trajectory information of the valve hall equipment surface. Specifically, after weighted fusion of motion cost, appearance cost, and structural cost, a comprehensive matching cost matrix is ​​constructed between all detection boxes in the current frame and the prediction boxes of all active trackers. The comprehensive matching cost comprehensively reflects the consistency of targets in spatial location, similarity in appearance features, and consistency in local structural texture. Detection confidence is introduced for adjustment, giving higher-confidence detection results higher priority in the matching process. Subsequently, the comprehensive matching cost matrix is ​​input into the Hungarian algorithm for global optimal allocation. The Hungarian algorithm uses "minimizing the total cost" as its optimization objective. Under the constraints that each detection box matches at most one tracker and each tracker matches at most one detection box, it calculates the one-to-one matching relationship between the detection boxes and prediction boxes in the current frame. Compared to matching methods based on local greedy strategies, this method can avoid mismatches when multiple targets coexist, are close to each other, or briefly cross each other, thereby ensuring the global optimality and stability of the association results.

[0053] It should be noted that after obtaining the matching results, different types of matching relationships are processed separately: For successfully matched detection boxes and trackers, the Kalman filter prediction state is updated and corrected using the position information of the detection boxes. At the same time, the appearance feature vector and local structural feature set stored by the tracker are updated, and the unmatched counter is reset so that the target trajectory can continue continuously; For unmatched detection boxes, they are identified as newly appearing valve hall equipment targets or auxiliary equipment targets, and a new tracker instance is initialized for them, a unique identifier is assigned, and an initial motion state is established; For unmatched trackers, it is determined that they may be in a short-term occlusion or detection miss state, and their motion trajectory is extrapolated only based on the Kalman prediction results. The unmatched count is incremented frame by frame. When the number of consecutive unmatched frames exceeds a preset threshold, the tracker is terminated and resources are released. In the scenario of valve hall equipment inspection, when the converter valve or sleeve is temporarily blocked by adjacent equipment or valve hall structural beams and columns, the Hungarian algorithm, combined with the dynamic weight adjustment mechanism of comprehensive matching cost, can achieve target re-identification in subsequent frames through appearance features and local structural verification, so as to re-establish the matching relationship with the original tracker, thereby keeping the target identifier unchanged and avoiding trajectory interruption or ID switching problems.

[0054] In some embodiments, each of the predicted bounding boxes is obtained by extrapolating the historical detection bounding boxes of the valve hall equipment using a Kalman filter algorithm. This includes: extracting the center position coordinates, width, height, and corresponding velocity components of each historical detection bounding box to form an initial state vector; performing a linear state transition operation on the initial state vector based on the state prediction formula of the Kalman filter algorithm to obtain a predicted state vector; and extracting the predicted state vector to reconstruct the boundary box geometry information of the valve hall equipment to obtain each of the predicted bounding boxes.

[0055] In some embodiments, the center position coordinates, width, height, and corresponding velocity components of each historical detection frame are extracted to form an initial state vector. Specifically, when the target detection module outputs a new valve hall equipment target (including converter valve or bushing) detection frame in a certain frame, the target tracking module first performs parameter parsing on the detection frame. Let the coordinates of the upper left corner of the detection frame be... The coordinates of the lower right corner are Then its center position coordinates, width, and height are calculated as follows: ; in, , This indicates the position of the center of the target bounding box in the image coordinate system; Indicates the width of the target bounding box; This represents the height of the target bounding box. The initialization of the velocity component can be set when the target first appears: When the target is detected in two consecutive frames, the velocity component is calculated based on the difference in center coordinates between the adjacent frames. , ,in, , The coordinates of the center of the current frame; , The coordinates of the center of the previous frame; The time interval between two adjacent frames is determined by the camera frame rate. Considering that the target in the valve hall can be approximated as uniform linear motion within a short time (less than 1 second), the above parameters are combined to construct the initial state vector for the Kalman filter: ; in, Indicates the rate of change of width and height, which can be initially set to 0; superscript This represents the transpose operation. Through the above modeling, the state vector simultaneously characterizes the target's spatial location, motion trend, and scale change features, providing a unified state description for subsequent prediction and matching.

[0056] In some embodiments, a linear state transition operation is performed on the initial state vector based on the state prediction formula of the Kalman filter algorithm to obtain the predicted state vector. Specifically, after the state vector initialization is completed, a Kalman filter prediction step is performed on each frame. For the approximately rigid, uniform motion characteristics of the converter valve or bushing in the image, a linear state transition model is adopted: ,in, Represents the predicted state vector of the current frame; This represents the updated state vector from the previous frame; Let be the state transition matrix. Under the assumption of uniform motion, the state transition matrix can be constructed by including the time interval. A linear model is used to achieve the following: position is obtained by adding the velocity to the previous position and multiplying by the time interval; width and height are updated linearly according to their rate of change. Simultaneously, the prediction covariance matrix is ​​updated as follows: ,in, To predict the covariance matrix; This is the process noise covariance matrix. In this embodiment, the process noise covariance matrix is ​​specifically set according to the UAV's flight speed and the camera's frame rate. When the UAV's flight speed is high or there is slight jitter, the position-related noise parameter is appropriately increased to better model the motion uncertainty of the casing in the image; when the flight state is stable, the noise intensity is reduced to improve prediction accuracy. Through the above linear state transition operation, the predicted state vector of each target in the current frame is obtained for subsequent data association and matching.

[0057] In some embodiments, the predicted state vector is extracted to reconstruct the bounding box geometry information of the valve hall device, thereby obtaining each predicted box. Specifically, this involves: after obtaining the predicted state vector... Then, components related to geometric information are extracted: predicting the center location. Predicted width Predicted height Furthermore, the geometric information in the center form is converted into a bounding box form for matching with the detection box in the current frame. The coordinates of the top-left and bottom-right corners of the predicted box can be reconstructed as follows: , , , This yields the predicted bounding box for each tracker in the current frame, i.e., the predicted box. The predicted box serves as input to the motion association stage, performing Mahalanobis distance or intersection-union ratio calculations with the detection boxes output by the detection module, and participating in subsequent multi-level matching mechanisms, including appearance feature matching and local structure verification. Through the aforementioned state vector construction, linear prediction, and bounding box reconstruction process, continuous extrapolation from historical detection box information to the predicted box in the current frame is achieved. This provides a fundamental motion model support for stable tracking of valve hall equipment targets under partial occlusion, short-term missed detection, and complex backgrounds, thereby ensuring the continuity and consistency of the trajectory.

[0058] Through the above steps, precise zero-bias compensation for UAV motion data, fuzzy recognition and enhanced restoration of valve hall equipment video, and collaborative optimization processing of multi-target detection and multi-level matching tracking were achieved. A complete perception link from data preprocessing and target detection to continuous trajectory generation was constructed, which effectively improved the detection stability and tracking continuity of valve hall equipment targets in complex environments, and provided reliable visual and motion status support for UAVs to carry out high-precision and autonomous inspection in valve hall environments.

[0059] Step S103: Calculate the inspection attitude error vector of the UAV relative to the valve hall equipment based on the continuous trajectory information and the UAV's degree of freedom pose, and generate a motor drive signal based on the inspection attitude error vector to realize the UAV's autonomous inspection of the valve hall equipment. The UAV's degree of freedom pose is determined by solving a joint optimization objective function constructed based on preset camera feature points and the motion data.

[0060] In some embodiments, calculating the inspection attitude error vector of the UAV relative to the valve hall equipment based on the continuous trajectory information and the UAV's degree of freedom pose includes: extracting the reference position vector and reference velocity vector of the continuous trajectory information in the world coordinate system; obtaining the position error vector and velocity error vector respectively by performing difference operations on the reference position vector and the target position vector, and the reference velocity vector and the target velocity vector; performing relative rotation calculation on the reference rotation matrix and the attitude rotation matrix to obtain the attitude deviation matrix, and performing attitude angle decomposition on the attitude deviation matrix to obtain the attitude error vector, wherein the target position vector, the target velocity vector, and the attitude rotation matrix are determined based on the UAV's degree of freedom pose, and the reference rotation matrix is ​​determined based on the reference velocity vector; and combining the position error vector, the velocity error vector, and the attitude error vector to obtain the inspection attitude error vector of the UAV relative to the valve hall equipment.

[0061] In some embodiments, the reference position vector and reference velocity vector of the continuous trajectory information in the world coordinate system are extracted. Specifically, the continuous trajectory information output by the target tracking module is the time-continuous position and velocity estimation result of the valve hall device target in the image sequence. To facilitate flight control, the trajectory points in the image coordinate system are first back-projected to the world coordinate system using the camera extrinsic parameter matrix and the current UAV attitude, thus obtaining the spatial trajectory of the valve hall device in the world coordinate system. Let the first... The three-dimensional coordinates of the timekeeping device center are: ,in, The reference position vector; These represent the three-dimensional coordinate components of the device in the world coordinate system. The reference velocity vector is obtained from the position difference between adjacent time points: ; in, Used as the reference velocity vector; For time intervals; subscript This represents the previous moment. To enhance smoothness, a sliding window averaging or Kalman filtering can be used to smoothly estimate the velocity. The aforementioned reference position vector and reference velocity vector are used as inputs to the flight control module as the target motion state that the UAV intends to follow.

[0062] In some embodiments, position error vectors and velocity error vectors are obtained by performing difference operations on the reference position vector and the target position vector, and on the reference velocity vector and the target velocity vector, respectively. Specifically, the attitude estimation and flight control module outputs the UAV's current attitude and velocity estimation results in the world coordinate system through a VIO fusion algorithm, defined as: ; in, The target position vector (the current position of the UAV); Let the target velocity vector be (the current velocity of the UAV). Based on the above definition, construct the position error vector: Construct the velocity error vector: ,in, This indicates the spatial positional deviation between the drone and the valve hall equipment; This represents the velocity deviation between the drone and the surface motion trend of the equipment. This error is used in subsequent control law calculations.

[0063] In some embodiments, a relative rotation calculation is performed between the reference rotation matrix and the attitude rotation matrix to obtain an attitude deviation matrix, and the attitude deviation matrix is ​​decomposed into attitude angles to obtain an attitude error vector. The target position vector, the target velocity vector, and the attitude rotation matrix are determined based on the UAV's degrees of freedom pose, and the reference rotation matrix is ​​determined based on the reference velocity vector. Specifically, the VIO fusion module outputs the current attitude rotation matrix of the UAV. This represents the rotational relationship between the UAV's body coordinate system and the world coordinate system. To achieve stable tracking and inspection of equipment surfaces within the valve hall environment, a reference attitude needs to be constructed. First, the desired motion direction unit vector is determined based on the reference velocity vector: ,in, The reference heading direction is the unit vector; This represents the vector L2 norm. Taking this direction as the forward axis of the organism, and combining it with the gravity direction vector in the world coordinate system... Construct an orthogonal basis and generate the reference rotation matrix: This matrix represents the ideal attitude of the UAV, ensuring its nose points in the desired direction of movement along the surface of the device, while maintaining reasonable pitch and roll. The attitude deviation matrix is ​​calculated as follows: ,in, This is the attitude deviation matrix; superscript This represents the matrix transpose. Further decomposition of the attitude deviation matrix into Euler angle errors: ,in, This refers to the roll angle error; This refers to the pitch angle error; This represents the yaw angle error. The attitude error vector described above describes the degree of deviation of the current UAV attitude from the desired device surface following attitude.

[0064] In some embodiments, the position error vector, the velocity error vector, and the attitude error vector are combined to obtain the inspection attitude error vector of the UAV relative to the valve hall equipment. Specifically, the position error, velocity error, and attitude error are uniformly represented as a control input error vector. ,in, This is the inspection attitude error vector; This refers to the three-dimensional position error; This refers to the three-dimensional velocity error. This represents the three-dimensional attitude angle error. This error vector serves as the input to the PID control law. Control the output vector: This is further mapped to a motor PWM control signal to achieve attitude and altitude adjustment of the UAV. Through the above steps, the process of constructing the inspection attitude error vector from continuous trajectory information is completed, so that the vision-inertial fusion estimation result and the flight control law form a closed loop, thereby realizing the UAV's accurate, stable, and autonomous flight control of the valve hall equipment in an environment without a global positioning system.

[0065] It should be noted that the UAV's degree-of-freedom pose is estimated in real time using a vision-inertial fusion algorithm. The UAV's degree-of-freedom pose includes three-dimensional spatial position and three-dimensional attitude angle information, specifically solved by fusing data from a monocular camera and an inertial measurement unit (IMU). The UAV's degree-of-freedom pose is defined as: ,in, This represents the three-dimensional position vector of the UAV in the world coordinate system; The rotation matrix represents the rotation of the UAV relative to the world coordinate system; rotation matrix It can be further decomposed into three attitude angles: roll angle, pitch angle, and yaw angle. In this embodiment, the pose estimation module includes: a monocular industrial camera for acquiring video image sequences of valve hall equipment inspection; and an inertial measurement unit (IMU) for acquiring angular velocity and linear acceleration data. The IMU outputs an angular velocity vector and a linear acceleration vector at each moment. Between two image frames, the attitude is first predicted briefly using the IMU angular velocity. The attitude update can be expressed as: ,in, To predict attitude; The posture at the previous moment; For time intervals; It is the antisymmetric matrix form of angular velocity; This involves matrix exponential mapping. While this step provides high-frequency pose prediction, it suffers from long-term drift errors. To eliminate accumulated IMU errors, this technique employs visual feature tracking and reprojection error optimization for pose correction. Feature points are extracted and matched in adjacent image frames to establish a reprojection relationship between 3D spatial points and 2D image points. ,in, Image coordinates; This is the camera intrinsic parameter matrix; The coordinates of the spatial feature points; It is a rotation matrix; Let be the translation vector. By minimizing the reprojection error: The corrected attitude rotation matrix is ​​obtained through optimization. Using the acceleration measured by the IMU, velocity and position updates are obtained by integration after gravity compensation. Due to the scale uncertainty of the monocular camera, this embodiment uses a sliding window nonlinear optimization method to jointly construct the optimization objective function by combining visual observations from multiple historical frames with IMU pre-integration results. The optimal position estimate is obtained by solving the nonlinear least squares optimization problem. Optimal attitude estimation This achieves scale observability and drift suppression. After optimization, the UAV's pose at the current moment can be expressed as: ; in, Indicates attitude (three degrees of freedom: roll, pitch, and yaw); Indicates position (three degrees of freedom: Therefore, this embodiment obtains the continuous six-degree-of-freedom pose estimation result of the UAV in the world coordinate system by optimizing the fusion of visual and inertial information.

[0066] In some embodiments, generating motor drive signals based on the inspection attitude error vector to achieve autonomous inspection of valve hall equipment by the UAV includes: performing proportional, integral, and differential operations on the inspection attitude error vector to obtain corresponding proportional, integral, and differential operation results; weighting and summing the proportional, integral, and differential operation results according to a preset proportional gain matrix, integral gain matrix, and differential gain matrix to obtain an attitude correction control quantity; and mapping the attitude correction control quantity to pulse width modulation signals corresponding to each motor in the UAV based on a preset attitude allocation matrix, so as to achieve autonomous inspection flight of valve hall equipment by the UAV based on the pulse width modulation signals.

[0067] In some embodiments, the inspection attitude error vector is subjected to proportional, integral, and differential operations to obtain corresponding proportional, integral, and differential operation results, respectively. Specifically, after completing the vision-inertial fusion optimization, the real-time state estimate of the UAV in the world coordinate system is obtained, including: position estimation vector: Velocity estimation vector: Attitude angle vector: ,in, For the three-dimensional position of the drone; For linear velocity components; These are the roll angle, pitch angle, and yaw angle, respectively. The reference state from the target tracking module is: Reference position vector: Reference velocity vector: Based on this, construct the inspection attitude error vector: ,in, , Let and represent the position error vector and velocity error vector, respectively. For this error vector, the control module performs proportional (P), integral (I), and derivative (D) operations: Proportional operation result: , representing the current instantaneous position error. Integration result: ,in, The integral term is the time variable; it is used to eliminate steady-state error. Differential calculation result: In this embodiment, the velocity error is essentially an approximation of the time derivative of the position error; therefore, the differential calculation result can be expressed as: This refers to the rate of change of the current position error. The proportional calculation results are obtained through the above three operations. Integral results and the result of differential operation This provides a foundation for subsequent attitude correction control calculations.

[0068] In some embodiments, the proportional gain result, the integral gain result, and the differential gain result are weighted and summed according to a preset proportional gain matrix, integral gain matrix, and differential gain matrix to obtain the attitude correction control quantity. Specifically, to adapt to the three-dimensional space control characteristics of the UAV, the system presets a proportional gain matrix. Integral gain matrix and differential gain matrix Its form is a diagonal matrix: ; in, The gain is controlled by the position ratio. For integral control gain; This is the differential control gain. The formula for calculating the attitude correction control quantity is: ,in, This is the three-dimensional control output vector; each component corresponds to the correction control quantity for the three spatial axes of the UAV. At the attitude control level, the control vector is further converted into attitude correction quantities: ,in, This is the roll angle correction amount; This is the pitch angle correction amount; This is the yaw angle correction amount; This is the altitude correction value. This attitude correction control value is the attitude and altitude adjustment command that the UAV should execute in the next moment.

[0069] In some embodiments, the attitude correction control quantity is mapped to pulse width modulation signals corresponding to each motor in the UAV based on a preset attitude allocation matrix, so as to realize the autonomous inspection of the valve hall equipment by the UAV based on the pulse width modulation signals. Specifically, the UAV is a quadcopter structure, and its attitude control relies on the thrust differential of the four motors. Let... Total thrust; This is the roll control variable; For pitch control; This is the yaw control variable. The attitude assignment matrix is ​​defined as: ; in, For the first The pulse width modulation signals for each motor; the first matrix is ​​the attitude assignment matrix; the second column vector represents the total thrust and attitude control inputs. In actual implementation, to ensure stable hovering, the basic PWM values ​​of the motors are set as follows: ,in, This is the reference PWM value during hovering; This introduces an increment for attitude control. Through the above mapping relationship, the attitude correction control quantity is converted into a specific motor drive signal, which directly acts on the motor actuator, enabling the UAV to autonomously inspect the valve hall equipment.

[0070] Through the above steps, a complete control link is realized, from continuous trajectory information to the construction of inspection attitude error, and then to PID closed-loop control and motor PWM drive mapping. This enables the vision-inertial fusion attitude estimation results to directly participate in flight control decisions, forming an integrated closed-loop control system of perception-decision-execution. This significantly improves the attitude stability, trajectory following accuracy and anti-disturbance capability of the UAV during autonomous inspection in the valve hall environment.

[0071] like Figure 4 As shown, based on the above method embodiments, corresponding apparatus embodiments are provided; An embodiment of the present invention provides a schematic diagram of the structure of an unmanned aerial vehicle (UAV) autonomous inspection system for valve hall equipment, including: an acquisition module 100, a solution module 200, and a control module 300; The acquisition module 100 is used to acquire motion data of the UAV during the inspection flight and the video sequence of the valve hall equipment collected. The solution module 200 is used to input each image frame of the video sequence of the valve hall equipment into a preset target detection model to obtain several detection boxes of the valve hall equipment. Based on each predicted box and each of the detection boxes, a comprehensive matching cost function is determined. The Hungarian algorithm is used to solve the comprehensive matching cost function with the goal of minimizing the total cost to obtain continuous trajectory information on the surface of the valve hall equipment. Each predicted box is obtained by extrapolating the historical detection boxes of the valve hall equipment through the Kalman filter algorithm. The valve hall equipment includes a converter valve, a bushing, and a busbar. The control module 300 is used to calculate the inspection attitude error vector of the UAV relative to the valve hall equipment based on the continuous trajectory information and the UAV's degree of freedom pose, and to generate a motor drive signal based on the inspection attitude error vector to realize the UAV's autonomous inspection of the valve hall equipment. The UAV's degree of freedom pose is determined by solving a joint optimization objective function constructed based on preset camera feature points and the motion data.

[0072] It is understood that the above-described device embodiments correspond to the method embodiments of the present invention, and can realize the unmanned aerial vehicle (UAV) autonomous inspection method for valve hall equipment provided by any of the above-described method embodiments of the present invention. For a more detailed workflow and principle of this system, please refer to the relevant descriptions of the above methods.

[0073] It should be noted that the device embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can specifically be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0074] Based on the above-described embodiment of the UAV autonomous inspection method for valve hall equipment, another embodiment of the present invention provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the UAV autonomous inspection method for valve hall equipment according to any embodiment of the present invention.

[0075] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the terminal device.

[0076] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0077] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0078] Based on the above-described method embodiments, another embodiment of the present invention provides a computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the unmanned aerial vehicle autonomous inspection method for valve hall equipment described in any of the above-described method embodiments of the present invention.

[0079] The modules / units integrated in the device / terminal equipment, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0080] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A method for autonomous inspection of valve hall equipment using unmanned aerial vehicles (UAVs), characterized in that, include: Acquire motion data and video sequences of valve hall equipment during the inspection flight of the UAV; Each image frame of the video sequence of the valve hall equipment is input into a preset target detection model to obtain several detection boxes of the valve hall equipment. Based on each predicted box and each of the detection boxes, a comprehensive matching cost function is determined. The Hungarian algorithm is used to solve the comprehensive matching cost function with the goal of minimizing the total cost to obtain continuous trajectory information on the surface of the valve hall equipment. Each predicted box is obtained by extrapolating the historical detection boxes of the valve hall equipment through the Kalman filter algorithm. The valve hall equipment includes a converter valve, a bushing, and a busbar. Based on the continuous trajectory information and the UAV's degree of freedom pose, the inspection attitude error vector of the UAV relative to the valve hall equipment is calculated, and a motor drive signal is generated based on the inspection attitude error vector to realize the UAV's autonomous inspection of the valve hall equipment. The UAV's degree of freedom pose is determined by solving a joint optimization objective function constructed based on preset camera feature points and the motion data.

2. The unmanned aerial vehicle (UAV) autonomous inspection method for valve hall equipment as described in claim 1, characterized in that, The determination of the comprehensive matching cost function based on each predicted bounding box and each detected bounding box includes: Calculate the Mahalanobis distance between each of the predicted boxes and each of the detected boxes, and determine the motion cost based on each of the Mahalanobis distances; Appearance features are extracted from each of the predicted boxes and each of the detected boxes to obtain first appearance features and second appearance features, respectively. The cosine distance between each of the first appearance features and each of the second appearance features is calculated to determine the appearance cost based on each of the cosine distances. Feature point matching is performed on each of the predicted boxes and each of the detected boxes to obtain matching point pairs. The structural similarity score is calculated based on the spatial distribution consistency of each matching point pair, and the structural cost is determined based on the structural similarity score. The motion cost, appearance cost, and structural cost are weighted and fused according to preset weights to obtain the comprehensive matching cost function between each predicted box and each detected box.

3. The unmanned aerial vehicle (UAV) autonomous inspection method for valve hall equipment as described in claim 1, characterized in that, The step of inputting each image frame of the video sequence of the valve hall equipment into a preset target detection model to obtain several detection boxes of the valve hall equipment includes: Each image frame of the video sequence of the valve hall equipment is input into a preset target detection model, and the backbone network in the target detection model is used to extract features from each image frame at several levels to obtain a backbone feature map for enhancing small-scale equipment components in the valve hall equipment, wherein the small-scale equipment components include equalizing rings and bolts. The backbone feature map is jointly weighted and calculated based on the channel attention mechanism and the spatial attention mechanism to obtain the target area enhancement feature map for enhancing the converter valve and bushing in the valve hall equipment. Based on the feature pyramid structure in the target detection model, the enhanced feature map of the target region is subjected to multi-scale fusion processing to obtain a multi-scale fused feature map. Boundary box regression calculation is performed on the multi-scale fused feature map to obtain several detection boxes containing category information and location parameters.

4. The unmanned aerial vehicle (UAV) autonomous inspection method for valve hall equipment as described in claim 1, characterized in that, The acquisition of motion data of the UAV during the inspection flight and the collection of video sequences of valve hall equipment include: The initial motion data is calibrated with zero bias to obtain the motion data; Multi-scale wavelet decomposition is performed on the acquired initial valve hall equipment video sequence to calculate the high-frequency subband energy of each image frame in the initial valve hall equipment video sequence and compare it with a preset energy threshold to obtain the comparison result, and the blurred region mask image is determined based on the comparison result. The blurred region mask image is input into a preset deblurring neural network model, so that the blurred region mask image is processed by the deformation convolution and channel attention mechanism in the deblurring neural network model to extract features and output a deblurred and enhanced image sequence. Calculate the perspective transformation matrix between each image frame in the deblurred and enhanced image sequence, and perform inverse perspective transformation processing on each image frame based on each perspective transformation matrix to obtain the valve hall equipment video sequence.

5. The unmanned aerial vehicle (UAV) autonomous inspection method for valve hall equipment as described in claim 1, characterized in that, Each of the predicted frames is obtained by extrapolating the historical detection frames of the valve hall equipment using the Kalman filter algorithm, including: Extract the center position coordinates, width, height, and corresponding velocity components of each historical detection box, and combine the center position coordinates, width, height, and velocity components to form an initial state vector; The state prediction formula based on the Kalman filter algorithm is used to perform a linear state transition operation on the initial state vector to obtain the predicted state vector. The predicted state vector is extracted to reconstruct the boundary box geometry of the valve hall equipment, thus obtaining each predicted box.

6. The unmanned aerial vehicle (UAV) autonomous inspection method for valve hall equipment as described in claim 1, characterized in that, The calculation of the inspection attitude error vector of the UAV relative to the valve hall equipment based on the continuous trajectory information and the UAV's degrees of freedom pose includes: Extract the reference position vector and reference velocity vector of the continuous trajectory information in the world coordinate system; By performing difference operations on the reference position vector and the target position vector, and on the reference velocity vector and the target velocity vector, respectively, the position error vector and the velocity error vector are obtained. The attitude deviation matrix is ​​obtained by performing relative rotation calculation between the reference rotation matrix and the attitude rotation matrix, and the attitude deviation matrix is ​​decomposed into attitude angles to obtain the attitude error vector. The target position vector, the target velocity vector, and the attitude rotation matrix are determined based on the UAV's degree of freedom pose, and the reference rotation matrix is ​​determined based on the reference velocity vector. The position error vector, the velocity error vector, and the attitude error vector are combined to obtain the inspection attitude error vector of the UAV relative to the valve hall equipment.

7. The unmanned aerial vehicle (UAV) autonomous inspection method for valve hall equipment as described in claim 1, characterized in that, The process of generating motor drive signals based on the inspection attitude error vector to achieve autonomous inspection of valve hall equipment by the UAV includes: The inspection attitude error vector is subjected to proportional, integral, and differential operations to obtain the corresponding proportional, integral, and differential operation results, respectively. The proportional calculation result, the integral calculation result, and the differential calculation result are weighted and summed according to a preset proportional gain matrix, integral gain matrix, and differential gain matrix to obtain the attitude correction control quantity; Based on a preset attitude assignment matrix, the attitude correction control quantity is mapped to the pulse width modulation signal corresponding to each motor in the UAV, so as to realize the UAV's autonomous inspection of the valve hall equipment based on the pulse width modulation signal.

8. An unmanned aerial vehicle (UAV) autonomous inspection system for valve hall equipment, characterized in that, The system includes: an acquisition module, a solution module, and a control module; The acquisition module is used to acquire motion data of the UAV during the inspection flight and the video sequence of the valve hall equipment collected. The solution module is used to input each image frame of the video sequence of the valve hall equipment into a preset target detection model to obtain several detection boxes of the valve hall equipment. Based on each predicted box and each of the detection boxes, a comprehensive matching cost function is determined. The Hungarian algorithm is used to solve the comprehensive matching cost function with the goal of minimizing the total cost to obtain continuous trajectory information on the surface of the valve hall equipment. Each predicted box is obtained by extrapolating the historical detection boxes of the valve hall equipment through the Kalman filter algorithm. The valve hall equipment includes a converter valve, a bushing, and a busbar. The control module is used to calculate the inspection attitude error vector of the UAV relative to the valve hall equipment based on the continuous trajectory information and the UAV's degree of freedom pose, and to generate a motor drive signal based on the inspection attitude error vector to realize the UAV's autonomous inspection of the valve hall equipment. The UAV's degree of freedom pose is determined by solving a joint optimization objective function constructed based on preset camera feature points and the motion data.

9. A terminal device, characterized in that, The device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements a method for autonomous inspection of valve hall equipment by unmanned aerial vehicles as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, include: A stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform an unmanned aerial vehicle (UAV) autonomous inspection method for valve hall equipment as described in any one of claims 1-7.

Images