Unmanned aerial vehicle intelligent maintenance recognition system based on image detection

Abstract

The present application relates to the technical field of image detection, in particular to an unmanned aerial vehicle intelligent maintenance identification system based on image detection, which comprises: an unmanned aerial vehicle image acquisition module, which receives a uniform rotation test instruction to drive a holder to rotate, starts an external industrial camera to record a continuous frame video sequence, and generates a set of static reference pixel coordinates; extracts pixel coordinate points of a holder surface area in a first frame image, matches pixel coordinate points of corresponding positions in a second frame image for position binding and pairing, and generates a set of holder test pixel coordinates. In the present application, a frequency domain complex modulus value, an amplitude-frequency curve, a theoretical fault characteristic frequency interval and a target frequency domain harmonic peak value are extracted, the holder bearing jamming is converted from naked eye observation or single point threshold judgment to harmonic peak value judgment in the characteristic frequency interval, and the identifiable nature of micro jamming, periodic blockage and intermittent rotation abnormalities can be enhanced.

Description

Technical Field

[0001] This invention relates to the field of image detection technology, and in particular to an intelligent maintenance and recognition system for unmanned aerial vehicles based on image detection. Background Technology

[0002] Image detection technology uses digital images, video frame sequences, and visual sensor data as processing objects. By analyzing the pixel distribution, grayscale changes, color features, texture structure, edge contours, target morphology, spatial location, and temporal changes in images, it can achieve target area localization, structural state recognition, defect feature extraction, and abnormal behavior judgment.

[0003] Existing image detection technologies typically analyze pixel distribution, grayscale variations, color features, texture structure, edge contours, target morphology, spatial location, and temporal changes. They primarily focus on target region localization, structural state recognition, defect feature extraction, and abnormal behavior identification. In practice, these technologies are better suited for identifying objects with obvious spatial features such as cracks, deformations, occlusions, surface contamination, and notches. However, when the detected object is facing a stuck bearing on a drone gimbal, the abnormal behavior often manifests as subtle angular velocity fluctuations during continuous rotation. Relying solely on image appearance features or changes in adjacent frames can easily conflate camera shake, overall drone attitude drift, and normal gimbal rotation displacement into a single type of image change, making it difficult to pinpoint the source of the fault. Therefore, improvements are needed. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and propose an image detection-based intelligent maintenance and recognition system for unmanned aerial vehicles.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: An image detection-based intelligent maintenance and recognition system for unmanned aerial vehicles includes:

[0006] The UAV image acquisition module receives a uniform rotation test command to drive the gimbal to rotate, starts an external industrial camera to record a continuous frame video sequence, and generates a static reference pixel coordinate set; it extracts the pixel coordinate points of the gimbal surface area in the first frame image, matches the corresponding pixel coordinate points in the second frame image for position binding and pairing, and generates a gimbal test pixel coordinate set.

[0007] The global motion decoupling module calculates the mean square error of pixel offsets in two frames of images within the static reference pixel coordinate set to construct a parameter matrix. It then uses this parameter matrix to calculate the pixel coordinates of the gimbal surface region in the first frame of the gimbal test pixel coordinate set, obtaining the theoretical mapping coordinates of the second frame. Finally, it calculates the difference between the theoretical mapping coordinates of the second frame and the corresponding pixel coordinates of the gimbal surface region in the first frame to generate a theoretical displacement vector field. Finally, it extracts the pixel coordinates of the gimbal surface region corresponding to the two frames of images within the gimbal test pixel coordinate set and calculates them to generate the actual displacement vector field.

[0008] Preferably, the system further includes:

[0009] The rotational motion residual extraction module calls the actual displacement vector data of each pixel in the actual displacement vector field and the corresponding theoretical displacement vector data in the theoretical displacement vector field to obtain the net displacement vector field; it extracts the pixel data of each pixel in the net displacement vector field and performs coordinate operations with the pixel coordinates of the gimbal surface area of ​​the first frame image to generate decoupled mapping pixel coordinates; it performs matrix parameter estimation based on the pixel coordinates of the gimbal surface area of ​​the first frame image and the decoupled mapping pixel coordinates to obtain the homography matrix; it performs matrix decomposition on the homography matrix in combination with the intrinsic parameter matrix of the external industrial camera to obtain the rotation matrix in three-dimensional space, and converts the rotation matrix into the instantaneous actual angular velocity value of the gimbal; it arranges the instantaneous actual angular velocity values ​​of the gimbal corresponding to multiple consecutive adjacent images covering the complete gimbal test rotation cycle in timestamp order to construct a one-dimensional time series sequence; it performs subtraction operations on the one-dimensional time series sequence according to the uniform rotation test command to generate a residual time series signal sequence;

[0010] The bearing jamming identification module extracts the amplitude of each data point within the residual time-series signal sequence to obtain the complex modulus value in the frequency domain; arranges the complex modulus values ​​in the frequency domain and plots the corresponding amplitude-frequency curve; combines the uniform rotation test command to calculate the theoretical fault characteristic frequency range of the gimbal bearing; locates the amplitude pole values ​​falling within the theoretical fault characteristic frequency range in the amplitude-frequency curve to generate the target frequency domain harmonic peak value; compares the relationship between the target frequency domain harmonic peak value and the preset gimbal bearing jamming alarm threshold value to generate a jamming anomaly diagnosis result.

[0011] Preferably, the steps for obtaining the static reference pixel coordinate set are as follows:

[0012] Receive the uniform rotation test command, read the rotation start / stop flag and rotation direction flag in the uniform rotation test command, transmit the rotation start / stop flag to the gimbal driver end, transmit the rotation direction flag to the gimbal attitude control end, trigger the recording port of the external industrial camera after the gimbal enters the uniform rotation state, record the image frame sequence, frame timestamp, frame number, and image pixel coordinate range during the gimbal rotation process according to the adjacent frame continuous storage rule, and generate a continuous frame video sequence.

[0013] Based on the continuous frame video sequence, the first and second frames of two adjacent frames are selected. The pixel coordinates of the fuselage shell area boundary in the first frame are located, and the pixel coordinates of the landing gear area structure in the first frame are located. According to the frame number correspondence, the pixel coordinates of the same fuselage shell area and the same landing gear area are retrieved in the second frame. The pixel coordinates of the first frame and the second frame are paired according to the area identifier, frame number, and coordinate index to generate a static reference pixel coordinate set.

[0014] Preferably, the step of obtaining the gimbal test pixel coordinate set is as follows:

[0015] Based on the continuous frame video sequence, the first and second frames of two adjacent frames are selected. The edge pixel coordinates, texture pixel coordinates, and corner pixel coordinates of the gimbal surface area in the first frame are located. The corresponding edge pixel coordinates, texture pixel coordinates, and corner pixel coordinates are retrieved in the second frame according to the gimbal surface area coordinate index. The gimbal surface area pixel coordinates of the first and second frames are bound and paired according to the coordinate index, frame number, and surface area identifier to generate a gimbal test pixel coordinate set.

[0016] Preferably, the steps for obtaining the parameter matrix are as follows:

[0017] Read the pixel coordinates of the fuselage area of ​​the first frame image, the pixel coordinates of the landing gear area of ​​the first frame image, the pixel coordinates of the fuselage area of ​​the second frame image, and the pixel coordinates of the landing gear area of ​​the second frame image within the static reference pixel coordinate set. Calculate the lateral and longitudinal coordinate offsets point by point according to the coordinate index. Accumulate the squared errors of the lateral and longitudinal coordinate offsets respectively. Filter the translation and rotation parameters corresponding to the minimum value of the accumulated squared error. Arrange the translation and rotation parameters according to the coordinate transformation order to construct a parameter matrix.

[0018] Preferably, the steps for obtaining the actual displacement vector field are as follows:

[0019] Read the pixel coordinates of the gimbal surface region of the first frame image in the gimbal test pixel coordinate set, arrange the pixel coordinates of the gimbal surface region of the first frame image according to the horizontal coordinate, vertical coordinate, and coordinate index, apply the parameter matrix item by item to the pixel coordinates of the gimbal surface region of the first frame image to obtain the theoretical mapping coordinates of the second frame, subtract the horizontal coordinates of the pixel coordinates of the gimbal surface region of the first frame image from the horizontal coordinates of the theoretical mapping coordinates of the second frame image, and subtract the vertical coordinates of the pixel coordinates of the gimbal surface region of the first frame image from the vertical coordinates of the theoretical mapping coordinates of the second frame image to generate the theoretical displacement vector field;

[0020] Read the gimbal surface area pixel coordinates of the first frame image and the second frame image within the gimbal test pixel coordinate set. Bind the gimbal surface area pixel coordinates of the first frame image and the second frame image point by point according to the same coordinate index. Subtract the horizontal coordinate of the gimbal surface area pixel coordinates of the first frame image from the horizontal coordinate of the second frame image. Subtract the vertical coordinate of the gimbal surface area pixel coordinates of the first frame image from the vertical coordinate of the second frame image. Record the actual horizontal displacement component and the actual vertical displacement component according to the coordinate index to generate the actual displacement vector field.

[0021] Preferably, the steps for obtaining the net displacement vector field are as follows:

[0022] The actual displacement vector data of each pixel in the actual displacement vector field is called, and the theoretical displacement vector data corresponding to the same coordinate index in the theoretical displacement vector field is read. The horizontal actual displacement component, the vertical actual displacement component, the horizontal theoretical displacement component, and the vertical theoretical displacement component of the theoretical displacement vector data are checked item by item according to the pixel coordinate index. The horizontal theoretical displacement component is subtracted from the horizontal actual displacement component, and the vertical theoretical displacement component is subtracted from the vertical actual displacement component. The horizontal net displacement component and the vertical net displacement component are recorded according to the pixel coordinate index to obtain the net displacement vector field.

[0023] Preferably, the step of obtaining the residual time-series signal sequence is as follows:

[0024] The intrinsic parameter matrix of the external industrial camera is invoked. The lateral and longitudinal net displacement components are read according to the pixel coordinate index in the net displacement vector field. The lateral and longitudinal coordinates of the pixel coordinates in the gimbal surface area of ​​the first frame image are read. The lateral coordinates are added to the lateral net displacement components, and the longitudinal coordinates are added to the longitudinal net displacement components. The coordinates are recorded as decoupled mapping pixel coordinates according to the coordinate index. Homography parameters are estimated using the pixel coordinates in the gimbal surface area of ​​the first frame image and the decoupled mapping pixel coordinates of the corresponding coordinate index to obtain the homography matrix. The homography matrix is ​​decomposed in combination with the intrinsic parameter matrix of the external industrial camera to obtain the rotation matrix in three-dimensional space. The rotation matrix in three-dimensional space is converted into the instantaneous actual angular velocity value of the gimbal according to the frame timestamp interval of two adjacent frames. The instantaneous actual angular velocity values ​​of the gimbal corresponding to multiple consecutive adjacent frames covering the complete gimbal test rotation cycle are arranged in the order of timestamps to construct a one-dimensional time series.

[0025] Extract the theoretical angular velocity setting value from the uniform rotation test command, write the theoretical angular velocity setting value into the motion baseline of the same length according to the timestamp order in the one-dimensional time sequence, read the instantaneous actual angular velocity value of the gimbal in the one-dimensional time sequence point by point, subtract the theoretical angular velocity setting value corresponding to the same timestamp in the motion baseline from the instantaneous actual angular velocity value of the gimbal corresponding to each timestamp, record the angular velocity residual value according to the timestamp order, and generate the residual time sequence signal sequence.

[0026] Preferably, the steps for obtaining the diagnostic results of the lag anomaly are as follows:

[0027] The amplitude values ​​of each data point in the residual time-series signal sequence are read in the order of timestamps. The sampling number, sampling timestamp, and angular velocity residual value corresponding to each data point amplitude are recorded. The angular velocity residual values ​​are arranged into a sequence of amplitude values ​​to be converted according to the sampling number. Frequency component decomposition operation is performed on the sequence of amplitude values ​​to be converted. The complex real part and complex imaginary part under each frequency index are recorded respectively. The frequency domain complex modulus value corresponding to each frequency index is calculated based on the complex real part and complex imaginary part. The frequency domain complex modulus values ​​are arranged in ascending order of frequency. The frequency index is used as the horizontal identifier and the frequency domain complex modulus value is used as the vertical identifier to generate an amplitude-frequency curve.

[0028] Extract the theoretical angular velocity setting value from the uniform rotation test command, read the rotation correspondence identifier and harmonic order identifier of the gimbal bearing, convert the theoretical angular velocity setting value into the reference rotation frequency of the gimbal bearing, expand the reference rotation frequency according to the rotation correspondence identifier and harmonic order identifier to form the theoretical fault characteristic frequency range of the gimbal bearing, retrieve the frequency domain complex modulus value corresponding to each frequency index in the amplitude-frequency curve point by point, filter the frequency domain complex modulus values ​​that fall within the theoretical fault characteristic frequency range, compare the frequency domain complex modulus value change relationship corresponding to adjacent frequency indices, locate the amplitude pole value within the theoretical fault characteristic frequency range, and generate the target frequency domain harmonic peak value.

[0029] The system calls a preset gimbal bearing jamming alarm threshold value, reads the peak amplitude, peak frequency index, and theoretical fault characteristic frequency range of the target frequency domain harmonic peak value, compares the peak amplitude with the preset gimbal bearing jamming alarm threshold value, outputs an abnormal state if the peak amplitude is greater than the preset gimbal bearing jamming alarm threshold value, and writes the abnormal state, peak frequency index, and theoretical fault characteristic frequency range of the peak value into the diagnostic record. If the peak amplitude is less than or equal to the preset gimbal bearing jamming alarm threshold value, outputs a non-triggered abnormal state, and writes the non-triggered abnormal state, peak frequency index, and theoretical fault characteristic frequency range of the peak value into the diagnostic record to generate a jamming abnormality diagnostic result.

[0030] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0031] In this invention, the gimbal's motion state is constrained by a uniform rotation test command, and a continuous frame video sequence is recorded by an external industrial camera. This expands the image detection object from a single static appearance to a motion process with a time-stamped sequence. The static reference pixel coordinate set serves as a global reference for the fuselage and landing gear areas, while the gimbal test pixel coordinate set serves as a local motion representation for the gimbal surface area. After separating the two types of pixel coordinate sets, the translational and rotational offsets during the overall shooting process can be separated from the gimbal's own motion. A progressively layered comparison relationship is formed between the parameter matrix, the theoretical mapping coordinate points of the second frame, the theoretical displacement vector field, and the actual displacement vector field, so that the actual motion of the gimbal surface area is no longer directly affected by changes in camera angle, minor body swaying, and take-off and landing. The reference offset of the frame is affected; further, the net displacement vector field is obtained from the actual displacement vector data and the theoretical displacement vector data, and then the decoupled mapping pixel coordinate points are generated by combining the pixel coordinate points of the gimbal surface area of ​​the first frame image, so that the two-dimensional pixel changes can be transformed into the rotation matrix and the instantaneous actual angular velocity value of the gimbal in three-dimensional space. The one-dimensional time sequence and the residual time sequence signal sequence extend the single frame detection results into the dynamic deviation record within the complete rotation cycle; on this basis, the frequency domain complex modulus, amplitude-frequency curve, theoretical fault characteristic frequency range and target frequency domain harmonic peak value are extracted, and the gimbal bearing jamming is transformed from visual observation or single-point threshold judgment to harmonic peak judgment within the characteristic frequency range, which can enhance the identifiability of minor jamming, periodic jamming and intermittent rotation anomalies. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the principle of the present invention;

[0033] Figure 2 This is a diagram of the residual time-series signal sequence;

[0034] Figure 3 This is an amplitude-frequency curve. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0036] Please see Figure 1-3 The present invention provides a technical solution: an image detection-based intelligent maintenance and recognition system for unmanned aerial vehicles (UAVs) includes:

[0037] The UAV image acquisition module receives a uniform rotation test command to drive the gimbal to rotate, starts an external industrial camera to record a continuous frame video sequence, and generates a static reference pixel coordinate set; it extracts the pixel coordinate points of the gimbal surface area in the first frame image, matches the corresponding pixel coordinate points in the second frame image for position binding and pairing, and generates a gimbal test pixel coordinate set.

[0038] The global motion decoupling module calculates the mean square error of pixel offsets between two frames of images within the stationary reference pixel coordinate set, constructing a parameter matrix. It then uses this parameter matrix to calculate the pixel coordinates of the gimbal surface region in the first frame of the gimbal test pixel coordinate set, obtaining the theoretical mapping coordinates for the second frame. Finally, it calculates the difference between the theoretical mapping coordinates of the second frame and the corresponding pixel coordinates of the gimbal surface region in the first frame, generating a theoretical displacement vector field. Finally, it extracts the pixel coordinates of the gimbal surface region corresponding to the two frames of the gimbal test pixel coordinate set and calculates them to generate the actual displacement vector field.

[0039] The rotational motion residual extraction module calls the actual displacement vector data of each pixel in the actual displacement vector field and the corresponding theoretical displacement vector data in the theoretical displacement vector field to obtain the net displacement vector field. It then extracts the pixel data from the net displacement vector field and performs coordinate calculations with the pixel coordinates of the gimbal surface area in the first frame image to generate decoupled mapping pixel coordinates. Based on the pixel coordinates of the gimbal surface area in the first frame image and the decoupled mapping pixel coordinates, it performs matrix parameter estimation to obtain the homography matrix. Combining this with the intrinsic parameter matrix of the external industrial camera, it performs matrix decomposition on the homography matrix to obtain the rotation matrix in three-dimensional space and converts the rotation matrix into the instantaneous actual angular velocity value of the gimbal. It then arranges the instantaneous actual angular velocity values ​​of the gimbal corresponding to multiple consecutive adjacent frames covering the complete gimbal test rotation cycle in timestamp order to construct a one-dimensional time series. Finally, according to the uniform rotation test command, it performs subtraction operations on the one-dimensional time series to generate a residual time series signal sequence.

[0040] The bearing jamming identification module extracts the amplitude of each data point within the residual time-series signal to obtain the complex modulus value in the frequency domain; arranges the complex modulus values ​​in the frequency domain and plots the corresponding amplitude-frequency curve; combines the uniform rotation test command to calculate the theoretical fault characteristic frequency range of the gimbal bearing; locates the amplitude pole values ​​falling within the theoretical fault characteristic frequency range in the amplitude-frequency curve to generate the target frequency domain harmonic peak value; compares the relationship between the target frequency domain harmonic peak value and the preset gimbal bearing jamming alarm threshold value to generate a jamming anomaly diagnosis result.

[0041] The steps to obtain the static reference pixel coordinate set are as follows:

[0042] Receive the uniform rotation test command, read the rotation start / stop flag and rotation direction flag in the uniform rotation test command, transmit the rotation start / stop flag to the gimbal driver end, transmit the rotation direction flag to the gimbal attitude control end, trigger the recording port of the external industrial camera after the gimbal enters the uniform rotation state, record the image frame sequence, frame timestamp, frame number, and image pixel coordinate range during the gimbal rotation process according to the adjacent frame continuous storage rule, and generate a continuous frame video sequence.

[0043] Based on the continuous frame video sequence, the first and second frames of two adjacent frames are selected. The pixel coordinates of the fuselage shell area boundary and the landing gear area structure in the first frame are located. According to the frame number correspondence, the pixel coordinates of the same fuselage shell area and the same landing gear area are retrieved in the second frame. The pixel coordinates of the first and second frames are paired according to the area identifier, frame number, and coordinate index to generate a static reference pixel coordinate set.

[0044] Specifically, upon receiving the uniform rotation test command, the system first parses the data structure within the command. This command is a JSON-formatted data packet containing a rotation start / stop flag, a rotation direction flag, and the target angular velocity value, for example, {"action": "start", "direction": "clockwise", "velocity_dps": 30.0}. The `action` field corresponds to the rotation start / stop flag, and the `direction` field corresponds to the rotation direction flag. The "start" in the `action` field is converted to the start command code 0xA1 and sent to the PTZ driver via the Controller Area Network (CAN) bus. The "clockwise" in the `direction` field is converted to the direction command code 0xC1. The command is sent to the gimbal attitude control terminal. After sending the command, a status confirmation loop is entered. The instantaneous actual angular velocity fed back by the gimbal encoder is continuously read at a frequency of 100Hz via the CAN bus. This velocity is compared with the target angular velocity value in the command (e.g., 30.0 degrees / second). A stable state judgment threshold is set at ±1.5% of the target angular velocity, that is, the actual angular velocity must be within the range of 29.55 degrees / second to 30.45 degrees / second. When the actual angular velocity remains within this range for 500 consecutive milliseconds (i.e., 50 consecutive samples), it is determined that the gimbal has entered a uniform rotation state, and then a rising edge 5V is output through the GPIO port. A TTL level signal is sent to the hardware trigger port of the external industrial camera to start camera recording. The camera shoots at a preset frame rate of 100 frames per second and a resolution of 1920x1080 pixels. To ensure data integrity, a circular buffer queue is used to manage the image data stream. When each frame of the image is stored in the storage medium, it is bound to a metadata record containing the current frame number, a high-precision timestamp from the start of the test, and the pixel coordinate range of the image (from the top left corner (0, 0) to the bottom right corner (1919, 1079)). All image frames with metadata in the entire test cycle are collected to generate a continuous frame video sequence.

[0045] Based on a continuous video sequence, adjacent first and second frames are iteratively selected sequentially for processing. The first frame is first input into a pre-trained Mask R-CNN model for instance segmentation. Its network structure consists of a ResNet-101 backbone for feature extraction, a Region Proposal Network (RPN), and a head network for classification, bounding box regression, and mask prediction. The model is trained on a dataset of manually labeled UAV images of the fuselage and landing gear regions using stochastic gradient descent with an initial learning rate of 0.002, momentum of 0.9, and 50 training epochs. After processing the first frame, the model outputs binary mask images of the fuselage and landing gear regions. For the fuselage mask, the Canny edge detection algorithm is applied to extract its boundary contours. The high and low thresholds are automatically determined using the Otsu method based on the image grayscale histogram; for example, a low threshold of 60 and a high threshold of 180 are calculated to obtain the boundary pixel coordinates. For the landing gear mask, its... Internally, an ORB feature detector is used to extract up to 150 corner points with the highest response values ​​as structural pixel coordinates of the landing gear area. Then, a pyramid-style Lucas-Kanade optical flow method is used to trace these pixels extracted from the first frame image to their corresponding positions in the second frame image. The optical flow search window size is set to 15x15 pixels, and the number of pyramid layers is set to 3. To eliminate erroneous matches, a forward-backward consistency check is performed, i.e., tracing from the first frame to the second frame and then from the second frame back to the first frame. If the Euclidean distance between the final position and the initial position is less than 1.0 pixel, the match is considered valid. Finally, all pixel coordinates of the first frame image that pass the consistency check and their corresponding pixel coordinates in the second frame image are paired and combined with their respective area identifiers (fuselage shell or landing gear), frame number, and a unique coordinate index to generate a static reference pixel coordinate set.

[0046] The steps to obtain the gimbal test pixel coordinate set are as follows:

[0047] Based on the continuous frame video sequence, the first and second frames of two adjacent frames are selected. The edge pixel coordinates, texture pixel coordinates, and corner pixel coordinates of the gimbal surface area in the first frame are located. The corresponding edge pixel coordinates, texture pixel coordinates, and corner pixel coordinates in the second frame are retrieved according to the gimbal surface area coordinate index. The gimbal surface area pixel coordinates of the first and second frames are bound and paired according to the coordinate index, frame number, and surface area identifier to generate a gimbal test pixel coordinate set.

[0048] Specifically, based on the continuous video frame sequence, the same two adjacent frames (the first frame and the second frame) are processed. The first frame is input into the previously trained Mask R-CNN model to extract the instance segmentation mask of the gimbal surface region. Within the obtained gimbal region mask, multiple feature detection algorithms are used to ensure the diversity and stability of feature points. First, the Harris corner detection algorithm is applied, and its corner response function is... ,in The image gradient covariance matrix is... An empirical constant was set to 0.04, retaining pixels with response values ​​greater than 1% of the global maximum response value as corner pixel coordinates. Next, the Canny edge detection algorithm was applied, with fixed high and low thresholds of 200 and 100 to extract clear edge pixel coordinates. Then, the SIFT algorithm was applied to detect scale- and rotation-invariant texture pixel coordinates, generating a 128-dimensional feature descriptor for each point. Next, the SIFT algorithm was also performed in the corresponding gimbal region of the second frame image (this region can be predicted using the mask position of the first frame and the known maximum rotational angular velocity) to extract feature points and their descriptors. Then, a FLANN-based matcher was used to match the SIFT descriptors of the two frames. To improve matching accuracy, the Lowe ratio test was used to filter the matching results. For each feature point from the first frame, the distance between its best and second-best match in the second frame was calculated. and Only when the conditions are met Only when the time is right will the matching pair be accepted. The ratio threshold of 0.7 here is an empirical value that balances recall and precision. For all valid matching pairs that pass the screening, the pixel coordinates of the gimbal surface area in the first frame image and the corresponding pixel coordinates of the gimbal surface area in the second frame image are paired according to the unique coordinate index, the current frame number being processed, and the "gimbal" surface area identifier to generate a gimbal test pixel coordinate set.

[0049] The steps to obtain the parameter matrix are as follows:

[0050] Read the pixel coordinates of the fuselage area, landing gear area, and other areas of the first frame image from the static reference pixel coordinate set. Calculate the lateral and longitudinal coordinate offsets point by point according to the coordinate index. Accumulate the squared errors of the lateral and longitudinal coordinate offsets respectively. Filter the translation and rotation parameters corresponding to the minimum value of the accumulated squared error. Arrange the translation and rotation parameters in the order of coordinate transformation to construct a parameter matrix.

[0051] Specifically, each pair of matching points contained in the static reference pixel coordinate set is read, and each pair of matching points is composed of pixel coordinates from the first frame image. Corresponding pixel coordinates of the second frame image The system is composed of a coordinate index (i) and a two-dimensional rigid body transformation model describing the global motion between two frames. This model contains three parameters to be solved: the lateral translation parameter, the horizontal translation parameter, and the coordinate index (i). Longitudinal translation parameters and rotation parameters For any point in the first frame image Its theoretically predicted coordinates in the second frame after this transformation Calculated in the following way:

[0052] ;

[0053] in, It is the rotation angle between two frames. These are translation vectors. To solve for these three parameters, we construct an objective function, which is the sum of squared Euclidean distances between the predicted and actual observed positions of all matched point pairs, i.e., the cumulative squared error. Specifically: ,in It is the total number of static reference pixel coordinate pairs. and The predicted coordinates of the i-th point are calculated based on the transformation parameters, and the Levenberg-Marquardt iterative optimization algorithm is used to find the optimal coordinates. The parameter combination that achieves the minimum state is given by setting the initial parameters of the algorithm as follows: , , The termination condition for the iteration is set to the error between two consecutive iterations. The optimal parameters are obtained when the change is less than the preset convergence threshold 1e-7, or when the number of iterations reaches the upper limit of 100. This convergence threshold is set based on a large amount of experimental data to achieve a balance between computational accuracy and convergence speed. This represents the best estimate describing the global background motion. Finally, these optimal translation and rotation parameters are arranged in the order of the affine transformation matrices to construct a 2x3 parameter matrix: The parameter matrix is ​​what we are looking for.

[0054] The steps to obtain the actual displacement vector field are as follows:

[0055] Read the pixel coordinates of the gimbal surface region of the first frame image in the gimbal test pixel coordinate set. Arrange the pixel coordinates of the gimbal surface region of the first frame image according to the horizontal coordinate, vertical coordinate, and coordinate index. Apply the parameter matrix to the pixel coordinates of the gimbal surface region of the first frame image item by item to obtain the theoretical mapping coordinates of the second frame. Subtract the horizontal coordinates of the pixel coordinates of the gimbal surface region of the first frame image from the horizontal coordinates of the theoretical mapping coordinates of the second frame image. Subtract the vertical coordinates of the pixel coordinates of the gimbal surface region of the first frame image from the vertical coordinates of the theoretical mapping coordinates of the second frame image to generate the theoretical displacement vector field.

[0056] Read the pixel coordinates of the gimbal surface area of ​​the first frame image and the second frame image within the gimbal test pixel coordinate set. Bind the pixel coordinates of the gimbal surface area of ​​the first frame image and the second frame image point by point according to the same coordinate index. Subtract the horizontal coordinate of the pixel coordinates of the gimbal surface area of ​​the first frame image from the horizontal coordinate of the pixel coordinates of the second frame image. Subtract the vertical coordinate of the pixel coordinates of the gimbal surface area of ​​the first frame image from the vertical coordinate of the pixel coordinates of the second frame image. Record the actual horizontal displacement component and the actual vertical displacement component according to the coordinate index to generate the actual displacement vector field.

[0057] Specifically, read all pixel coordinates of the gimbal surface area belonging to the first frame image within the gimbal test pixel coordinate set, and organize these points according to their coordinate indices. For each point in the set... Where j is the coordinate index, the parameter matrix representing the global background motion constructed in the previous step is called. Apply the matrix to the point Calculate the theoretical mapped coordinates of the point in the second frame image. The calculation process involves converting points... The coordinates are augmented to homogeneous coordinates Then perform matrix multiplication:

[0058] ;

[0059] in, arrive It is a parameter matrix The element in this step is to predict the position that a point on the gimbal should move to under the influence of global motion such as drone body shaking. Then, the calculated theoretical mapping coordinates of the second frame are used. The pixel coordinates of the gimbal surface area in the corresponding first frame image Perform component-by-component subtraction to generate a theoretical displacement vector. Its lateral displacement component is The longitudinal displacement component is This vector represents the pixel displacement caused only by background motion. The theoretical displacement vectors calculated from the pixel coordinates of all gimbal surface areas are stored according to their original coordinate index j, forming a data structure. This data structure maps the index of each point to its corresponding two-dimensional theoretical displacement vector, ultimately generating a theoretical displacement vector field.

[0060] Read the gimbal test pixel coordinate set. This set contains pairs of corresponding pixel coordinates of the gimbal surface area obtained through feature matching in the first and second frame images. Each pair of coordinates is bound by the same coordinate index. For example, a pair of points with index k is represented as... ,in These are the coordinates in the first frame of the image. This refers to the coordinates in the second frame image. Then, each pair of points in this coordinate set is traversed, and the actual pixel displacement from the first frame to the second frame is calculated point by point. For the pair of points with coordinate index k, its actual lateral displacement component is... By calculating the horizontal coordinates of corresponding points in the second frame image Subtract the horizontal coordinates of the original points in the first frame image Similarly, its actual longitudinal displacement components are obtained. By calculating the vertical coordinates of corresponding points in the second frame image Subtract the vertical coordinates of the original points in the first frame image Thus, these two components together constitute a two-dimensional actual displacement vector. This vector represents the sum of all the motions that occur at the pixel point between two frames, including the global motion of the drone body and the rotational motion of the gimbal itself. The calculated actual lateral displacement components and actual longitudinal displacement components are associated with their coordinate indices k and recorded. The actual displacement vectors of all pixels in the gimbal surface area are collected together to form a data mapping table with coordinate indices as keys and two-dimensional displacement vectors as values, generating an actual displacement vector field.

[0061] The steps to obtain the net displacement vector field are as follows:

[0062] The actual displacement vector data of each pixel in the actual displacement vector field is retrieved, and the theoretical displacement vector data corresponding to the same coordinate index in the theoretical displacement vector field is read. The horizontal actual displacement component, the vertical actual displacement component, the horizontal theoretical displacement component, and the vertical theoretical displacement component of the theoretical displacement vector data are checked item by item according to the pixel coordinate index. The horizontal theoretical displacement component is subtracted from the horizontal actual displacement component, and the vertical theoretical displacement component is subtracted from the vertical actual displacement component. The horizontal net displacement component and the vertical net displacement component are recorded according to the pixel coordinate index to obtain the net displacement vector field.

[0063] Specifically, the actual displacement vector field and the theoretical displacement vector field are invoked. Both vector fields use pixel coordinate indices as keys and store the corresponding two-dimensional displacement vectors. All coordinate indices of one of the vector fields are traversed. For example, for a pixel with index j, its lateral actual displacement component is read from the actual displacement vector field. and longitudinal actual displacement components Meanwhile, the corresponding lateral theoretical displacement components are read from the theoretical displacement vector field using the same index j. and longitudinal theoretical displacement components Perform vector subtraction to calculate the lateral net displacement components. and longitudinal net displacement components The purpose of this subtraction operation is to separate the background motion caused by the drone's body translation and minor jitter from the total motion of the pixels, thereby separating the motion component purely contributed by the gimbal's own rotation, and calculating the net lateral displacement component. and longitudinal net displacement components Combined into a net displacement vector The vector is associated with the pixel coordinate index j and stored. This process is repeated for all pixels that are successfully matched within the gimbal surface area. Finally, the coordinate indices of all pixels and their corresponding net displacement vectors are summarized to obtain the net displacement vector field.

[0064] The steps for obtaining the residual time series signal sequence are as follows:

[0065] The intrinsic parameter matrix of the external industrial camera is called, and the lateral and longitudinal net displacement components are read according to the pixel coordinate index in the net displacement vector field. The lateral and longitudinal coordinates of the pixel coordinates of the gimbal surface area in the first frame image are read. The lateral coordinates are added to the lateral net displacement components, and the longitudinal coordinates are added to the longitudinal net displacement components. The coordinates are recorded as decoupled mapping pixel coordinates according to the coordinate index. Homography parameters are estimated using the pixel coordinates of the gimbal surface area in the first frame image and the decoupled mapping pixel coordinates of the corresponding coordinate index to obtain the homography matrix. The homography matrix is ​​decomposed in combination with the intrinsic parameter matrix of the external industrial camera to obtain the rotation matrix in three-dimensional space. The rotation matrix in three-dimensional space is converted into the instantaneous actual angular velocity value of the gimbal according to the frame timestamp interval of two adjacent frames. The instantaneous actual angular velocity values ​​of the gimbal corresponding to multiple consecutive adjacent frames covering the complete gimbal test rotation cycle are arranged in the order of timestamps to construct a one-dimensional time series.

[0066] Extract the theoretical angular velocity setting value from the uniform rotation test command, write the theoretical angular velocity setting value into the motion baseline of the same length according to the timestamp order in the one-dimensional time sequence, read the instantaneous actual angular velocity value of the gimbal in the one-dimensional time sequence point by point, subtract the theoretical angular velocity setting value corresponding to the same timestamp in the motion baseline from the instantaneous actual angular velocity value of the gimbal corresponding to each timestamp, record the angular velocity residual value according to the timestamp order, and generate the residual time sequence signal.

[0067] Specifically, it calls the intrinsic parameter matrix of the external industrial camera, which was pre-obtained through camera calibration. The matrix is ​​in the form of ,in The focal length of the camera in pixels. The image's principal point coordinates are determined. Then, according to the pixel coordinate indices in the net displacement vector field, each point is processed individually. For the point with index j, its pixel coordinates in the gimbal surface region of the first frame image are read. and the corresponding net lateral displacement components and longitudinal net displacement components The original coordinates are added to the net displacement components to obtain the decoupled mapped pixel coordinates. This yields a set of point pairs. These point pairs describe the mapping relationship of pure gimbal rotation on the image plane. Using these point pairs, the Direct Linear Transform (DLT) algorithm with RANSAC (Random Sample Consensus) is employed to estimate the 3x3 homography matrix between them. The RANSAC iteration count was set to 1000, and the reprojection error threshold for distinguishing interior and exterior points was set to 1.5 pixels. This threshold was empirically determined based on the sub-pixel accuracy of feature point detection, resulting in the homography matrix. Then, combined with the intrinsic parameter matrix Decompose it to solve for the rotation matrix in three-dimensional space. This decomposition process is based on relational expressions. (In the pure rotational motion model), the rotation matrix is ​​obtained through the standard matrix decomposition algorithm. Then, rotate the matrix. Convert to axis-angle representation ,in It is a rotating axis. It is the rotation angle (in radians), based on the frame timestamp interval between two adjacent frames. (For example, for a 100fps camera,) (seconds), calculate the instantaneous actual angular velocity value of the gimbal. This process is applied to each pair of consecutive frames covering the entire gimbal test rotation cycle, and all calculated values ​​are arranged in timestamp order. Values ​​are used to construct a one-dimensional time series.

[0068] From the initial uniform rotation test command (e.g., a JSON object {"velocity_dps": 30.0}), the value of the "velocity_dps" field is extracted as the theoretical angular velocity setting value, for example, 30.0 degrees / second. Then, referring to the timestamp information of the one-dimensional time series constructed in the previous step, a motion baseline with the same timestamp sequence and the same data length is generated. The value of this motion baseline at each timestamp is set to this constant theoretical angular velocity setting value, i.e., 30.0. Next, the one-dimensional time series is traversed point by point, and at each timestamp t_i, the instantaneous actual angular velocity value of the gimbal at that moment is read. Simultaneously, the theoretical angular velocity setting value corresponding to the same timestamp t_i is read from the motion baseline. (i.e., 30.0), perform subtraction to calculate the residual value of angular velocity. This residual value reflects the instantaneous deviation or fluctuation of the actual angular velocity relative to the ideal uniform motion. Each time stamp t_i and its corresponding angular velocity residual value are then used to... Record each data point as a data point, arrange all these data points in chronological order according to their timestamps to form a new time series, and generate a residual time series signal sequence.

[0069] The steps to obtain the diagnostic results of the lag / abnormality are as follows:

[0070] Read the amplitude values ​​of each data point in the residual time-series signal sequence according to the timestamp order, record the sampling sequence number, sampling timestamp, and angular velocity residual value corresponding to each data point amplitude value, arrange the angular velocity residual values ​​according to the sampling sequence number to form the amplitude sequence to be converted, perform frequency component decomposition operation on the amplitude sequence to be converted, record the complex real part and complex imaginary part under each frequency index, calculate the frequency domain complex modulus value corresponding to each frequency index based on the complex real part and complex imaginary part, arrange the frequency domain complex modulus values ​​in ascending order of frequency, use the frequency index as the horizontal identifier and the frequency domain complex modulus value as the vertical identifier to generate the amplitude-frequency curve;

[0071] Extract the theoretical angular velocity setting value from the uniform rotation test command, read the rotation correspondence identifier and harmonic order identifier of the gimbal bearing, convert the theoretical angular velocity setting value into the reference rotation frequency of the gimbal bearing, expand the reference rotation frequency according to the rotation correspondence identifier and harmonic order identifier to form the theoretical fault characteristic frequency range of the gimbal bearing, search the frequency domain complex modulus value corresponding to each frequency index in the amplitude-frequency curve point by point, filter the frequency domain complex modulus values ​​that fall within the theoretical fault characteristic frequency range, compare the frequency domain complex modulus value change relationship corresponding to adjacent frequency indices, locate the amplitude pole value within the theoretical fault characteristic frequency range, and generate the target frequency domain harmonic peak value.

[0072] The system calls the preset gimbal bearing jamming alarm threshold value, reads the peak amplitude, peak frequency index, and theoretical fault characteristic frequency range of the target frequency domain harmonic peak value, compares the peak amplitude with the preset gimbal bearing jamming alarm threshold value, outputs an abnormal state if the peak amplitude is greater than the preset gimbal bearing jamming alarm threshold value, and writes the abnormal state, peak frequency index, and theoretical fault characteristic frequency range of the peak value into the diagnostic record. If the peak amplitude is less than or equal to the preset gimbal bearing jamming alarm threshold value, outputs an abnormal state that has not been triggered, and writes the abnormal state that has not been triggered, peak frequency index, and theoretical fault characteristic frequency range of the peak value into the diagnostic record to generate a jamming abnormality diagnosis result.

[0073] Specifically, the residual time-series signal sequence is read in timestamp order. This sequence consists of a series of data points containing sampling numbers, sampling timestamps, and angular velocity residual values. The angular velocity residual values ​​of all data points are extracted according to their sampling numbers to form a sequence of amplitude values ​​to be converted of length N, where N is the total number of sampling points. Before frequency analysis, a Hanning window function is first applied to this sequence to reduce spectral leakage caused by signal truncation. The window function smoothly decays to zero at both ends of the sequence. Then, a Fast Fourier Transform (FFT) algorithm is performed on the windowed amplitude value sequence to be converted to perform frequency component decomposition. The FFT algorithm converts the time-domain signal into a complex frequency-domain signal, and a complex result is obtained for each frequency index k (from 0 to N / 2). The complex result contains the complex real part. and the imaginary part of a complex number Based on these two components, calculate the frequency domain complex modulus, i.e., the amplitude, corresponding to each frequency index. The calculation method is as follows The obtained amplitude needs to be normalized. For non-zero frequency components (k > 0), the normalized amplitude is: For the DC component (k = 0), the normalized amplitude is Simultaneously, each frequency index k is converted to the actual physical frequency f, and the conversion relationship is as follows: ,in The sampling frequency of the original signal is determined by the frame rate of the instantaneous actual angular velocity calculated in the previous step, for example, 100Hz. Finally, all normalized frequency domain complex modulus values ​​are arranged in ascending order according to their corresponding physical frequencies f. The physical frequencies are used as the horizontal labels of the graph, and the corresponding frequency domain complex modulus values ​​are used as the vertical labels to generate the amplitude-frequency curve.

[0074] Extract the theoretical angular velocity setting value, such as 30 degrees / second, from the uniform rotation test command and convert it into the reference rotation frequency of the gimbal. ,Right now Next, it reads the pre-stored rotation correspondence identifiers and harmonic order identifiers for the bearings used in this model of gimbal. The rotation correspondence identifiers contain a set of formulas for calculating fault characteristic frequencies based on bearing geometric parameters, including the number of rolling elements n, the rolling element diameter d, the bearing pitch diameter D, and the contact angle. For example, the formula for calculating the inner race fault characteristic frequency (BPFI) is: The formula for calculating the outer ring fault characteristic frequency (BPFO) is as follows: The formula for calculating the characteristic frequency of rolling element failure (BSF) is as follows: The formula for calculating the cage failure characteristic frequency (FTF) is as follows: For example, the bearing parameters are n=10, d=5mm, D=40mm. =0, then BPFO can be calculated to be approximately The Hz harmonic order identifier specifies the octave range of the fault frequency to be of interest. For example, an identifier of [1, 2, 3] indicates interest in the 1st, 2nd, and 3rd harmonics. Based on this, a tolerance window with a width of 5% of the base fault frequency is set around each calculated fundamental fault characteristic frequency (e.g., BPFO) and its harmonics (e.g., 2BPFO, 3BPFO), forming multiple discrete theoretical fault characteristic frequency intervals. For example, the first harmonic interval for BPFO is [0.364 * 0.975, 0.364 * 1.025], i.e., [0.355, ...]. [0.373]Hz, then, the amplitude-frequency curve is searched point by point, and all frequency indices falling within these theoretical fault characteristic frequency intervals are selected as complex modulus values ​​in the frequency domain. Within each interval, local maxima are located by comparing the amplitudes of adjacent frequency points. If the amplitude of a point is greater than the amplitudes of the two points before and after it, then the point is identified as an amplitude pole value. All amplitude pole values ​​found within the theoretical fault characteristic frequency intervals are collected to generate the target frequency domain harmonic peak value.

[0075] The system calls a preset threshold value for gimbal bearing jamming alarms. This threshold is not a fixed empirical value, but rather a statistically generated value based on standard tests performed on a group of gimbals of the same model that are in good condition. Specifically, the system executes a complete test procedure on each healthy gimbal, generating the amplitude-frequency curve of its residual timing signal. Within the same theoretical fault characteristic frequency range as the device under test, the system extracts the maximum amplitude of each healthy gimbal in these ranges, forming a healthy sample amplitude set. The average value of this set is then calculated. and standard deviation The alarm threshold is set to For example, if the average amplitude of a healthy sample in the BPFO characteristic frequency range is 0.02 degrees / second and the standard deviation is 0.005 degrees / second, then the alarm threshold for this range is set as follows: Next, each record in the target frequency domain harmonic peak values ​​is read, including the peak amplitude, peak frequency index, and the theoretical fault characteristic frequency interval to which the peak belongs (e.g., "BPFO second harmonic"). The amplitude of each peak is compared with the gimbal bearing jamming alarm threshold value corresponding to its interval. If the peak amplitude is greater than the preset gimbal bearing jamming alarm threshold value, for example, the measured BPFO second harmonic peak is 0.04 degrees / second, which is greater than the threshold of 0.035 degrees / second, then the diagnostic status is marked as "abnormal status", and this status, peak amplitude (0.04), peak frequency index, and interval ("BPFO second harmonic") are written as a record into the diagnostic result. If the peak amplitude is less than or equal to the preset gimbal bearing jamming alarm threshold value, then the diagnostic status is marked as "abnormal status not triggered", and detailed information is written into the diagnostic record. The comparison results of all target frequency domain harmonic peak values ​​are summarized to generate a jamming abnormality diagnostic result.

Claims

1. An image detection-based intelligent maintenance and recognition system for unmanned aerial vehicles (UAVs), characterized in that, The system includes: The UAV image acquisition module receives a uniform rotation test command to drive the gimbal to rotate, starts an external industrial camera to record a continuous frame video sequence, and generates a static reference pixel coordinate set; it extracts the pixel coordinate points of the gimbal surface area in the first frame image, matches the corresponding pixel coordinate points in the second frame image for position binding and pairing, and generates a gimbal test pixel coordinate set. The global motion decoupling module calculates the mean square error of pixel offsets in two frames of images within the static reference pixel coordinate set to construct a parameter matrix. It then uses this parameter matrix to calculate the pixel coordinates of the gimbal surface region in the first frame of the gimbal test pixel coordinate set, obtaining the theoretical mapping coordinates of the second frame. Finally, it calculates the difference between the theoretical mapping coordinates of the second frame and the corresponding pixel coordinates of the gimbal surface region in the first frame to generate a theoretical displacement vector field. Finally, it extracts the pixel coordinates of the gimbal surface region corresponding to the two frames of images within the gimbal test pixel coordinate set and calculates them to generate the actual displacement vector field.

2. The image detection-based intelligent maintenance and recognition system for unmanned aerial vehicles according to claim 1, characterized in that, The system also includes: The rotational motion residual extraction module calls the actual displacement vector data of each pixel in the actual displacement vector field and the corresponding theoretical displacement vector data in the theoretical displacement vector field to obtain the net displacement vector field; it extracts the pixel data of each pixel in the net displacement vector field and performs coordinate operations with the pixel coordinates of the gimbal surface area of ​​the first frame image to generate decoupled mapping pixel coordinates; it performs matrix parameter estimation based on the pixel coordinates of the gimbal surface area of ​​the first frame image and the decoupled mapping pixel coordinates to obtain the homography matrix; it performs matrix decomposition on the homography matrix in combination with the intrinsic parameter matrix of the external industrial camera to obtain the rotation matrix in three-dimensional space, and converts the rotation matrix into the instantaneous actual angular velocity value of the gimbal; it arranges the instantaneous actual angular velocity values ​​of the gimbal corresponding to multiple consecutive adjacent images covering the complete gimbal test rotation cycle in timestamp order to construct a one-dimensional time series sequence; it performs subtraction operations on the one-dimensional time series sequence according to the uniform rotation test command to generate a residual time series signal sequence; The bearing jamming identification module extracts the amplitude of each data point within the residual time-series signal sequence to obtain the complex modulus value in the frequency domain; arranges the complex modulus values ​​in the frequency domain and plots the corresponding amplitude-frequency curve; combines the uniform rotation test command to calculate the theoretical fault characteristic frequency range of the gimbal bearing; locates the amplitude pole values ​​falling within the theoretical fault characteristic frequency range in the amplitude-frequency curve to generate the target frequency domain harmonic peak value; compares the relationship between the target frequency domain harmonic peak value and the preset gimbal bearing jamming alarm threshold value to generate a jamming anomaly diagnosis result.

3. The image detection-based intelligent maintenance and recognition system for unmanned aerial vehicles according to claim 1, characterized in that, The steps for obtaining the static reference pixel coordinate set are as follows: Receive the uniform rotation test command, read the rotation start / stop flag and rotation direction flag in the uniform rotation test command, transmit the rotation start / stop flag to the gimbal driver end, transmit the rotation direction flag to the gimbal attitude control end, trigger the recording port of the external industrial camera after the gimbal enters the uniform rotation state, record the image frame sequence, frame timestamp, frame number, and image pixel coordinate range during the gimbal rotation process according to the adjacent frame continuous storage rule, and generate a continuous frame video sequence. Based on the continuous frame video sequence, the first and second frames of two adjacent frames are selected. The pixel coordinates of the fuselage shell area boundary in the first frame are located, and the pixel coordinates of the landing gear area structure in the first frame are located. According to the frame number correspondence, the pixel coordinates of the same fuselage shell area and the same landing gear area are retrieved in the second frame. The pixel coordinates of the first frame and the second frame are paired according to the area identifier, frame number, and coordinate index to generate a static reference pixel coordinate set.

4. The image detection-based intelligent maintenance and recognition system for unmanned aerial vehicles according to claim 1, characterized in that, The steps for obtaining the gimbal test pixel coordinate set are as follows: Based on the continuous frame video sequence, the first and second frames of two adjacent frames are selected. The edge pixel coordinates, texture pixel coordinates, and corner pixel coordinates of the gimbal surface area in the first frame are located. The corresponding edge pixel coordinates, texture pixel coordinates, and corner pixel coordinates are retrieved in the second frame according to the gimbal surface area coordinate index. The gimbal surface area pixel coordinates of the first and second frames are bound and paired according to the coordinate index, frame number, and surface area identifier to generate a gimbal test pixel coordinate set.

5. The image detection-based intelligent maintenance and recognition system for unmanned aerial vehicles according to claim 1, characterized in that, The steps for obtaining the parameter matrix are as follows: Read the pixel coordinates of the fuselage area of ​​the first frame image, the pixel coordinates of the landing gear area of ​​the first frame image, the pixel coordinates of the fuselage area of ​​the second frame image, and the pixel coordinates of the landing gear area of ​​the second frame image within the static reference pixel coordinate set. Calculate the lateral and longitudinal coordinate offsets point by point according to the coordinate index. Accumulate the squared errors of the lateral and longitudinal coordinate offsets respectively. Filter the translation and rotation parameters corresponding to the minimum value of the accumulated squared error. Arrange the translation and rotation parameters according to the coordinate transformation order to construct a parameter matrix.

6. The image detection-based intelligent maintenance and recognition system for unmanned aerial vehicles according to claim 1, characterized in that, The steps for obtaining the actual displacement vector field are as follows: Read the pixel coordinates of the gimbal surface region of the first frame image in the gimbal test pixel coordinate set, arrange the pixel coordinates of the gimbal surface region of the first frame image according to the horizontal coordinate, vertical coordinate, and coordinate index, apply the parameter matrix item by item to the pixel coordinates of the gimbal surface region of the first frame image to obtain the theoretical mapping coordinates of the second frame, subtract the horizontal coordinates of the pixel coordinates of the gimbal surface region of the first frame image from the horizontal coordinates of the theoretical mapping coordinates of the second frame image, and subtract the vertical coordinates of the pixel coordinates of the gimbal surface region of the first frame image from the vertical coordinates of the theoretical mapping coordinates of the second frame image to generate the theoretical displacement vector field; Read the gimbal surface area pixel coordinates of the first frame image and the second frame image within the gimbal test pixel coordinate set. Bind the gimbal surface area pixel coordinates of the first frame image and the second frame image point by point according to the same coordinate index. Subtract the horizontal coordinate of the gimbal surface area pixel coordinates of the first frame image from the horizontal coordinate of the second frame image. Subtract the vertical coordinate of the gimbal surface area pixel coordinates of the first frame image from the vertical coordinate of the second frame image. Record the actual horizontal displacement component and the actual vertical displacement component according to the coordinate index to generate the actual displacement vector field.

7. The image detection-based intelligent maintenance and recognition system for unmanned aerial vehicles according to claim 2, characterized in that, The steps for obtaining the net displacement vector field are as follows: The actual displacement vector data of each pixel in the actual displacement vector field is called, and the theoretical displacement vector data corresponding to the same coordinate index in the theoretical displacement vector field is read. The horizontal actual displacement component, the vertical actual displacement component, the horizontal theoretical displacement component, and the vertical theoretical displacement component of the theoretical displacement vector data are checked item by item according to the pixel coordinate index. The horizontal theoretical displacement component is subtracted from the horizontal actual displacement component, and the vertical theoretical displacement component is subtracted from the vertical actual displacement component. The horizontal net displacement component and the vertical net displacement component are recorded according to the pixel coordinate index to obtain the net displacement vector field.

8. The image detection-based intelligent maintenance and recognition system for unmanned aerial vehicles according to claim 2, characterized in that, The steps for obtaining the residual time-series signal sequence are as follows: Call the intrinsic parameter matrix of the external industrial camera, read the horizontal net displacement component and the vertical net displacement component according to the pixel coordinate index in the net displacement vector field, read the horizontal coordinate and vertical coordinate of the pixel coordinate point in the gimbal surface area of ​​the first frame image, add the horizontal net displacement component to the horizontal coordinate, add the vertical net displacement component to the vertical coordinate, and record it as the decoupled mapping pixel coordinate point according to the coordinate index. Homography parameters are estimated using the pixel coordinates of the gimbal surface area and the decoupled mapping pixel coordinates of the corresponding coordinate indices in the first frame image to obtain the homography matrix. The homography matrix is ​​then decomposed in combination with the intrinsic parameter matrix of the external industrial camera to obtain the rotation matrix in three-dimensional space. The rotation matrix in three-dimensional space is converted into the instantaneous actual angular velocity value of the gimbal according to the frame timestamp interval of two adjacent frames. The instantaneous actual angular velocity values ​​of the gimbal corresponding to multiple consecutive adjacent frames covering the complete test rotation cycle of the gimbal are arranged in the order of timestamps to construct a one-dimensional time series. Extract the theoretical angular velocity setting value from the uniform rotation test command, write the theoretical angular velocity setting value into the motion baseline of the same length according to the timestamp order in the one-dimensional time sequence, read the instantaneous actual angular velocity value of the gimbal in the one-dimensional time sequence point by point, subtract the theoretical angular velocity setting value corresponding to the same timestamp in the motion baseline from the instantaneous actual angular velocity value of the gimbal corresponding to each timestamp, record the angular velocity residual value according to the timestamp order, and generate the residual time sequence signal sequence.

9. The image detection-based intelligent maintenance and recognition system for unmanned aerial vehicles according to claim 2, characterized in that, The steps for obtaining the diagnostic results of the lag are as follows: The amplitude values ​​of each data point in the residual time-series signal sequence are read in the order of timestamps. The sampling number, sampling timestamp, and angular velocity residual value corresponding to each data point amplitude are recorded. The angular velocity residual values ​​are arranged into a sequence of amplitude values ​​to be converted according to the sampling number. Frequency component decomposition operation is performed on the sequence of amplitude values ​​to be converted. The complex real part and complex imaginary part under each frequency index are recorded respectively. The frequency domain complex modulus value corresponding to each frequency index is calculated based on the complex real part and complex imaginary part. The frequency domain complex modulus values ​​are arranged in ascending order of frequency. The frequency index is used as the horizontal identifier and the frequency domain complex modulus value is used as the vertical identifier to generate an amplitude-frequency curve. Extract the theoretical angular velocity setting value from the uniform rotation test command, read the rotation correspondence identifier and harmonic order identifier of the gimbal bearing, convert the theoretical angular velocity setting value into the reference rotation frequency of the gimbal bearing, expand the reference rotation frequency according to the rotation correspondence identifier and harmonic order identifier to form the theoretical fault characteristic frequency range of the gimbal bearing, retrieve the frequency domain complex modulus value corresponding to each frequency index in the amplitude-frequency curve point by point, filter the frequency domain complex modulus values ​​that fall within the theoretical fault characteristic frequency range, compare the frequency domain complex modulus value change relationship corresponding to adjacent frequency indices, locate the amplitude pole value within the theoretical fault characteristic frequency range, and generate the target frequency domain harmonic peak value. The system calls a preset gimbal bearing jamming alarm threshold value, reads the peak amplitude, peak frequency index, and theoretical fault characteristic frequency range of the target frequency domain harmonic peak value, compares the peak amplitude with the preset gimbal bearing jamming alarm threshold value, outputs an abnormal state if the peak amplitude is greater than the preset gimbal bearing jamming alarm threshold value, and writes the abnormal state, peak frequency index, and theoretical fault characteristic frequency range of the peak value into the diagnostic record. If the peak amplitude is less than or equal to the preset gimbal bearing jamming alarm threshold value, outputs a non-triggered abnormal state, and writes the non-triggered abnormal state, peak frequency index, and theoretical fault characteristic frequency range of the peak value into the diagnostic record to generate a jamming abnormality diagnostic result.

Images