Unmanned aerial vehicle hoisting system swing angle measurement method for solving visual motion blur

Abstract

Aiming at the technical problem that the swing angle measurement precision is reduced due to the swing characteristic of a load and a fuzzy effect generated in a motion process in an unmanned aerial vehicle hoisting task, the invention provides the swing angle measurement method for the unmanned aerial vehicle hoisting system, which solves the problem of visual motion blurring. The method comprises the following steps: firstly, establishing a geometrical relationship model between a camera and a load, and acquiring characteristics of a load marker by using the camera; secondly, analyzing high-order state information of the marker between adjacent frames, predicting the position of the next frame and setting boundary constraints of the region of interest by combining a Kalman filter, and reducing the data processing amount; then, a self-adaptive template learning rate is introduced, edge information is enhanced by using gray multiplication, and the tracking robustness of the algorithm under a fuzzy condition is improved; and finally, realizing accurate calculation of the load swing angle by utilizing priori knowledge of the length of the lifting rope and combining a trigonometric function relationship. According to the method, the problem of motion blur is effectively solved, the cost is low, the implementation is simple, and the real-time performance is relatively high.

Description

Technical Field

[0001] The present invention belongs to the field of unmanned systems and robot intelligent perception and control equipment, especially to the field of machine vision and image processing, and more particularly to a method for measuring the swing angle of an unmanned aerial vehicle lifting system for solving visual motion blur. Background Art

[0002] With the development of artificial intelligence technology, computer vision technology has been widely used in diversified mission scenarios of unmanned systems, and target detection and tracking technology for lifting tasks has become a research hotspot. At present, drone lifting systems have been widely used in emergency disaster reduction, material transportation, smart agriculture and other fields. However, during the rapid maneuvering of drones, the load often swings violently, which not only reduces the operating efficiency of the lifting task, but also may cause other safety hazards. Therefore, in order to improve the swing reduction control effect of the drone lifting system, how to accurately obtain the load swing angle has become a key technical problem that needs to be solved urgently.

[0003] In order to measure the load swing angle in the UAV lifting system, the existing technology mainly adopts two methods: contact measurement and non-contact measurement. Among them, the contact measurement method mainly uses inertial measurement elements or potentiometers to obtain the load swing angle information, but it is affected by the flight vibration of the UAV, not only the measurement noise is difficult to avoid, but also there are problems such as difficulty in installation and maintenance; the non-contact measurement method mainly uses an optical motion capture system to obtain the load swing angle information. Although this technology has high measurement accuracy, it has disadvantages such as limited application environment and high cost. In addition, some studies have tried to use computer vision technology to obtain the swing angle information by detecting and tracking the load target, but the motion blur problem caused by the rapid swing of the load is not considered, which seriously affects the reliability of the measurement. Therefore, it is necessary to propose a swing angle measurement method that can solve the problem of unreliable swing angle information caused by motion blur in the UAV lifting system and has wide application value.

[0004] On November 22, 2021, invention patent CN114035598A disclosed a visual swing angle detection and reduction method for a multi-rotor suspension system. The technical problem to be solved by this method is to use computer vision measurement methods to obtain the relative position of the load in real time in order to calculate the swing angle of the load. The main technical method adopted is to install a binocular camera on the multi-rotor base plate, obtain the three-dimensional coordinates of the marker set between the load and the suspension rope in the camera coordinate system, and then calculate the swing angle of the load. This patent uses a binocular camera to capture the features of the markers on the load, aiming to provide more accurate spatial positioning. However, the problem of visual motion blur that occurs during the rapid swing of the load has not been fully considered in the study. The hardware cost and the computational cost of data processing are high. Especially for resource-constrained drone systems, the engineering implementation of the technology is extremely challenging.

[0005] The journal article "Real-time Detection Method of Bridge Crane Load Swing Angle Based on Improved Kernel Correlation Filtering" proposed an improved kernel correlation filtering algorithm. First, the Kalman filter prediction technology is combined to predict the position of the marker; secondly, the scale update strategy is introduced to adjust the area of ​​interest; then, the center and radius of the marker are extracted through the fast circle detection algorithm; finally, the load position and swing angle information are calculated. However, this method has the following problems:

[0006] 1. The Kalman filter proposed in the paper only considers the low-order state information of the marker, which is prone to produce large cumulative errors in the process of adapting to nonlinear changes, thereby reducing the robustness of the model;

[0007] 2. The adaptive scale update strategy proposed in the paper does not fully consider the image boundary conditions, resulting in the risk that the adjusted region of interest exceeds the image area. At the same time, the speed threshold of the template learning rate may cause the learning behavior near the critical point to be not smooth enough and prone to jitter;

[0008] 3. This method relies on the assumption of similar triangles when calculating the swing angle. However, this assumption may not be completely valid in the case of nonlinear perspective distortion, especially when the experimental scene is changed frequently, and the monocular camera needs to be calibrated each time.

[0009] Therefore, the swing angle measurement methods provided in the above-mentioned documents all have certain limitations. The method of the present invention fully considers the influence of motion blur caused by rapid swing of the load on the image acquisition process of the monocular camera, and designs a swing angle measurement method for a UAV lifting system that solves the visual motion blur, aiming to solve the problems of low swing angle measurement accuracy and poor real-time performance in the UAV lifting system. Summary of the invention

[0010] In view of the problems existing in the prior art, the present invention provides a method for measuring the swing angle of a UAV lifting system that solves visual motion blur, with the purpose of solving the technical problem that the swing characteristics of the load and the blurring effect produced during rapid movement during the UAV lifting task lead to a decrease in the accuracy of the swing angle measurement. By introducing state information into the tracking of the load marker, the Kalman filter is integrated to predict the position of the marker and dynamically adjust the region of interest, which significantly reduces the amount of data processing. In combination with the movement speed of the marker and the degree of edge blur, the filter template learning rate is updated in real time to adapt to the changes in the marker features in a timely manner. At the same time, the grayscale of the image of the region of interest is enhanced to improve the method's adaptability to motion blur. Finally, based on the prior knowledge of the length of the lifting rope and the geometric relationship, the accurate calculation of the load swing angle is achieved.

[0011] In order to achieve the above object, the present invention adopts the following specific technical solutions to solve it:

[0012] A method for measuring the swing angle of a UAV lifting system to solve visual motion blur comprises the following steps:

[0013] S1: Build the swing angle measurement device of the UAV lifting system and establish the coordinate system of the UAV lifting system;

[0014] S2: Based on the kernel correlation filtering algorithm of the fused Kalman filter, the position of the marker in the subsequent image frames is accurately predicted by considering the motion state information of the marker, and the size of the region of interest is dynamically adjusted according to the motion speed of the marker, while adding boundary constraints;

[0015] S3: During the continuous frame processing, the learning rate of the filter template is adaptively adjusted according to the movement speed of the marker and the blur degree of the marker edge;

[0016] S4: The swing angle measuring device for the UAV lifting system according to S1, wherein the lightweight marker is a reflective ball, the edge features of the marker are improved by a grayscale enhancement method, and then the outline of the marker is extracted by a gradient search, and the center position of the marker is calculated;

[0017] S5: Establish a geometric relationship model between the body and the load, take the fixed length of the suspension rope as prior knowledge, use the trigonometric function relationship to calculate the depth information of the load in the camera coordinate system, and then obtain the swing angle information.

[0018] The technical solution of this method is characterized by:

[0019] S1 specifically includes the following steps:

[0020] S1.1: A sling is provided at the center of the UAV base plate for suspending the load. A lightweight marker is fixed above the load. The marker is located directly above the connection between the sling and the load.

[0021] S1.2: A monocular camera is installed at the end of the bottom plate of the UAV. Specifically, the monocular camera is installed tilted downward and can be adjusted within a range of plus or minus 45 degrees in the vertical plane with vertical downward as the reference, so as to ensure that its field of view completely covers the movement area of ​​the payload.

[0022] S1.3: Considering that the monocular camera is fixedly installed on the drone body, the coordinate system of the drone lifting system consists of four relatively independent parts, namely the world coordinate system, the camera coordinate system, the image coordinate system and the pixel coordinate system. The conversion relationship is as follows:

[0023]

[0024] In the formula, (X w ,Y w ,Z w ) is the three-dimensional coordinate of the marker in the world coordinate system, (Xc ,Y c ,Z c ) is the three-dimensional coordinate of the marker in the camera coordinate system, R is the rotation matrix from the camera coordinate system to the world coordinate system, T is the translation component from the camera coordinate system to the world coordinate system, and Z c is the depth information of the center of the marker in the camera coordinate system, (x, y) is the projection coordinate of the marker on the normalized plane, f is the focal length of the monocular camera, and f x and f y is the pixel size, u 0 and v 0 is the optical center position of the monocular camera image.

[0025] S2 specifically includes the following steps:

[0026] S2.1: Use Kalman filter to predict the motion state of the marker and define the marker state vector S k =[x k x k 'x k ″x k ″′y k y k ′y k ″y k ″′] Τ , where (x k ,y k ) is the initial coordinate of the marker, (x k ′,y k ′) is the velocity component of the marker along the x-axis and y-axis, (x k ″,y k ″) is the acceleration component of the marker along the x-axis and y-axis, (x k ″′,y k ″′) is the acceleration component of the marker along the x-axis and y-axis, and the subscript k represents the current image frame. The specific formula is as follows:

[0027]

[0028] Wherein, the subscript k-1 represents the adjacent previous image frame.

[0029] S2.2: Initialize the kernel correlation filter algorithm by inputting the first frame image, consider the motion state information of the marker and use the Kalman filter to predict the position of the marker in the next frame. The specific formula is as follows:

[0030]

[0031] In the formula, is the predicted coordinate of the marker, Δt is the sampling time interval between adjacent image frames, and the subscript k+1 represents the adjacent next image frame.

[0032] S2.3: Use a multi-scale strategy to adjust the region of interest. The initial size of the region of interest is determined by the first frame of image. According to the velocity components of the marker along the x-axis and y-axis described in S2.1, calculate the motion velocity v of the marker in the current image frame. k , the specific formula is as follows:

[0033]

[0034] S2.4: Set speed threshold v th To limit the marker speed range, dynamically adjust the size of the region of interest according to the marker speed. The judgment condition of the region of interest size is:

[0035]

[0036] Where W k ×H k is the size of the region of interest, W 0 and H 0 are the width and height of the region of interest in the initial frame, W k and H k are the width and height of the region of interest in the current frame, s 1 and 2 is the scale factor.

[0037] S2.5: Add boundary constraints. If the adjustment of the ROI exceeds the set image boundary, the ROI is corrected again. The boundary conditions are as follows:

[0038]

[0039] Where W max and H max are the horizontal and vertical resolutions of the image, respectively. and are the horizontal and vertical coordinates of the center point of the region of interest in the image, respectively.

[0040] S3 specifically includes the following steps:

[0041] S3.1: Constructing a learning rate μ related to the marker speed 1 , the specific formula is as follows:

[0042]

[0043] In the formula, v k Indicates the movement speed of the marker in the current image frame.

[0044] S3.2: Calculate the gradient change ΔG between adjacent frames k (x,y), the specific formula is as follows:

[0045] ΔG k (x,y)=|G k+1 (x,y)-G k (x,y)| (10)

[0046] In the formula, G k (x, y) represents the gradient amplitude of the marker in the current image frame, G k+1 (x, y) represents the gradient amplitude of the marker in the next image frame.

[0047] S3.3: Gradient normalization, calculate the mean value of the gradient change between adjacent frames. The specific formula is as follows:

[0048]

[0049] Where N represents the total number of pixels in the current image frame.

[0050] S3.4: Construct a learning rate μ related to the gradient change between adjacent frames 2 , the specific formula is as follows:

[0051]

[0052] In the formula, μ 0 is the base learning rate of the filter template.

[0053] S3.5: Calculate the filter template learning rate μ based on the movement speed of the marker and the gradient change between adjacent frames. The specific formula is as follows:

[0054] μ=λ 1 μ 1 +λ 2 μ 2 (13)

[0055] In the formula, λ 1 and λ 2 is the weight coefficient and satisfies λ 1 <λ 2 .

[0056] According to the filter template learning rate, the filter coefficient of the relevant kernel filter is updated. The specific formula is as follows:

[0057] α k =(1-μ)α k +μα k-1 (14)

[0058] In the formula, α kis the filter coefficient of the current image frame, α k-1 is the filter coefficient of the previous image frame, and the subscript k-1 represents the adjacent previous image frame.

[0059] S4 specifically includes the following steps:

[0060] S4.1: Implement grayscale multiplication enhancement processing for the region of interest and construct a binarization judgment criterion. The specific formula is as follows:

[0061]

[0062] In the formula, I(x k ,y k ) represents the gray value corresponding to the marker, I Bin (x k ,y k ) represents the binary image, and T is the threshold.

[0063] S4.2: Based on the center position of the region of interest, the gradient search is used to expand the search outward, the grayscale gradient value of each search point is calculated, and the point set with the largest gradient value is determined as the edge point set of the marker. The least squares method is used to fit the obtained edge point set. The specific formula is as follows:

[0064]

[0065] Where J is the search length, I(j) is the grayscale value of the jth pixel, argmax(G i ) is the pixel distance between the maximum point of the gray gradient in the i direction and the starting point, is the starting position of the search, E i Represents the edge point set in direction i.

[0066] S4.3: Select two edge point pairs with equal gradients and opposite directions on the fitted circle, connect the edge point pairs to form a diameter segment, calculate the midpoint coordinates of the diameter segment, repeat multiple times to obtain multiple sets of midpoint coordinates, and calculate the arithmetic mean of all midpoint coordinates as the final center coordinates of the circle.

[0067] S5 specifically includes the following steps:

[0068] S5.1: According to the coordinate system of the drone lifting system described in S1, the two-dimensional pixel coordinates (u, v) of the center of the marker circle are converted to the three-dimensional world coordinates (X w ,Y w ,Z w ) can be described as:

[0069]

[0070] In the formula, [] -1Represents the inverse of a matrix.

[0071] S5.2: Based on the system prior knowledge, the depth information of the center of the marker in the camera coordinate system is approximately calculated. The specific formula is as follows:

[0072]

[0073] In the camera coordinate system, the normalized direction vector (X c ′,Y c ′,1) can be obtained from the pixel coordinates (u,v) and the camera intrinsic parameter matrix M:

[0074]

[0075] In the formula, (X c ′,Y c ′,1) represents the direction of the marker’s center position in the camera coordinate system.

[0076] According to the geometric relationship, the three-dimensional coordinates (X c ,Y c ,Z c ) can be expressed as:

[0077]

[0078] Where σ is the proportionality coefficient and the depth information Z of the center of the marker c Directly proportional.

[0079] Calculate the depth information of the center of the marker in the camera coordinate system. The specific formula is as follows:

[0080]

[0081] Where L is the fixed length of the suspension rope.

[0082] In the world coordinate system, the three-dimensional coordinates of the center of the marker can be further written as:

[0083]

[0084] In the formula, u 0 、v 0 、f x 、f y Both can be uniquely determined by the camera intrinsic parameter matrix M.

[0085] S5.3: Calculate the load swing angle information. The specific formula is as follows:

[0086]

[0087] Where θ is the swing angle of the load in the world coordinate system.

[0088] The present invention has the following beneficial effects:

[0089] 1. The present invention can calculate the load swing angle by only collecting the image position of the marker through a monocular camera. The detection efficiency, durability and noise resistance are far higher than contact measurement methods such as photoelectric encoders, and the cost, computing overhead and software and hardware requirements are far lower than similar non-contact measurement methods such as binocular cameras. In addition, the measurement accuracy does not depend on the load shape, and it has a good measurement effect on loads of various shapes.

[0090] 2. The present invention adds high-order load state information and integrates the kernel correlation algorithm of the Kalman filter to better predict the trajectory of load movement changes. The size of the region of interest is adjusted according to the prediction results. The average processing time under motion blur conditions is shortened by 43.69% compared with the KCF-MC method, which improves the calculation efficiency. At the same time, boundary conditions are set to prevent the region of interest from exceeding the actual boundary of the image, which can meet the real-time requirements in actual engineering applications.

[0091] 3. In order to address the influence of motion blur caused by rapid swinging of the load on the visual measurement results, the filter template learning rate can be adaptively updated in combination with the motion speed and blur degree of the marker. Experimental results show that under blurry conditions, the tracking success rate of this method is improved by 9.09% compared with the MS-HF method, thereby enhancing the robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0093] Figure 1 Roadmap for the swing angle measurement method of UAV lifting system to solve visual motion blur;

[0094] Figure 2 This is a schematic diagram of the coordinate system of the UAV lifting system;

[0095] Figure 3 This is a schematic diagram of the diameter sampling of the markers in the drone lifting system;

[0096] Figure 4 This is a flow chart of the swing angle measurement method for the UAV lifting system;

[0097] Figure 5 This is the experimental scene diagram of the drone lifting system;

[0098] Figure 6 This is a schematic diagram of the camera calibration template for the drone lifting system;

[0099] Figure 7 This is the effect diagram of the visual tracking method of the UAV lifting system;

[0100] Figure 8 This is the measurement result of the load swing angle of the UAV lifting system. DETAILED DESCRIPTION

[0101] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and understandable, a method for measuring the swing angle of a UAV lifting system to solve visual motion blur is provided. Figure 1 As shown, the following steps are included:

[0102] S1: Build the swing angle measurement device of the UAV lifting system and establish the coordinate system of the UAV lifting system, which specifically includes the following sub-steps:

[0103] S1.1: A sling is provided at the center of the UAV base plate for suspending the load. A lightweight marker is fixed above the load. The marker is located directly above the connection between the sling and the load.

[0104] S1.2: A monocular camera is installed at the end of the bottom plate of the UAV. Specifically, the monocular camera is installed tilted downward and can be adjusted within a range of plus or minus 45 degrees in the vertical plane with vertical downward as the reference, so as to ensure that its field of view completely covers the movement area of ​​the payload.

[0105] S1.3: Considering that the monocular camera is fixedly installed on the UAV body, the coordinate system of the UAV lifting system consists of four relatively independent parts, namely the world coordinate system O w -X w Y w Z w , camera coordinate system O c -X c Y c Z c , image coordinate system o-xy and pixel coordinate system o uv -uv, such as Figure 2 As shown, the conversion relationship is as follows:

[0106]

[0107] In the formula, (X w ,Y w ,Z w ) is the three-dimensional coordinate of the marker in the world coordinate system, (X c ,Y c ,Z c) is the three-dimensional coordinate of the marker in the camera coordinate system, R is the rotation matrix from the camera coordinate system to the world coordinate system, T is the translation component from the camera coordinate system to the world coordinate system, and Z c is the depth information of the center of the marker in the camera coordinate system, (x, y) is the projection coordinate of the marker on the normalized plane, f is the focal length of the monocular camera, and f x and f y is the pixel size, u 0 and v 0 is the optical center position of the monocular camera image.

[0108] S2: Based on the kernel correlation filtering algorithm of the fused Kalman filter, the position of the marker in the subsequent image frames is accurately predicted by considering the motion state information of the marker, and the size of the region of interest is dynamically adjusted according to the motion speed of the marker, while adding boundary constraints. It specifically includes the following sub-steps:

[0109] S2.1: Use Kalman filter to predict the motion state of the marker and define the marker state vector S k =[x k x k 'x k ″x k ″′y k y k ′y k ″y k ″′] Τ , where (x k ,y k ) is the initial coordinate of the marker, (x k ′,y k ′) is the velocity component of the marker along the x-axis and y-axis, (x k ″,y k ″) is the acceleration component of the marker along the x-axis and y-axis, (x k ″′,y k ″′) is the acceleration component of the marker along the x-axis and y-axis, and the subscript k represents the current image frame. The specific formula is as follows:

[0110]

[0111] Wherein, the subscript k-1 represents the adjacent previous image frame.

[0112] S2.2: Initialize the kernel correlation filter algorithm by inputting the first frame image, consider the motion state information of the marker and use the Kalman filter to predict the position of the marker in the next frame. The specific formula is as follows:

[0113]

[0114] In the formula, is the predicted coordinate of the marker, Δt is the sampling time interval between adjacent image frames, and the subscript k+1 represents the adjacent next image frame.

[0115] S2.3: Use a multi-scale strategy to adjust the region of interest. The initial size of the region of interest is determined by the first frame of image. According to the velocity components of the marker along the x-axis and y-axis described in S2.1, calculate the motion velocity v of the marker in the current image frame. k , the specific formula is as follows:

[0116]

[0117] S2.4: Set speed threshold v th To limit the marker speed range, dynamically adjust the size of the region of interest according to the marker speed. The judgment condition of the region of interest size is:

[0118]

[0119] Where W k ×H k is the size of the region of interest, W 0 and H 0 are the width and height of the region of interest in the initial frame, W k and H k are the width and height of the region of interest in the current frame, s 1 and 2 is the scale factor.

[0120] S2.5: Add boundary constraints. If the adjustment of the ROI exceeds the set image boundary, the ROI is corrected again. The boundary conditions are as follows:

[0121]

[0122] Where W max and H max are the horizontal and vertical resolutions of the image, respectively. and are the horizontal and vertical coordinates of the center point of the region of interest in the image, respectively.

[0123] S3: In the process of continuous frame processing, the learning rate of the filter template is adaptively adjusted according to the movement speed of the marker and the blur degree of the marker edge, which specifically includes the following sub-steps:

[0124] S3.1: Constructing a learning rate μ related to the marker speed 1 , the specific formula is as follows:

[0125]

[0126] In the formula, v k Indicates the movement speed of the marker in the current image frame.

[0127] S3.2: Calculate the gradient change ΔG between adjacent frames k (x,y), the specific formula is as follows:

[0128] ΔG k (x,y)=|G k+1 (x,y)-G k (x,y)| (10)

[0129] In the formula, G k (x, y) represents the gradient amplitude of the marker in the current image frame, G k+1 (x, y) represents the gradient amplitude of the marker in the next image frame.

[0130] S3.3: Gradient normalization, calculate the mean value of the gradient change between adjacent frames. The specific formula is as follows:

[0131]

[0132] Where N represents the total number of pixels in the current image frame.

[0133] S3.4: Construct a learning rate μ related to the gradient change between adjacent frames 2 , the specific formula is as follows:

[0134]

[0135] In the formula, μ 0 is the base learning rate of the filter template.

[0136] S3.5: Calculate the filter template learning rate μ based on the movement speed of the marker and the gradient change between adjacent frames. The specific formula is as follows:

[0137] μ=λ 1 μ 1 +λ 2 μ 2 (13)

[0138] In the formula, λ 1 and λ 2 is the weight coefficient and satisfies λ 1 <λ 2 .

[0139] According to the filter template learning rate, the filter coefficient of the relevant kernel filter is updated. The specific formula is as follows:

[0140] α k =(1-μ)α k +μαk-1 (14)

[0141] In the formula, α k is the filter coefficient of the current image frame, α k-1 is the filter coefficient of the previous image frame, and the subscript k-1 represents the adjacent previous image frame.

[0142] S4: The swing angle measuring device for the UAV lifting system according to S1, wherein the lightweight marker is a reflective ball, the edge features of the marker are improved by a grayscale enhancement method, and then the outline of the marker is extracted by a gradient search, and the center position of the marker is calculated, specifically comprising the following sub-steps:

[0143] S4.1: Implement grayscale multiplication enhancement processing for the region of interest and construct a binarization judgment criterion. The specific formula is as follows:

[0144]

[0145] In the formula, I(x k ,y k ) represents the gray value corresponding to the marker, I Bin (x k ,y k ) represents the binary image, and T is the threshold.

[0146] S4.2: Based on the center position of the region of interest, the gradient search is used to expand the search outward, the grayscale gradient value of each search point is calculated, and the point set with the largest gradient value is determined as the edge point set of the marker. The least squares method is used to fit the obtained edge point set. The specific formula is as follows:

[0147]

[0148] Where J is the search length, I(j) is the grayscale value of the jth pixel, argmax(G i ) is the pixel distance between the maximum point of the gray gradient in the i direction and the starting point, is the starting position of the search, E i Represents the edge point set in direction i.

[0149] S4.3: Select two edge point pairs with equal gradient magnitude and opposite direction on the fitted circle, and connect the edge point pairs to form a diameter line segment, such as Figure 3 As shown, let A and B be the edge points on the circle, and the coordinates of A and B are (x a ,y a ) and (x b ,y b ), calculate the coordinates of the midpoint of the diameter segment (x c ,y c), the specific formula is as follows:

[0150]

[0151] Where r is the radius of the circle, the subscript i can be a or b, and multiple sets of midpoint coordinates are obtained repeatedly, and the arithmetic mean of all midpoint coordinates is calculated as the final center coordinate of the circle.

[0152] S5: Establish a geometric relationship model between the body and the load, take the fixed length of the suspension rope as prior knowledge, use the trigonometric function relationship to calculate the depth information of the load in the camera coordinate system, and then obtain the swing angle information, which specifically includes the following sub-steps:

[0153] S5.1: According to the coordinate system of the drone lifting system described in S1, the two-dimensional pixel coordinates (u, v) of the center of the marker circle are converted to the three-dimensional world coordinates (X w ,Y w ,Z w ) can be described as:

[0154]

[0155] In the formula, [] -1 Represents the inverse of a matrix.

[0156] S5.2: Based on the system prior knowledge, the depth information of the center of the marker in the camera coordinate system is approximately calculated. The specific formula is as follows:

[0157]

[0158] In the camera coordinate system, the normalized direction vector (X c ′,Y c ′,1) can be obtained from the pixel coordinates (u,v) and the camera intrinsic parameter matrix M:

[0159]

[0160] In the formula, (X c ′,Y c ′,1) represents the direction of the marker’s center position in the camera coordinate system.

[0161] According to the geometric relationship, the three-dimensional coordinates (X c ,Y c ,Z c ) can be expressed as:

[0162]

[0163] Where σ is the proportionality coefficient and the depth information Z of the center of the marker c Directly proportional.

[0164] Calculate the depth information of the center of the marker in the camera coordinate system. The specific formula is as follows:

[0165]

[0166] Where L is the fixed length of the suspension rope.

[0167] In the world coordinate system, the three-dimensional coordinates of the center of the marker can be further written as:

[0168]

[0169] In the formula, u 0 、v 0 、f x 、f y Both can be uniquely determined by the camera intrinsic parameter matrix M.

[0170] S5.3: Calculate the load swing angle information. The algorithm flow is as follows: Figure 4 As shown, the specific formula is as follows:

[0171]

[0172] Where θ is the swing angle of the load in the world coordinate system.

[0173] The present invention is verified by combining the experimental results below: Figure 5 A scene diagram of the UAV lifting system experimental platform built for the laboratory. The platform has 8 optical motion capture cameras arranged on the top of the frame. Since the motion capture system calculates the swing angle by tracking markers, the actual load can be omitted in the experiment. Only reflective markers need to be installed at the corresponding positions to accurately obtain the spatial position of the UAV and its hanging markers, thereby calculating the swing angle information.

[0174] Before the experiment begins, the monocular camera installed under the drone base is calibrated. The calibration process is as follows: Figure 6 The standard black and white chessboard pattern shown is used as the calibration template, where the size of the black and white grid is 20mm*20mm. After calibration calculation, the internal and external parameters of the monocular camera are obtained as shown in Table 1.

[0175] Table 1 Monocular camera internal and external parameters

[0176]

[0177] Preferably, the rope length L is set to 0.5 m, and the basic learning rate μ of the filter template is set to 0 is 0.01, and the learning rate weight coefficient λ 1 =0.2,λ 2=0.8, scale factor of the region of interest s 1 =1.5,s 2 =1.5, the sampling time interval Δt between adjacent image frames is 15ms, Figure 7 The processing effects of different methods under clear and blurred conditions are shown: (a) KCF-MC method effect diagram under clear conditions; (b) MS-HF method effect diagram under clear conditions; (c) the method of the present invention effect diagram under clear conditions; (d) KCF-MC method effect diagram under blurred conditions; (e) MS-HF method effect diagram under blurred conditions; (f) the method of the present invention effect diagram under blurred conditions. Table 2 gives the processing result data of the method described in the embodiment of the present invention and the comparative method.

[0178] Table 2 Comparison of tracking performance of different methods under clear and blurred conditions

[0179]

[0180] It can be seen from the comparative data in Table 2 that the method of the present invention is superior to the existing KCF-MC method and MS-HF method in terms of three performance indicators: tracking success rate, positioning result and average time per frame.

[0181] (1) The tracking success rate of the proposed method under image blur conditions reaches 0.96, which is significantly better than the existing methods and higher than the best MS-HF method (tracking success rate 0.88) and KCF-MC method (tracking success rate 0.75) among the comparison methods. The tracking success rate is increased by 9.09% compared with the MS-HF method. The proposed method can effectively predict and track the position of the marker, and can timely update the filter template during the rapid movement of the marker, and has stronger tracking performance under motion blur conditions.

[0182] (2) The method of the present invention maintains stable positioning performance under both clear and blurred image conditions, effectively overcoming the motion blur problem caused by the rapid swing of the marker. In contrast, the KCF-MC method can maintain a low average center error of the marker under clear conditions, but its error increases sharply under blurred conditions, while the MS-HF method is completely unable to complete the positioning of the marker under blurred conditions.

[0183] (3) The average processing time of the method of the present invention under the condition of image blur is 0.01826s, which has a faster image processing speed. Compared with other methods, the average processing time of this method is 43.69% shorter than that of the KCF-MC method and nearly 28% shorter than that of the MS-HF method. At the same time, the fast image processing capability ensures that this method can measure the swing angle of the load in real time.

[0184] Figure 8The figure shows the comparison between the swing angle results measured by the present invention and the swing angle data obtained by the motion capture system. It can be seen from the figure that the swing angle measured by the present method (solid line) is highly consistent with the result of the motion capture system (dash-dotted line). Except for some peak points, the waveforms and amplitudes of the two curves show good consistency, indicating that the present method can accurately reflect the swing state of the load, which fully confirms the effectiveness and reliability of the present method in practical applications.

[0185] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to be limiting. Although the technical solutions of the present invention have been described in detail with reference to preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the purpose and scope of the present invention, and all of them should be included in the protection scope of the present invention.

Claims

1. A method for measuring the swing angle of a UAV lifting system to solve visual motion blur, characterized in that: The following steps are involved: S1: Build the swing angle measurement device of the UAV lifting system and establish the coordinate system of the UAV lifting system, which specifically includes the following sub-steps: S1.1: A suspension rope is provided at the center of the drone bottom plate for suspending the load, and a lightweight marker is fixed above the load. The marker is located directly above the connection between the suspension rope and the load; S1.2: A monocular camera is installed at the end of the drone bottom plate. Specifically, the monocular camera is installed tilted downward and can be adjusted within a range of plus or minus 45 degrees based on vertical downward in the vertical plane to ensure that its field of view completely covers the movement area of ​​the payload; S1.3: Considering that the monocular camera is fixedly installed on the UAV body, the coordinate system of the UAV lifting system consists of four relatively independent parts, namely the world coordinate system, the camera coordinate system, the image coordinate system and the pixel coordinate system; S2: Based on the kernel correlation filtering algorithm of the fused Kalman filter, the position of the marker in the subsequent image frames is accurately predicted by considering the motion state information of the marker, and the size of the region of interest is dynamically adjusted according to the motion speed of the marker, while adding boundary constraints. The specific implementation steps are as follows: S2.1: Use Kalman filter to predict the motion state of the marker and define the marker state vector S k =[x k x k 'x k ″x k ″′y k y k ′y k ″y k ″′] Τ , where (x k ,y k ) is the initial coordinate of the marker, (x k ′,y k ′) is the velocity component of the marker along the x-axis and y-axis, (x k ″,y k ″) is the acceleration component of the marker along the x-axis and y-axis, (x k ″′,y k ″′) is the acceleration component of the marker along the x-axis and y-axis, and the subscript k represents the current image frame; S2.2: Initialize the kernel correlation filter algorithm by inputting the first frame image, consider the motion state information of the marker and use the Kalman filter to predict the position of the marker in the next frame. The specific formula is as follows: In the formula, is the predicted coordinate of the marker, Δt is the sampling time interval between adjacent image frames, and the subscript k+1 represents the adjacent next image frame; S2.3: Use a multi-scale strategy to adjust the region of interest. The initial size of the region of interest is determined by the first frame of image. According to the velocity components of the marker along the x-axis and y-axis described in S2.1, calculate the motion velocity v of the marker in the current image frame. k , the specific formula is as follows: S2.4: Set speed threshold v th To limit the marker speed range, dynamically adjust the size of the region of interest according to the marker speed. The judgment condition of the region of interest size is: Where W k ×H k is the size of the region of interest, W0 and H0 are the width and height of the region of interest in the initial frame, respectively. k and H k are the width and height of the region of interest in the current frame, s1 and s2 are scale coefficients; S2.5: Add boundary constraints. If the adjustment of the ROI exceeds the set image boundary, the ROI is corrected again. The boundary conditions are as follows: Where W max and H max are the horizontal and vertical resolutions of the image, respectively. and are the horizontal and vertical coordinates of the center point of the region of interest in the image, respectively; S3: In the process of continuous frame processing, the learning rate of the filter template is adaptively adjusted according to the movement speed of the marker and the blur degree of the marker edge, which specifically includes the following sub-steps: S3.1: Construct a learning rate μ1 related to the marker speed. The specific formula is as follows: In the formula, v k Indicates the moving speed of the marker in the current image frame; S3.2: Calculate the gradient change ΔG between adjacent frames k (x,y), the specific formula is as follows: In the formula, G k (x, y) represents the gradient amplitude of the marker in the current image frame, G k+1 (x, y) represents the gradient amplitude of the marker in the next image frame; S3.3: Gradient normalization, calculate the mean value of the gradient change between adjacent frames. The specific formula is as follows: Where N represents the total number of pixels in the current image frame; S3.4: Construct a learning rate μ2 related to the gradient change between adjacent frames. The specific formula is as follows: Where μ0 is the basic learning rate of the filter template; S3.5: Calculate the filter template learning rate μ based on the movement speed of the marker and the gradient change between adjacent frames. The specific formula is as follows: μ=λ1μ1+λ2μ2 (9) In the formula, λ1 and λ2 are weight coefficients, and they satisfy λ1<λ2; S4: The swing angle measuring device for the UAV lifting system according to S1, wherein the lightweight marker is a reflective ball, the edge features of the marker are improved by a grayscale enhancement method, and then the outline of the marker is extracted by a gradient search, and the center position of the marker is calculated, specifically comprising the following sub-steps: S4.1: Implement grayscale multiplication enhancement processing for the region of interest and construct a binarization judgment criterion. The specific formula is as follows: In the formula, I(x k ,y k ) represents the gray value corresponding to the marker, I Bin (x k ,y k ) represents the binary image, T is the threshold; S4.2: Based on the center position of the region of interest, the gradient search is used to expand the search outward, the grayscale gradient value of each search point is calculated, the point set with the largest gradient value is determined as the edge point set of the marker, and the obtained edge point set is fitted using the least squares method; S4.3: Select two edge point pairs with equal gradients and opposite directions on the fitted circle, connect the edge point pairs to form a diameter segment, calculate the midpoint coordinates of the diameter segment, repeat multiple times to obtain multiple sets of midpoint coordinates, and calculate the arithmetic mean of all midpoint coordinates as the final center coordinates of the circle; S5: Establish a geometric relationship model between the body and the load, take the fixed length of the suspension rope as prior knowledge, use the trigonometric function relationship to calculate the depth information of the load in the camera coordinate system, and then obtain the swing angle information, which specifically includes the following sub-steps: S5.1: According to the coordinate system of the drone lifting system described in S1, the two-dimensional pixel coordinates (u, v) of the center of the marker circle are converted to the three-dimensional world coordinates (X w ,Y w ,Z w ) can be described as: Where R is the rotation matrix from the camera coordinate system to the world coordinate system, T is the translation component from the camera coordinate system to the world coordinate system, and Z c is the depth information of the center of the marker in the camera coordinate system, [] -1 represents the inverse of the matrix, f is the focal length of the monocular camera, and f x and f y is the pixel size, u0 and v0 are the optical center positions of the monocular camera image; S5.2: Calculate the depth information of the center of the marker in the camera coordinate system based on the system prior knowledge. The specific formula is as follows: In the formula, (X c ,Y c ) is the plane coordinate of the center of the marker in the camera coordinate system, and L is the fixed length of the rope; S5.3: Calculate the load swing angle information. The specific formula is as follows: Where θ is the swing angle of the load in the world coordinate system.

Images