Method and device for flexible wing deformation detection and motion reconstruction based on binocular vision

Abstract

The application provides a flexible wing deformation detection and motion reconstruction method and device based on binocular vision, which comprises a binocular vision detection method, an adaptive KCF tracking algorithm, a binocular camera synchronous correction method and an experimental device. The binocular vision detection method can reconstruct a motion model of the flexible wing and detect the deformation of the wing. The adaptive KCF tracking algorithm can stably track and locate the spatial position of the whole and local area of the moving wing. The synchronous error correction method can correct the time difference of the asynchronously triggered binocular cameras and realize the synchronization of the images shot by the binocular cameras. The experimental device comprises binocular cameras, a flexible wing, markers and a binocular synchronous correction device. The application improves the tracking accuracy of the high-speed moving target through the adaptive KCF tracking algorithm and the synchronous correction method, reduces the binocular camera synchronous error, improves the flexible wing deformation detection accuracy and motion model reconstruction accuracy, and reduces the cost of the measuring equipment.

Description

Technical Field

[0001] This invention relates to the fields of high-precision optical measurement and flapping-wing flying robots, specifically to a method and apparatus for detecting deformation and reconstructing motion of flexible wings based on binocular vision. Background Technology

[0002] The lift and thrust of flapping-wing robots are primarily generated by their flapping wings, which deform due to inertia and aerodynamic forces. The deformation of the flexible wings significantly affects the lift-thrust coefficient and can improve the robot's flight performance. To further investigate the flight mechanism of flapping-wing robots at low Reynolds number flow velocities and optimize wing design, it is necessary to detect flexible wing deformation and reconstruct a motion model of the flexible wings. Therefore, wing deformation detection is a key technology for the research and development of flapping-wing flying robots.

[0003] Wing deformation detection technology can be divided into contact measurement and non-contact measurement. Traditional contact measurement increases the load on the wing and may affect the mechanical properties of the wing structure itself, leading to a decrease in measurement accuracy. Binocular vision measurement, as a novel non-contact optical measurement technology, does not interfere with the moving wing. Addressing the challenge of matching wings due to low texture and few features, there are two main solutions: using multiple high-resolution cameras and projection devices to achieve global wing matching, and tracking and locating manual markers on the wings to achieve local wing matching. Global matching increases experimental costs, and the marker scale for local matching changes with the wing flapping angle, increasing tracking difficulty. The measurement accuracy of binocular vision is related to the synchronization accuracy of the binocular cameras. The system time of the cameras is independent of each other. Most industrial cameras achieve time synchronization through external trigger signals and software commands, but cameras without matching I / O ports can only be manually triggered asynchronously, resulting in a significant reduction in measurement accuracy.

[0004] Previous researchers recognized the importance of wing deformation detection and employed various techniques to detect it. Furthermore, researchers have proposed a new scheme for binocular synchronous correction.

[0005] Patent CN209727054U discloses a technical solution for integrating fiber optic gratings on the surface of a soft wing and measuring the wing shape. The fiber optic gratings are arranged horizontally and vertically on the wing surface to convert the detected strain signals into deformation information. However, the fiber optic grating lines and their blocking blocks increase the load on the wing. The physical properties of the optical fiber affect the measurement of the physical properties of the wing structure itself.

[0006] Patent CN108163229A designs a device for detecting wing movement information based on a piezoelectric sensor, but the output signal is unstable and the response is poor, resulting in low measurement accuracy.

[0007] Patent CN110207603A discloses a method for measuring the dynamic deformation of thin wings, using the rich veins and folds on the wings as features for digital correlation matching, and achieving full-field measurement of the three-dimensional deformation of the wings through three sets of high-resolution industrial cameras. However, this method cannot effectively extract the features of wings with low texture.

[0008] Patent CN115190288A provides a method and apparatus for simultaneous image acquisition by multiple cameras. Specifically designed for camera setups lacking hardware synchronization interfaces, it uses the image of an LED illuminating as a timestamp for camera synchronization, performing synchronization correction on each industrial camera. However, the LED illuminating process introduces a delay, making the synchronization correction using the LED as a reference signal insufficiently accurate for high-frame-rate cameras capturing fast-moving objects.

[0009] Both patent CN 105157592A and this patent employ binocular vision measurement methods. While the two patents are similar in image correction and stereo matching methods, they also differ significantly:

[0010] Differences between the two patents:

[0011] The camera synchronization methods differ: Patent CN 105157592 A uses an industrial camera equipped with a synchronization cable and synchronization I / O ports for synchronization. This patent uses a non-industrial DJI action camera, which lacks synchronization I / O ports and requires manual activation of both cameras by pressing their buttons separately, resulting in asynchronous data acquisition. A synchronization correction device is then used to calculate the startup time difference between the two cameras and perform synchronization correction on the binocular cameras to achieve synchronized data acquisition. This binocular synchronization correction method is also one of the innovative aspects of this patent.

[0012] The measurement objects differ: CN 105157592A measures a deformable wing, primarily measuring the flexible trailing edge that undergoes active deformation. This patent measures a wing that, due to inertia and aerodynamic forces, exhibits complex deformation in both the spanwise and chordwise directions during flapping. Furthermore, the wing can flap up and down around a rotation axis near the wing root. The binocular system needs to detect not only the amount of wing deformation but also the wing's motion parameters.

[0013] Tracking algorithm: To improve data processing efficiency, this patent proposes an adaptive KCF tracking algorithm to track markers on the wings, avoiding the need for manual matching of each marker point in each frame captured by the left and right cameras. Furthermore, this patent improves the traditional KCF tracking algorithm based on target scale prediction values, enhancing tracking accuracy. The improved adaptive KCF tracking algorithm is also an innovation of this patent. CN 5157592 A does not describe a marker tracking algorithm.

[0014] The construction objects differ: CN 105157592 A uses discrete marker points as samples and employs a cubic polynomial fitting method to construct the upper and lower edges of the flexible trailing edge. This patent, on the other hand, uses discrete marker points as samples and employs an interpolation fitting method to construct the flexible wing surface.

[0015] Circular marker detection algorithm: CN 105157592 A. The circular marker detection method adopts the Hough circle transformation principle, while the circular marker detection method in this paper is based on the least squares method.

[0016] The calculation parameters differ: CN 105157592 A calculates the deformation rate of the flexible trailing edge endpoint using the difference principle based on the endpoint positions at different times. This patent calculates the deformation of the wing based on the wingtip position and the undeformed plane at the same time.

[0017] Therefore, to address the challenges of matching low-texture wings, tracking multi-scale targets, and insufficient measurement accuracy of asynchronously triggered cameras, this invention designs a method and device for flexible wing deformation detection and motion reconstruction based on binocular vision. The improved adaptive KCF tracking algorithm can stably track high-speed moving targets with large scale changes. A binocular synchronization correction method improves the synchronization accuracy of binocular images. This device does not affect wing movement and has high detection accuracy, which is of great significance for exploring the flapping wing flight mechanism and wing optimization. Furthermore, this method and device can be applied to the deformation measurement and motion model construction of other flexible objects. Summary of the Invention

[0018] To address the problems of existing wing detection technologies, such as interference with the measured object, poor detection accuracy, and high hardware configuration and cost of the measurement equipment, this invention proposes a method and device for flexible wing deformation detection and motion reconstruction based on binocular vision. This system does not interfere with wing movement and deformation, can stably track targets and improve binocular synchronization accuracy, and can accurately reconstruct flexible wing models with large range of motion, complex deformation, and high-speed flapping.

[0019] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0020] The method for flexible wing deformation detection and motion reconstruction based on binocular vision includes the following steps:

[0021] S1: The binocular camera acquires images, the fill light is turned on, and the shutter button of the binocular camera is manually triggered. The binocular camera then begins to acquire images.

[0022] S2: Calibrate the stereo camera. Import the calibration image set acquired by the stereo camera into the Sereo CameraCalibration toolbox in Matlab. Use the Zhang Youzheng calibration method to obtain the internal and external camera parameters of the stereo camera. Then, correct the images acquired by the stereo camera based on these parameters to ensure the row alignment of the acquired images.

[0023] S3: Determine the pixel coordinates and three-dimensional coordinates of the circular markers in the acquired and corrected image;

[0024] S4: Calculate the time difference between the two cameras;

[0025] S5: Tracks and locates markers on moving wings;

[0026] S6: Perform binocular synchronization correction based on the calculated time difference;

[0027] S7: Reconstruct the motion model of the wing, using all the markers on the wing as sample points, and construct the wing motion plane through interpolation fitting;

[0028] S8: Calculate the wing deformation. Construct the undeformed plane of the wing using three marked points on the upper surface of the connector at the leading edge of the wing and the tail of the frame. Use the distance between the wingtip marked point and the undeformed plane as the wing deformation.

[0029] As a further improvement to the method of the present invention, step S1 is specifically as follows;

[0030] First, acquire camera calibration images. Place the checkerboard calibration board on the plane of the wings at different flapping angles. The binocular camera records images of the calibration board at different positions. The calibration board remains still for several tens of seconds when it reaches the designated position to ensure that the binocular camera captures the same position of the calibration board under the same acquisition time, and to avoid the influence of the time difference of asynchronous triggering of the binocular camera on the calibration results.

[0031] Then, images of binocular synchronous correction are acquired. By adjusting the universal clamp, the slide rail in the binocular synchronous correction device is parallel to the binocular plane, ensuring that the movement plane of the correction plate is parallel to the binocular plane. The binocular recording LED light is turned on and the correction plate moves in one direction on the slide rail.

[0032] Finally, capture images of wing flapping. Once the acquisition is complete, import the images captured by the camera into the host computer.

[0033] As a further improvement to the method of the present invention, step S3 is specifically as follows;

[0034] First, the Canny operator is used to perform edge detection on the four circular markers in the synchronously corrected image. Based on the detected edge point set, the equation of the ellipse in the image is determined using the least squares method.

[0035] Ax 2+Bxy+Cy 2 +Dx+Ey+F=0

[0036] The sum of squares of the algebraic distances from the point set to the ellipse is minimized as follows:

[0037]

[0038] To minimize min(W·G) 2 Construct the Lagrangian function L(G,λ)=GWW T G T -λ(W T RW-1), the optimal parameters (A,B,C,D,E,F) for ellipse fitting are determined based on the fact that the partial derivative of L(G,λ) with respect to G is zero, and then the coordinates of the ellipse center (x) are obtained. c ,y c ), Major and minor axis lengths (a, b) and rotation angle θ:

[0039]

[0040] Based on the center coordinates (u) of the elliptical projections of the left and right cameras l ,v l ), (u r ,v r The reprojection matrix, composed of the internal and external binocular camera parameters during binocular positioning, yields the spatial coordinate system (X) of the center of the circular marker in the world coordinate system. w ,Y w Z w ).

[0041] As a further improvement to the method of the present invention, step S4 is specifically as follows;

[0042] The right and left cameras are designated as master and slave, respectively.

[0043] First, taking the LED turn-on signal as a reference, the current frame where the LED was off in the previous frame and on in the current frame is taken as the starting image for camera acquisition. Since the time between adjacent frames of the camera is fixed, the time difference between the two cameras is replaced by the frame difference to facilitate the determination of the time difference. Since there is a delay in the LED turn-on time, the frame difference i between the starting images of the left and right cameras is taken as the estimated value of the time difference between the two cameras, thus narrowing the search range for the time difference.

[0044] Secondly, based on the synchronization of images captured by the binocular cameras and the inference of the minimum measurement error, the starting image of the right camera is used as the reference, and the images of the left camera are matched with the starting image of the right camera at intervals of i+j (j=-2,-1…2) frames respectively. The error between the mark spacing and the actual mark spacing is calculated, and the matching left camera image with the smallest error is used as its starting image. The binocular camera interval i+J is determined as the time difference.

[0045] Finally, using the projection points of the marker points in the adjacent frames of the starting frame of the left camera as sample points, n-1 virtual pixels are linearly inserted and matched with the left camera images that are i+J+k / n (k=-n,-n+1…n) frames apart from the right camera images. The marker spacing error is calculated, and the binocular spacing i+j+K / n with the smallest error is determined as the time difference.

[0046] As a further improvement to the method of the present invention, step S5 is specifically as follows;

[0047] In the first frame of the wing flapping image, each marker on the wing is selected, and the HOG features of the image within the box are extracted to initialize the KCF tracker. Edge detection and ellipse recognition techniques from S3 are used to determine the coordinates of the ellipse center within the box, and the circumscribed rectangle of the ellipse is drawn based on the ellipse parameters, and its area S is calculated. The input image is then used as the current frame, and through cyclic shifting, candidate regions consisting of the sample set to be evaluated are generated. The target region is used as the positive sample, and the background region as the negative sample to generate a classifier. The classifier is trained using a ridge regression equation to construct a minimum sample set x. i and corresponding regression label y i Objective function for the squared error between:

[0048] f(z) = w T Z

[0049] The linear ridge regression of the objective function is expressed as:

[0050] min∑(f(x i )-y i ) 2 +λ||w|| 2

[0051] Where (f(x) i )-y i ) 2 Let λ be the loss function, and w|| be the loss function. 2 λ is the penalty term, λ is the regularization parameter to prevent overfitting, and w is the weight coefficient. By taking the derivative with respect to x, the optimal solution of the ridge regression equation is obtained.

[0052] The trained classifier calculates the similarity between candidate regions and the tracking target, and determines the candidate region with the highest similarity as the target region. Since there is a scale transformation on the markings on the wings, in order to accurately track the markings on the wings;

[0053] Then, the ratio C of the areas of the bounding rectangles of the corresponding markers in the previous frame and the current frame is calculated. i =S i / S i-1 The scaling value of the tracking box in the next frame is used to incorporate the features of the current frame into the training model for continuous training and updating of the classifier;

[0054] Finally, in the scaled target box of the next frame, the center of the ellipse and the area of ​​the circumscribed rectangle are redefined. By continuously tracking and locating image markers, the pixel coordinates of each marker in each frame of the stereo camera image are accurately obtained.

[0055] As a further improvement to the method of the present invention, step S6 is specifically as follows;

[0056] Using the pixel coordinates of the marker points in adjacent frames of the left camera as sample points, n-1 virtual pixels are linearly inserted. The pixel coordinates of the Kth point are calculated using the following formula:

[0057]

[0058] v represents the horizontal / vertical coordinate of a pixel. Then, the image coordinates of the corresponding markers from the left and right cameras at intervals of i+j+K / n frames are matched respectively. Based on the principle of binocular stereo imaging, the three-dimensional spatial coordinates of the marker points are calculated to ensure that the binocular acquisition of markers is synchronized.

[0059] This invention provides an apparatus for flexible wing deformation detection and motion reconstruction based on binocular vision. The apparatus comprises a binocular camera, a supplementary light, a wing drive device and frame, wings, manual markers, and a binocular synchronization correction device. The binocular camera consists of two motion cameras. The supplementary light provides an external light source for the detection environment, ensuring the brightness of the acquired images. The wing drive device and frame are used to drive wing flapping and provide mounting positions for the drive device and wings, respectively. The wing drive device and frame include a spatial four-bar linkage, a reduction gear pair, a motor, and the frame. The wings include a wing rotation axis, a wing leading edge connector, a carbon fiber rod, and a wing membrane. Driven by the drive device, the wings reciprocate around the wing rotation axis. The wing leading edge connector is connected to the carbon fiber rod. The manual markers include mark 1 on the end face of the wing leading edge connector, mark 2 on the end face of the wing leading edge connector, a mark on the tail of the frame, and a mark on the wing. Markings are affixed to the wingtips, with the artificial markings evenly distributed across the upper surface of the wings. Markings are also affixed to the wingtips, and three additional markings are affixed to the upper surface of the wingtips, the tail of the frame, and the upper surface of the wingtips (mark 1, mark 2, and mark 3 on the tail of the frame). The binocular synchronization correction device includes a slide rail, a correction plate, a protractor, a universal clamp, LED lights, and a control circuit. The protractor measures the attitude angle between the slide rail and the binocular camera plane. The universal clamp is adjusted so that the slide rail is parallel to the binocular camera plane. The correction plate slides on the slide rail and has a CAD drawing template affixed to it. The template has four circular markings with a fixed and known spacing. The measurement results of the marking spacing are used to evaluate the binocular synchronization correction accuracy and assist in calculating the precise binocular time difference. The LED light source activation signal is used to determine the starting frame of the binocular camera and estimate the binocular camera time difference.

[0060] Beneficial effects:

[0061] (1) The present invention uses a non-contact measurement method based on binocular vision to detect wing deformation, which will not interfere with wing movement and deformation, and improves the accuracy of wing deformation measurement.

[0062] (2) By using artificial markers on the wings for positioning and tracking, the problem of being unable to extract matching features due to low texture of the wings is solved. This also improves the speed and accuracy of image data processing.

[0063] (3) Based on the LED light activation as a synchronization timestamp, this invention adds a new binocular correction device and method to solve the problem of large synchronization errors caused by the delay in LED activation, and further improves the synchronization accuracy of binocular or even multiple cameras. At the same time, it provides a new synchronization method for cameras without synchronization I / O ports.

[0064] (4) This invention addresses the difficulty in accurately tracking targets that move at high speeds and undergo large scale changes. It proposes an adaptive KCF tracking algorithm to improve upon the traditional KCF tracking algorithm. By predicting the target scaling scale, the tracking box can be scaled, enabling stable tracking of multi-scale targets and improving tracking accuracy.

[0065] (5) The equipment used in this invention is inexpensive and easy to deploy, and can be applied to difficult detection tasks such as high-speed movement of targets, complex deformation, and large size changes. Attached Figure Description

[0066] Figure 1 This is an overall schematic diagram of the flexible wing deformation detection and motion reconstruction device in an example of the present invention;

[0067] Figure 2 This is a schematic diagram of the binocular synchronous correction device in an example of the present invention;

[0068] Figure 3 This is a schematic diagram of the distribution of marking points on the wings in an example of the present invention;

[0069] Figure 4 This is a flowchart illustrating the flexible wing detection and motion reconstruction process in an example of the present invention.

[0070] Figure 5 This is a flowchart of the binocular synchronization correction steps in an example of the present invention;

[0071] Figure 6 This is a line graph showing the binocular synchronization correction results in an example of the present invention;

[0072] Figure 7 This is a flowchart of the adaptive KCF tracking algorithm in an example of the present invention;

[0073] List of reference numerals in the attached diagram:

[0074] 1. Binocular camera; 2. Fill light; 3. Wing drive device and frame; 3-1. Spatial four-bar linkage; 3-2. Reduction gear pair; 3-3. Motor; 3-4. Frame; 4. Wing; 4-1. Wing rotation shaft; 4-2. Wing leading edge connector; 4-3. Carbon fiber rod; 4-4. Wing membrane; 5. Manual marking; 5-1. Mark 1 on the end face of the wing leading edge connector; 5-2. Mark 2 on the end face of the wing leading edge connector; 5-3. Mark on the tail of the frame; 5-4. Mark pasted on the wingtip; 6. Binocular synchronous correction device; 6-1. Slide rail; 6-2. Correction plate; 6-3. Protractor; 6-4. Universal clamp; 6-5. LED beads and control circuit; 6-5-1. LED beads; 6-5-2. LED control circuit. Detailed Implementation

[0075] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0076] Reference Figure 1 , Figure 2 and Figure 3 The flexible wing deformation detection and motion reconstruction device consists of a binocular camera 1, a supplementary light 2, a wing drive device and frame 3, wings 4, manual markers 5, and a binocular synchronization correction device. The binocular camera 1 comprises two DJI OSMO ACTION action cameras; lacking a synchronization I / O port, it must be manually activated by pressing a camera button. The supplementary light 2 provides an external light source to ensure the brightness of the acquired images. The wing drive device and frame 3 are used to drive wing flapping and provide mounting positions for the drive device and wings, respectively. Driven by the drive device, wings 4 reciprocate around a rotation axis 4-1. The wing leading edge connector 4-2 connects to a carbon fiber rod 4-3. Due to the support of the carbon fiber rod and the constraint at the wing root, the stiffness distribution of the wing surface 4-4 is inconsistent, causing complex deformations in the spanwise and chordal directions of the flapping wings.

[0077] Reference Figure 1 and Figure 3 The artificial markings 5 ​​are evenly distributed on the upper surface of the wing, with marking 5-4 affixed to the wingtip. Additionally, three markings 5-1, 5-2, and 5-3 are affixed to the upper surface of the wing leading edge connector 4-2 and the tail of the frame 3-4. These three markings define the undeformed plane of the wing, and the distance between the wingtip marking point 5-4 and the undeformed plane is defined as the wing deformation amount.

[0078] refer to Figure 2The binocular synchronization correction device 6 includes a slide rail 6-1, a correction plate 6-2, a protractor 6-3, a universal clamp 6-4, LED lights, and a control circuit. The protractor 6-3 measures the attitude angles of the slide rail 6-1 and the plane of the binocular camera 1. The universal clamp 6-4 is adjusted so that the slide rail 6-1 is parallel to the plane of the binocular camera 1. The correction plate 6-2 can slide on the slide rail 6-1. A CAD drawing template is affixed to the correction plate 6-2. The template has four circular markers with a fixed and known spacing. The measurement results of the marker spacing are used to evaluate the binocular synchronization correction accuracy and assist in calculating the precise time difference between the two cameras. The LED light activation signal is used to determine the starting frame of the binocular camera and the estimated value of the time difference between the two cameras.

[0079] refer to Figure 4 The flexible wing deformation detection and motion reconstruction method is as follows: First, the binocular camera is manually activated, and binocular calibration patterns, binocular synchronization correction patterns, and wing flapping patterns are acquired sequentially. These acquired patterns are then imported into a host computer for processing. The binocular cameras are calibrated using the Zhang Youzheng calibration method to obtain their internal and external parameters. The pixel coordinates of the elliptical projection center of the circular marker are determined by edge detection and elliptical recognition on the calibration plate. Subsequently, the binocular synchronization correction device 6 calculates the time difference between the two eyes. An adaptive KCF tracking algorithm and an elliptical recognition algorithm based on least squares are used to track and locate the markers on the wing, obtaining the pixel coordinates of each marker in each frame. Based on the binocular time difference calculated by S4, the binocular wing flapping images are corrected to obtain the true three-dimensional spatial coordinates of the synchronously acquired markers. Using the markers as sample points, a wing motion model is constructed through interpolation fitting. Furthermore, an undeformed wing plane is constructed using the marker points on the wing's leading edge connector and the tail of the frame. The distance between the wingtip marker and the undeformed plane is used as the wing deformation.

[0080] refer to Figure 4 and Figure 5The binocular synchronization correction device 6 corrects the binocular synchronization method as follows: the binocular synchronization correction device is located within the field of view of the binocular camera. During the recording process of the binocular camera, the LED light is lit, and then the correction plate moves to the right end of the slide rail. According to the binocular acquisition synchronization correction image in S1, the binocular time difference calculation step in S4 is divided into three steps. Step 1 determines the starting frame of the left and right cameras based on the LED light lighting signal, and uses the difference i between the starting frames of the binocular cameras as the estimated value of the binocular time difference. Step 2 and Step 3 are based on the synchronization of the images captured by the binocular cameras and the inference of the minimum measurement error. Step 2 matches the left and right images at frame intervals i+j (j=-2,-1…2) respectively, calculates the mark spacing measurement error, and determines the frame interval i+J with the smallest error as the binocular camera time difference. Step 3 uses the mark projection points of adjacent frames of the left camera as samples, linearly inserts n-1 virtual sample points, calculates the matching spacing measurement error of the frame difference i+J+k / n, and determines the frame difference i+J+K / n with the smallest error as the binocular camera time difference. Finally, based on the pixel coordinates of each marker in each frame obtained from the left and right cameras by S5, the projection point trajectory of the wing marker recorded by the left camera is interpolated to match the image coordinates of the corresponding markers of the left and right cameras at intervals of i+j+K / n frames. For example... Figure 6 The measurement results of the marker spacing error after calculating the time difference of the uncorrected, Step 1, Step 2 and Step 3 binocular synchronization correction provide strong evidence for further improving the binocular synchronization accuracy of the present invention, and have high value and significance for further improving the synchronization technology of binocular and even multi-view cameras.

[0081] refer to Figure 4 and Figure 7 The flowchart of the adaptive KCF tracking algorithm is as follows: Input the image to be tracked, determine if the image is the first frame, and select each target marker in the first frame image, extracting the HOG features of the image within the selected frame to initialize the tracker. Then, sequentially use the input images as the current frames (if the current frame is not the first frame), generating candidate regions composed of the sample set to be evaluated through cyclic shifting. Use the target regions as positive samples and the background regions as negative samples to generate a classifier. The classifier is trained using the ridge regression equation to construct a minimum sample set x. i and corresponding regression label y i Objective function for the squared error between:

[0082] f(z) = w T Z

[0083] The linear ridge regression of the objective function is expressed as:

[0084] min∑(f(x i )-y i ) 2 +λ||w|| 2

[0085] Where (f(x) i )-y i ) 2 Let λ be the loss function, and w|| be the loss function. 2 λ is the penalty term, λ is the regularization parameter to prevent overfitting, and w is the weight coefficient. The optimal solution of the ridge regression equation is obtained by differentiating with respect to x. The classifier calculates the similarity between candidate regions and the tracked target. The candidate region with the highest similarity is determined as the target region, and the HOG features of the image are extracted. Edge detection and ellipse recognition techniques from S3 are used to determine the coordinates of the ellipse center in the frame, and the circumscribed rectangle of the ellipse is drawn according to the ellipse parameters, and the area S of the circumscribed rectangle is calculated. Since there is a scale transformation on the wing markings, as a further improvement to the tracking algorithm of this invention, the ratio C of the circumscribed rectangle area of ​​the corresponding marking in the previous frame and the current frame is used for accurate tracking. i =S i / S i-1 The scaling value is used to determine the tracking bounding box size for the next frame, and features from the previous frame are incorporated into the training model for continuous training and updating of the classifier. Subsequently, the center of the ellipse and the area of ​​the bounding rectangle are redefined within the scaled target bounding box of the next frame. By continuously tracking and locating image markers, the pixel coordinates of each marker in each frame of the stereo camera image are accurately obtained.

[0086] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A method for flexible wing deformation detection and motion reconstruction based on binocular vision, characterized in that: The specific steps are as follows: S1: The binocular camera acquires images, the fill light is turned on, and the shutter button of the binocular camera is manually triggered. The binocular camera then begins to acquire images. S2: Calibrate the stereo camera. Import the calibration image set acquired by the stereo camera into the Sereo CameraCalibration toolbox in Matlab. Use the Zhang Youzheng calibration method to obtain the internal and external camera parameters of the stereo camera. Then, correct the images acquired by the stereo camera based on these parameters to ensure the row alignment of the acquired images. S3: Determine the pixel coordinates and three-dimensional coordinates of the circular markers in the acquired binocular synchronous correction image; S4: Calculate the time difference between the two cameras; The specific steps of step S4 are as follows; The right and left cameras are designated as master and slave, respectively. First, taking the LED turn-on signal as a reference, the current frame where the LED was off in the previous frame and on in the current frame is taken as the starting image for camera acquisition. Since the time between adjacent frames of the camera is fixed, the time difference between the two cameras is replaced by the frame difference to facilitate the determination of the time difference. Since there is a delay in the LED turn-on time, the frame difference i between the starting images of the left and right cameras is taken as the estimated value of the time difference between the two cameras, thus narrowing the search range for the time difference. Secondly, based on the inference of synchronizing images captured by the binocular cameras and minimizing measurement error, the starting image of the right camera is used as the reference. It is matched with the left camera images at intervals of i+j (j=-2, -1…2) frames from the starting image of the right camera. The error between the marker spacing and the actual marker spacing is calculated. The matching left camera image with the smallest error is used as its starting image, and the binocular camera interval i+J is determined as the time difference. Finally, using the projection points of the marker points in the adjacent frames of the starting frame of the left camera as sample points, n-1 virtual pixels are linearly inserted and matched with the left camera images that are i+J+k / n (k=-n, -n+1…n) frames apart from the right camera images. The marker spacing error is calculated, and the binocular spacing i+j+K / n with the smallest error is determined as the time difference. S5: Tracks and locates markers on moving wings; S6: Perform binocular synchronous correction based on the calculated time difference; S7: Reconstruct the motion model of the wing, using all the markers on the wing as sample points, and construct the wing motion plane through interpolation fitting; S8: Calculate the wing deformation. Construct the undeformed plane of the wing using three marked points on the upper surface of the connector at the leading edge of the wing and the tail of the frame. Use the distance between the wingtip marked point and the undeformed plane as the wing deformation.

2. The method for flexible wing deformation detection and motion reconstruction based on binocular vision according to claim 1, characterized in that: The specific steps of step S1 are as follows; First, camera calibration images are acquired. The checkerboard calibration board is placed on the plane of the wings at different flapping angles. The binocular camera records images of the calibration board at different positions. The calibration board remains stationary for several tens of seconds after reaching the designated position to ensure that the positions of the calibration board captured by the binocular camera at the same time are consistent, thus avoiding the influence of the time difference of asynchronous triggering of the binocular camera on the calibration results. Then, binocular synchronous correction images are acquired. By adjusting the universal clamp, the slide rail in the binocular synchronous correction device is parallel to the binocular plane, ensuring that the movement plane of the correction plate is parallel to the binocular plane. The binocular recording system captures images of the LED lights being turned on and the correction plate moving in one direction on the slide rail. Finally, capture images of wing flapping. Once the acquisition is complete, import the images captured by the camera into the host computer.

3. The method for flexible wing deformation detection and motion reconstruction based on binocular vision according to claim 1, characterized in that: The specific steps of step S3 are as follows; First, the Canny operator is used to perform edge detection on the four circular markers in the synchronously corrected image. Based on the detected edge point set, the equation of the ellipse in the image is determined using the least squares method. ； The sum of squares of the algebraic distances from the point set to the ellipse is minimized as follows: ; To minimize min (Wּ∙G) 2 Construct the Lagrangian function L(G, λ)=GWW T G T −λ(W T RW−1), the optimal parameters (A, B, C, D, E, F) for ellipse fitting are determined based on the fact that the partial derivative of L(G, λ) with respect to G is zero, and then the coordinates of the ellipse center (x) are obtained. c , y c ), Major and minor axis lengths (a, b) and rotation angle θ: ； Based on the center coordinates (u) of the elliptical projections of the left and right cameras l , v l ), (u r , v r The reprojection matrix, composed of the internal and external binocular camera parameters during binocular positioning, yields the spatial coordinate system (X) of the center of the circular marker in the world coordinate system. w ,Y w Z w ).

4. The method for flexible wing deformation detection and motion reconstruction based on binocular vision according to claim 1, characterized in that: The specific steps of step S5 are as follows; In the first frame of the wing flapping image, each marker on the wing is selected, and the HOG features of the image within the box are extracted to initialize the KCF tracker. Edge detection and ellipse recognition techniques from S3 are used to determine the coordinates of the ellipse center within the box, and the circumscribed rectangle of the ellipse is drawn based on the ellipse parameters, and its area S is calculated. The input image is then used as the current frame, and through cyclic shifting, candidate regions consisting of the sample set to be evaluated are generated. The target region is used as the positive sample, and the background region as the negative sample to generate a classifier. The classifier is trained using a ridge regression equation to construct a minimum sample set x. i and corresponding regression label y i Objective function for the squared error between: f(z)=w T WITH The linear ridge regression of the objective function is expressed as: ; Where (f(x) i ) − y i ) 2 Let λ be the loss function, and w|| be the loss function. 2 λ is the penalty term, λ is the regularization parameter to prevent overfitting, and w is the weight coefficient. By taking the derivative with respect to x, the optimal solution of the ridge regression equation is obtained. The trained classifier calculates the similarity between candidate regions and the tracking target, and determines the candidate region with the highest similarity as the target region. Since there is a scale transformation on the markings on the wings, in order to accurately track the markings on the wings; Then, the ratio C of the areas of the bounding rectangles of the corresponding markers in the previous frame and the current frame is calculated. i =S i / S i-1 The scaling value of the tracking box in the next frame is used to incorporate the features of the current frame into the training model for continuous training and updating of the classifier; Finally, in the scaled target box of the next frame, the center of the ellipse and the area of ​​the circumscribed rectangle are redefined. By continuously tracking and locating image markers, the pixel coordinates of each marker in each frame of the stereo camera image are accurately obtained.

5. The method for flexible wing deformation detection and motion reconstruction based on binocular vision according to claim 1, characterized in that: The specific steps of step S6 are as follows; Using the pixel coordinates of the marker points in adjacent frames of the left camera as sample points, n-1 virtual pixels are linearly inserted, and the pixel coordinates of the Kth point are calculated using the following formula: ; v represents the horizontal / vertical coordinate of a pixel. Then, the image coordinates of the corresponding markers from the left and right cameras at intervals of i+j+K / n frames are matched respectively. Based on the principle of binocular stereo imaging, the three-dimensional spatial coordinates of the marker points are calculated to ensure that the binocular acquisition of markers is synchronized.

6. An apparatus for the flexible wing deformation detection and motion reconstruction method based on binocular vision according to any one of claims 1-5, comprising a binocular camera (1), a supplementary light (2), a wing drive device and frame (3), a wing (4), an artificial marker (5), and a binocular synchronization correction device (6), wherein the binocular camera (1) consists of two motion cameras, the supplementary light (2) provides an external light source for the detection environment to ensure the brightness of the acquired image, the wing drive device and frame (3) are respectively used to drive the flapping of the wing and to provide the installation position of the drive device and the wing, the wing drive device and frame (3) includes a spatial four-bar linkage (3-1), a reduction gear pair (3-2), a motor (3-3), and a frame (3-4), the wing (4) includes a wing rotation shaft (4-1), a wing leading edge connector (4-2), a carbon fiber rod (4-3), and a wing membrane (4-4), characterized in that, under the drive of the drive device, The wing (4) reciprocates around the wing rotation axis (4-1), and the wing leading edge connector (4-2) is connected to the carbon fiber rod (4-3); the artificial marking (5) includes marking 1 (5-1) on the end face of the wing leading edge connector, marking 2 (5-2) on the end face of the wing leading edge connector, marking 5-3 on the tail of the frame, and marking 5-4 pasted at the wing tip. The artificial marking (5) is evenly distributed on the upper surface of the wing, and marking 5-4 is pasted at the wing tip. Three markings are pasted on the upper end face of the wing leading edge connector (4-2) and the tail of the frame (3-4): marking 1 (5-1) on the end face of the wing leading edge connector, marking 2 (5-2) on the end face of the wing leading edge connector, and marking 5-3 on the tail of the frame; the binocular synchronous correction device (6) includes a slide rail (6 -1), calibration plate (6-2), protractor (6-3), universal clamp (6-4) and its LED light and its control circuit (6-5). The protractor (6-3) measures the attitude angle of the slide rail (6-1) and the plane of the binocular camera (1). The universal clamp (6-4) is adjusted so that the slide rail (6-1) is parallel to the plane of the binocular camera (1). The calibration plate (6-2) slides on the slide rail (6-1). A computer drawing software CAD drawing template is pasted on the calibration plate (6-2). There are four circular marks on the template. The spacing between the marks is fixed and known. The measurement result of the mark spacing is used to evaluate the binocular synchronization calibration accuracy and assist in calculating the binocular precise time difference. The turn-on signal of the LED light and its control circuit (6-5) is used to determine the starting frame of the binocular camera and the estimated value of the binocular camera time difference.

Images