Vision-based fruit tree natural vibration characteristic identification and excitation regulation and control method

Abstract

The invention discloses a fruit tree natural vibration characteristic identification and excitation regulation and control method based on vision. Belongs to the technical field of mechanical fruit picking. The method comprises the four steps of video acquisition, fruit natural vibration frequency information extraction based on machine vision, fruit tree structure parameter extraction based on machine vision, and optimization of the optimal force bearing point of the fruit tree based on space finite element analysis and a physical information deep neural network. Wherein the extraction of the fruit tree structure parameters based on the machine vision is to perform target detection and instance segmentation by using a Mask R-CNN algorithm, reinforce image details in combination with a Canny edge detection technology, and accurately extract geometric parameters such as the crown area, the slenderness ratio of branches and a trunk. According to the method, multiple independent modes of the fruit tree can be accurately separated, and the natural vibration frequency of the fruit can be accurately extracted. According to the innovative scheme, the limitation of a traditional method is broken through, and an efficient, low-cost and high-adaptability technical path is provided for intelligent agriculture and precise fruit tree management.

Description

Technical Field

[0001] The present invention relates to the technical field of mechanical picking, and in particular to a method for identifying and regulating the self-vibration characteristics of fruit trees based on vision. Background Art

[0002] Studying the vibration characteristics of fruit trees, especially the relationship between the application of external force and the matching of the natural frequency of the fruit, is of great significance for optimizing the picking method and fruit management. If the applied vibration frequency is close to the natural frequency of the fruit (i.e., the resonance phenomenon), it may cause the fruit to fall off. Therefore, precise control of vibration parameters can improve picking efficiency while reducing tree damage. In addition, reasonable vibration regulation may also affect the growth process of the fruit and provide a new means for fruit maturity management. Therefore, studying the natural frequency of fruit trees and their optimal force points will help improve the intelligence level of mechanical picking and expand the application of vibration technology in modern agriculture.

[0003] Traditionally, researchers have mainly used contact measurement and non-contact measurement to optimize the force points of vibration picking of fruit trees. Contact measurement relies on sensors such as accelerometers and strain gauges, which are fixed on the surface of fruit trees or picking equipment to collect local vibration data and extract vibration characteristics through signal processing methods such as Fourier transform. Although this method can reflect the local vibration of fruit trees to a certain extent, it has obvious defects: on the one hand, the installation of sensors is invasive and not easy to be effectively fixed on fruit trees, which may affect the growth of fruit trees and increase the complexity of experiments; on the other hand, due to the limited number of measurement points, the collected data can often only represent local areas and cannot fully reflect the dynamic response of the entire fruit tree. Non-contact measurement usually uses laser Doppler vibrometers or optical scanning systems. These devices collect vibration data through laser or optical means, reducing direct interference with fruit trees. However, the collection range of traditional non-contact measurement equipment is limited, and usually only point or local information can be obtained, making it difficult to fully monitor the overall vibration mode of fruit trees. In addition, this type of equipment is expensive, complicated to install and debug, and is easily affected by external factors such as ambient light and wind speed, resulting in insufficient data stability. Summary of the invention

[0004] The purpose of the present invention is to provide a method for identifying and regulating the self-vibration characteristics of fruit trees based on vision; it is used to solve the technical problems that traditional vibration pickup methods are difficult to stick on various parts of fruit trees in the existing research on the vibration characteristics of fruit trees, and are easily disturbed by external factors, affecting the stability of data acquisition, and the technical problem that the existing detection equipment has a limited acquisition range and cannot comprehensively monitor the dynamic response of various parts of fruit trees.

[0005] To achieve the above object, the technical solution adopted by the present invention is: A method for identifying and regulating the self-vibration characteristics of fruit trees based on vision comprises the following steps: S1. Video acquisition: Use a high-definition camera device to obtain the micro-motion video of fruit trees under natural excitations (such as wind, mechanical perturbation, etc.), and transmit the real-time collected data to the data processing end; S2. Extraction of the natural vibration frequency information of fruits based on machine vision: The data processing end corrects the vibration of the camera device itself; performs time-domain and frequency-domain processing on the fruit tree video to extract the vibration frequency of the target fruit; by comparing the vibration frequency of the fruit with the data after the vibration correction of the camera, ensure that the obtained vibration characteristics are accurate and error-free, and output the natural vibration frequency of the fruit; S3. Extraction of fruit tree structure parameters based on machine vision: Select clear images from the video, use the Mask R-CNN algorithm for object detection and instance segmentation, and combine the Canny edge detection technology to enhance the image details, and accurately extract geometric parameters such as the crown area, the aspect ratios of branches and the main trunk of the tree; S4. Optimization of the optimal force application points of fruit trees based on spatial finite element analysis and physics-informed deep neural network: First, select jujube trees with different aspect ratios, establish a three-dimensional finite element model and build a physics-informed deep neural network based on the geometric information of the main trunk and branches of the jujube tree obtained by computer vision technology; by simulating the dynamic responses of branches under different excitation positions, analyze and statistically obtain the functional relationship between the optimal excitation position and the aspect ratio of the tree trunk; integrate the information of the optimal excitation position and the vibration frequency data in step S2 to achieve the adaptive regulation of the excitation parameters for each jujube tree.

[0006] Further, in the step S1, the camera device can select the HDR function or an infrared camera.

[0007] Further, in the step S2, the process of correcting the vibration of the camera device itself is: by analyzing the phase difference between consecutive video frames, extract the vibration frequency characteristics of the camera during movement, and use the ground static reference object as the spatial reference to calculate the pixel offset caused by the camera jitter; based on the phase difference analysis result, use the affine transformation algorithm to perform frame-by-frame dynamic image stabilization processing on the video frames to eliminate the image blur and displacement error caused by device jitter.

[0008] Further, in the step S2, the process of video time-domain and frequency-domain processing is as follows: First, each frame of the input structural micro-vibration video image is transformed from the spatial domain to the frequency domain through a 2D Fourier transform, separating the high-frequency components (fruit vibration) and low-frequency components (branch swing) to obtain the amplitude spectrum and phase spectrum of the image sequence; then, the complex steerable pyramid is used to perform frequency-domain decomposition on the image sequence within each frequency domain to obtain the local amplitude spectrum and phase spectrum of the image sequence at different scales and different directions; the 2D Gabor wavelet filter is applied to the image sequence at different scales and different directions to extract local motion and identify the texture information at different scales and directions; since the motion information of the video is contained in the phase of each pixel, the first frame image in the video can be used as a reference frame, and the phase information of the subsequent image sequence is subtracted from the first frame to obtain the phase difference; then, the BPMM algorithm is used for amplification processing within the wide frequency band containing all the required frequencies, and the phase difference is multiplied by the amplification factor to achieve the amplification processing of the local phase in the video; finally, the image sequence is reconstructed through an inverse Fourier transform, and each frame of the image in the video is transformed from the frequency domain to the spatial domain to obtain the video image after motion amplification.

[0009] Further, in the step S2, the process of extracting the natural vibration frequency of the fruit is as follows: The video is processed using a 2D Gabor wavelet filter; the 2D Gabor wavelet filter is sensitive to the edge information of the image. The 2D Gabor wavelet filter is used to extract the texture features of the local motion of the structure and smooth the noise in the image to improve the signal-to-noise ratio of the images in the video.

[0010] The beneficial effects of the present invention are as follows: The patent solution adopts a high-definition imaging device combined with computer vision technology to achieve full-field non-contact measurement. Through video acquisition, the system can capture the motion information of each pixel point of the fruit tree, providing more comprehensive vibration data than traditional methods. This method can accurately separate multiple independent modes of the fruit tree and accurately extract the natural vibration frequency of the fruit. This not only avoids the frequency deviation caused by insufficient measurement points in traditional methods but also greatly improves the efficiency of data acquisition and processing. Compared with expensive devices such as laser vibrometers, this study only requires an ordinary high-definition imaging device to complete the measurement, reducing costs and being applicable to large-scale field environments. In addition, combined with finite element analysis and optimization algorithms, this method can intelligently calculate the optimal force application points, making the applied vibration force highly match the fruit detachment characteristics, thereby improving the picking efficiency and reducing tree body damage. This innovative solution breaks through the limitations of traditional methods and provides an efficient, low-cost, and highly adaptable technical path for intelligent agriculture and precise fruit tree management. Description of the Drawings

[0011] Figure 1Schematic diagram of fruit and tree trunk information extraction based on deep learning; Figure 2 Schematic diagram of frequency analysis of different stress points of fruit trees based on deep learning and finite element analysis. Specific implementation manner

[0012] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings.

[0013] As Figure 1 and 2 shown, a method for identifying the natural vibration characteristics of fruit trees and exciting vibration regulation based on vision; the method mainly includes four steps: video acquisition, extraction of fruit natural vibration frequency information based on machine vision, extraction of fruit tree structure parameters based on machine vision, and optimization of the best stress points of fruit trees based on spatial finite element analysis and physical information deep neural network.

[0014] S1. Video acquisition. Use a high-definition camera device to obtain the micro-motion video of the fruit tree under natural excitation (such as wind, mechanical perturbation, etc.), and transmit the real-time collected data to the data processing end. Provide a data basis for subsequent vibration analysis and structure parameter extraction. Adopt a high-frame-rate and high-resolution camera device to ensure that the tiny vibrations of the fruit tree can be accurately captured. To adapt to different lighting environments, an HDR function or an infrared camera can be selected to reduce the influence brought by lighting changes.

[0015] In terms of equipment installation, the camera can be fixed on an adjustable pan-tilt head to facilitate shooting fruit trees at different heights and angles. At the same time, an image stabilization pan-tilt head or a tripod can be used to reduce the error caused by equipment shaking. To ensure the stability and consistency of the data, automatic focus and image stabilization algorithms are adopted during the video acquisition process to improve the image quality. The collected video data will be used as the basic input for subsequent machine vision analysis for the extraction of the vibration characteristics and geometric structure of the fruit tree.

[0016] S2. Extraction of fruit natural vibration frequency information based on machine vision. First, the vibration video of the fruit tree collected by the high-definition camera device is wirelessly transmitted to the data processing end and analyzed using computer vision technology. Then, correct the vibration of the camera device itself, analyze the vibration frequency of the camera during driving using the phase difference between video frames, and perform image stabilization processing based on the ground static reference object to eliminate the error caused by equipment shaking. Then, perform time-domain and frequency-domain processing on the fruit tree video, adopt phase amplification technology to enhance the tiny vibration signal of the fruit tree, and use Fourier transform or time-frequency analysis to extract the vibration frequency of the target fruit. By separating the motion modes of the branches and the fruit, accurately identify the natural vibration frequency of the fruit under natural excitation. Finally, by comparing the vibration frequency of the fruit with the data after the vibration correction of the camera, ensure that the obtained vibration characteristics are accurate and error-free, and output the natural vibration frequency of the fruit, providing key parameters for subsequent vibration picking optimization.

[0017] The specific analysis process of video analysis is described as follows: 1) By analyzing the phase difference between consecutive video frames, the vibration frequency characteristics of the camera during movement are extracted, and the pixel offset caused by camera jitter is calculated with a ground static reference object (such as a fixed calibration board) as the spatial reference. Based on the phase difference analysis results, the affine transformation algorithm is used to perform frame-by-frame dynamic image stabilization processing on the video frames, eliminating image blurring and displacement errors caused by device jitter, and ensuring the stability and data reliability of subsequent vibration signal analysis.

[0018] 2) Time-domain and frequency-domain enhancement processing of fruit tree vibration signals. For the video recorded during the process of the device approaching the fruit tree, after preprocessing, time-domain and frequency-domain enhancement processing of the fruit tree vibration signals is carried out. The specific process is described as follows: First, each frame of the input video image of small structural vibrations is transformed from the spatial domain to the frequency domain through 2D Fourier transform, separating the high-frequency components (fruit vibrations) and low-frequency components (branch swings) to obtain the amplitude spectrum and phase spectrum of the image sequence. Then, the complex steerable pyramid is used to perform frequency-domain decomposition on the image sequence in each frequency band, obtaining the local amplitude spectrum and phase spectrum of the image sequence at different scales and different directions. The 2D Gabor wavelet filter is applied to the image sequence at different scales and different directions to extract local motion and identify texture information at different scales and directions. Since the motion information of the video is contained in the phase of each pixel, the first frame image in the video can be used as a reference frame, and the phase information of the subsequent image sequence is subtracted from the first frame to obtain the phase difference. Then, within the wide frequency band containing all the required frequencies, amplification processing is carried out using the BPMM algorithm, and the phase difference is multiplied by the amplification factor to achieve amplification processing of the local phase in the video. Finally, the image sequence is reconstructed through inverse Fourier transform, and each frame of the image in the video is transformed from the frequency domain to the spatial domain to obtain the video image after motion amplification.

[0019] The broadband phase motion amplification principle based on the BPMM algorithm is described as follows: Let represent the position of the small structural vibration image and the image intensity at time t, that is . When the small vibration displacement and occur to the structure, the image intensity is as shown in Equation (1).

[0020] (1) When performing amplification processing on the small vibration video of the structure, it is necessary to transform the input video signal into a frequency-domain signal through Fourier transform, as shown in Equations (2) and (3).

[0021] When t = 0: (2) When t > 0: (3) Where: is the natural frequency of the structure corresponding frequency baseband; is the vibration amplitude of the structure; and are the phase information containing the structure vibration at t = 0 and t > 0 respectively. Subtracting the phase information at t = 0 and t > 0 eliminates and After that, the phase difference can be obtained, as shown in Equation (4).

[0022] (4) Multiplying the phase difference in Equation (4) by the amplification factor can achieve the amplification processing of the local phase in the video, and the result is shown in Equation (5).

[0023] (5) The amplification result within a certain frequency baseband is shown in Equation (6).

[0024] (6) Performing amplification processing on all frequency bands within the wide frequency band containing all the required frequencies, and reconstructing the amplified structural micro-vibration signal, the final amplification result is shown in Equation (7).

[0025] (7) 3) Precise extraction of the natural vibration frequency of the fruit. First, process the video using a 2D Gabor wavelet filter. The 2D Gabor wavelet filter is sensitive to the edge information of the image. Use the 2D Gabor wavelet filter to extract the texture features of the local motion of the structure and smooth the noise in the image to improve the signal-to-noise ratio of the image in the video. The 2D Gabor wavelet filter is a sine function modulated by a Gaussian function, and its complex expression is shown in Equation (8).

[0026] (8) The real part is shown in Equation (9).

[0027] (9) The imaginary part is shown in Equation (10).

[0028] (10) Among them, is the wavelength of the sine function; is the phase offset of the tuning function; determines the spatial aspect ratio of the 2D Gabor function; is the standard deviation of the Gaussian function, which determines the size of the acceptable region of the 2D Gabor filter kernel; is the direction of the 2D Gabor wavelet filter kernel, and ; and are the pixel coordinates of the image. In Equations (8) to (10), and contain the direction information and spatial information of the video image sequence, as shown in Equation (11).

[0029] (11) Use the real and imaginary parts of the 2D Gabor wavelet filter to extract the texture features of the structural local motion from different directions.

[0030] Then, use the sub-pixel template matching algorithm to extract the displacement time-history response of the structure. Assume there are two images with the same size and where has a relative translation with the reference image . After Fourier transform, and The cross-correlation relationship between them is shown in Equation (12).

[0031] (12) In the formula: and are the sizes of the images; is the amount of coordinate shift; "" represents the complex conjugate; and respectively represent and The discrete Fourier transform (Discrete Fourier Transform, DFT). The expression of (13) According to Equation (12), first, extract the pixel-level displacement of the structural vibration by locating the peak of . Then, at Perform cross-correlation based on time-effective matrix multiplication DFT in the area near the initial peak to extract the sub-pixel level displacement of structural vibration.

[0032] Finally, perform frequency identification. When using the BPMM algorithm to amplify the video of the minute vibration of the structure, the displacement amplitude of the minute vibration of the structure is amplified times. The displacement time history response obtained based on the sub-pixel template matching algorithm is not the actual displacement time history response of the structure.

[0033] The displacement of the minute vibration of the structure without BPMM processing is shown in Equation (14).

[0034] (14) Where: is the true displacement of the minute vibration of the structure; is the displacement amplitude of the minute vibration of the structure; is the displacement identification error caused by video illumination change and environmental noise.

[0035] The displacement time history response of the structure obtained from the video processed by the BPMM algorithm is shown in Equation (15).

[0036] (15) Where: is the amplified displacement of the minute vibration of the structure.

[0037] Normalize the motion to obtain the actual displacement of the structure as shown in Equation (16). It can be concluded from Equation (16) that using the BPMM algorithm for amplification processing can reduce the influence of noise in the video on the displacement identification of the minute vibration of the structure.

[0038] (16) Obtain the actual displacement time history response of the structure, and analyze the normalized displacement time history signal through FFT (Fast Fourier Transform) , the vibration frequency of the structure can be extracted. The specific steps are as follows: Discrete sampling: With the sampling frequency uniformly sample to obtain the discrete sequence .

[0039] Spectrum calculation: Apply FFT to convert the time-domain signal to the frequency-domain, as shown in Equation (17), (17) Where, represents the complex amplitude of the th frequency component, and the corresponding frequency is shown in Equation (18).

[0040] (18) Frequency identification: spectral amplitude The peak value corresponds to the main vibration frequency of the structure.

[0041] Key constraint: sampling frequency The Nyquist criterion needs to be satisfied , and spectral leakage can be reduced by windowing (such as Hanning window) to ensure frequency resolution to meet the accuracy requirements.

[0042] S3. Extraction of fruit tree structure parameters based on machine vision.

[0043] 1) Extraction of fruit tree information based on deep learning.

[0044] In video processing, we first use the Mask R-CNN deep learning model to perform object detection and segmentation of each frame of the video input in real time. This model can accurately identify the trunk, crown, and fruits, and generate a segmentation mask to make the boundaries of different parts clearer. To enhance the detection effect, we convert the RGB of the video frame to grayscale, reduce the computational complexity by reducing the color image to a grayscale image through a one-dimensional dimensionality reduction transformation of formula (19), and use Canny edge detection to extract the main contour information of the fruit tree (as Figure 1 shown). Then, morphological closing operation is used to process the boundary details to eliminate detection errors and broken areas. (19) When extracting the crown information, first obtain the segmentation mask of the crown area through Mask R-CNN, and combine connected component analysis to extract the complete crown shape. For the trunk part, Mask R-CNN is also used for detection, and the minimum bounding rectangle method is used to determine the overall structure of the trunk to ensure the stability of morphological features. Fruit detection is based on the bounding box of Mask R-CNN, and further screening is carried out by combining features such as color and morphology to remove misdetected parts. After completing this process, the segmentation result of the video frame obtained can be used for subsequent crown area calculation and trunk slenderness ratio analysis.

[0045] 2) Calculation of target area features.

[0046] After extracting the information of the crown, trunk, and fruits, we further calculate the key feature parameters of the fruit tree. The area of the crown is an important indicator to measure the growth status of the fruit tree, and the calculation method is to sum the pixels in the crown area of the binary segmentation image, and its mathematical expression is shown in formula (20).

[0047] (20) Among them, represents a binary mask image, where the pixel values of the canopy area are set to 1 and those of the background area are 0. To improve the calculation accuracy, multiple frames are averaged to reduce the influence of video noise. The slenderness ratio of the tree trunk reflects the morphological characteristics of the tree trunk, and the expression is shown in Equation (21).

[0048] (21) Among them, is the maximum height of the tree trunk, determined by the height of the minimum bounding rectangle, is the maximum width of the tree trunk, corresponding to the width of the bounding rectangle. The detection results of the tree trunk area are processed through connected component analysis to ensure the integrity of the tree trunk contour and conform to the actual morphology.

[0049] S4. Optimization of the best stress point of fruit trees based on spatial finite element analysis and physics-informed deep neural network.

[0050] 1) Finite element modeling and simulation. First, 1000 jujube trees with different slenderness ratios are selected, and a three-dimensional finite element model is established based on the geometric information of the main trunk and branches of the jujube trees obtained through computer vision technology. In the vibration analysis of fruit trees, the tree trunk can be regarded as an elastic beam, the canopy and branches as a distributed mass damping system, and the fruits as a local additional small mass pendulum system. The dynamic behavior of the tree body follows the standard finite element motion equation, as shown in Equation (22).

[0051] (22) Among them, is the mass matrix, is the damping matrix, is the stiffness matrix, is the displacement vector, is the externally applied excitation force. In the finite element software, the finite element model is established using the structural mechanics module, the tree body is discretized, and the vibration mode of the tree trunk is calculated using the characteristic frequency analysis. The eigenvalue problem of the system is solved through modal analysis, as shown in Equation (23).

[0052] (23) Among them, is the angular frequency, is the natural frequency. By applying different force application points , the vibration frequencies of the tree body at different positions are calculated , and the results are exported for subsequent optimization.

[0053] 2) Computer vision processing and data analysis. Through computer vision technology, geometric information of fruit trees (such as crown area, trunk aspect ratio) and fruit natural vibration frequency data are extracted from videos. First, the input video is converted from RGB to grayscale and Mask R-CNN object detection is performed to segment different parts of the fruit tree. The area of the crown is calculated by summing the pixels in the binarized mask region, as shown in Equation (24).

[0054] (24) Where, represents the pixels in the crown area, represents the background. The aspect ratio of the trunk is defined by calculating the height and width of the minimum bounding rectangle, as shown in Equation (25).

[0055] (25) The extraction of the natural vibration frequency of the fruit requires analyzing the movement trajectory of the fruit, and the Fourier transform (FFT) is used to extract the main frequency components, as shown in Equation (26).

[0056] (26) Where, represents the displacement of the fruit at time t, represents the Fourier transform, is the time series signal of the fruit movement trajectory after Fourier transform to obtain the spectrum, represents the amplitude of the spectrum, that is, the intensity of each frequency component, represents finding the frequency that makes the spectrum amplitude the largest among all frequencies (as shown in Figure 2 ). This frequency is used for subsequent optimization to make the trunk vibration frequency close to the natural vibration frequency of the fruit .

[0057] 3) Deep learning and physics-informed deep neural network. After obtaining the finite element simulation data of different force application points, the physics-informed deep neural network (PINNs) is used to learn the relationship between the force application points and the trunk vibration frequency . PINNs are implemented through a deep learning framework (such as PyTorch), and physical constraints are fused to ensure that the calculation results conform to the mechanical laws. The loss function is defined as shown in Equation (27).

[0058] (27) Among them, the constraint network output satisfies the finite element dynamics equation, apply boundary conditions, and make the predicted values of the neural network consistent with the simulation data. After training, PINNs can predict the vibration frequency of the tree body at any point of force application as the input for the optimization process.

[0059] 4) Optimization solution of the force application point. The optimization goal is to find the best force application point such that the vibration frequency of the tree trunk is closest to the natural vibration frequency of the fruit, as shown in Equation (28).

[0060] (28) This optimization problem can be solved using gradient descent or evolutionary algorithms, such as SciPy.optimize. By continuously adjusting the position of the force application point and performing iterative calculations until the that minimizes is found.

[0061] 5) Fitting the function relationship curve. Use PINNs to fit the relationship between the slenderness ratio of the jujube tree and the best excitation position, and ensure that this relationship conforms to the dynamics equation obtained from finite element analysis (FEA). First, construct a neural network with the slenderness ratio of the jujube tree as the input and the best excitation position as the output, and use a multi-layer perceptron (MLP) for non-linear mapping. The mathematical representation of the network is shown in Equation (29).

[0062] (29) Among them, is the neural network model, are the parameters to be trained.

[0063] For the existing experimental data (such as 1000 groups of jujube tree data), use the mean square error (MSE) as the loss function, as shown in Equation (30).

[0064] (30) Among them, is the number of experimental data points, is the actually observed best excitation position, is the predicted value of the neural network.

[0065] To ensure that the model conforms to the results of finite element mechanical analysis, we need to introduce physical constraints. In finite element analysis, the tree body vibration dynamics follows the motion equation as shown in Equation (31).

[0066] (31) The physical loss function is defined as shown in Equation (32).

[0067] (32) Where: is the number of sampling points of the physical equation, and the second-order derivative is calculated by PINNs to satisfy the motion equation, and this loss term ensures that the vibration frequency predicted by the neural network conforms to the physical law.

[0068] In the training stage, first use PyTorch to build a neural network model and initialize the parameters . Then, calculate the data loss and the physical loss . Then, adopt the Adam or LBFGS optimization algorithm to iteratively adjust the neural network parameters through gradient descent to minimize the total loss and make the loss function converge. Finally, after sufficient training, a mathematical relationship as shown in Equation (33) that can predict the optimal excitation position of jujube trees with different slenderness ratios is obtained.

[0069] (33).

[0070] This method can not only accurately reflect the experimental data, but also conform to the constraints of the dynamic equation, providing theoretical support and engineering application basis for the subsequent vibration picking optimization. Using the real-time collected data to continuously update the model parameters can ensure that the system always responds to the latest tree state and provide a more accurate regulation basis for intelligent picking.

Claims

1. A vision-based method for identifying the self-vibration characteristics of fruit trees and exciting vibration regulation, characterized in that, It includes the following steps: S1. Video acquisition: Use a high-definition camera device to obtain the micro-motion video of fruit trees under natural excitation, and transmit the real-time collected data to the data processing end; S2. Extraction of natural vibration frequency information of fruits based on machine vision: The data processing end corrects the vibration of the camera device itself; performs time-domain and frequency-domain processing on the fruit tree video to extract the vibration frequency of the target fruit; by comparing the fruit vibration frequency with the data after the camera vibration correction, ensure that the obtained vibration characteristics are accurate and error-free, and output the natural vibration frequency of the fruit; S3. Extraction of fruit tree structure parameters based on machine vision: Select clear images from the video, use the Mask R-CNN algorithm for object detection and instance segmentation, and combine the Canny edge detection technology to enhance the image details, and accurately extract geometric parameters such as the crown area, the aspect ratio of branches and the main trunk of the tree; S4. Optimization of the best stress point of fruit trees based on spatial finite element analysis and physics-informed deep neural network: First, select jujube trees with different aspect ratios, establish a three-dimensional finite element model and build a physics-informed deep neural network based on the geometric information of the main trunk and branches of the jujube tree obtained by computer vision technology; by simulating the dynamic response of the branches under different excitation positions, analyze and statistically obtain the functional relationship between the best excitation position and the aspect ratio of the tree trunk; integrate the best excitation position information and the vibration frequency data in step S2 to realize the adaptive regulation of the excitation parameters of each jujube tree.

2. The regulation method according to claim 1, wherein In the step S1, the camera device can select the HDR function or an infrared camera.

3. The regulation method according to claim 1, characterized in that, In the step S2, the process of correcting the vibration of the camera device itself is: by analyzing the phase difference between consecutive video frames, extract the vibration frequency characteristics of the camera during movement, and use the ground static reference object as the spatial reference to calculate the pixel offset caused by the camera jitter; Based on the phase difference analysis result, use the affine transformation algorithm to perform frame-by-frame dynamic image stabilization processing on the video frames to eliminate the image blur and displacement error caused by device jitter.

4. The regulation method according to claim 1, wherein In the step S2, the processes of video time-domain and frequency-domain processing are as follows: First, each frame of the input video image of the structural micro-vibration is transformed from the spatial domain into the frequency domain through a 2D Fourier transform, the high-frequency components and the low-frequency components are separated, and the amplitude spectrum and the phase spectrum of the image sequence are obtained; then, the complex steerable pyramid is used to perform frequency-domain decomposition on the image sequence in each frequency band, and the local amplitude spectrum and phase spectrum of the image sequence with different scales and different directions are obtained; the 2D Gabor wavelet filter is applied to the image sequence with different scales and different directions to extract local motion and identify the texture information in different scales and directions; since the motion information of the video is contained in the phase of each pixel, the first frame image in the video can be used as a reference frame, and the phase information of the subsequent image sequence is subtracted from the first frame to obtain a phase difference; then, amplification processing is performed using the BPMM algorithm within a wide frequency band containing all the required frequencies, and the phase difference is multiplied by an amplification factor to achieve the amplification processing of the local phase in the video; finally, the image sequence is reconstructed through an inverse Fourier transform, and each frame of the image in the video is transformed from the frequency domain into the spatial domain to obtain the video image after motion amplification.

5. The regulation method according to claim 1, characterized in that In the step S2, the process of extracting the natural vibration frequency of the fruit is as follows: The video is processed using a 2D Gabor wavelet filter; the 2D Gabor wavelet filter is sensitive to the edge information of the image, and the 2D Gabor wavelet filter is used to extract the texture features of the local motion of the structure and smooth the noise in the image to improve the signal-to-noise ratio of the image in the video.

Images