Vision-based peach vibration measurement point positioning and alignment control method

Abstract

The invention discloses a peach vibration measurement point positioning and alignment control method based on vision. The method comprises the following steps: carrying out tree peach image enhancement processing by using an automatic multi-scale Retinex algorithm; training an on-tree peach fruit recognition model based on YOLOv8, and obtaining pixel coordinates of a center point of an on-tree peach fruit recognition frame; acquiring depth information of a target object based on binocular stereoscopic vision, and calculating to obtain three-dimensional coordinates of a central point of a peach recognition frame; a laser Doppler vibrometer, a binocular camera and a holder are utilized to construct an acoustic vibration response automatic acquisition device for peaches on a tree, and combined calibration of the laser Doppler vibrometer and the binocular camera is completed based on a PnP algorithm. The mapping relation between the peach vibration measurement points and the rotation angle of the laser Doppler vibration measurement instrument under a camera coordinate system is solved based on a PnP algorithm, the laser Doppler vibration measurement instrument is controlled to rotate through a holder, one-by-one automatic alignment of the peach vibration measurement points on the tree is achieved, and the method is used for assisting field automatic collection of sound and vibration response data of the peaches on the tree.

Description

Technical Field

[0001] This invention relates to the field of fruit quality inspection, and in particular to a vision-based method for locating and aligning vibration points on peaches. Background Technology

[0002] The ripeness of peaches at harvest affects both the harvesting and post-harvest processing. To improve the storage and transportation resistance of peaches, they are usually harvested earlier. However, the ripeness of the fruit at harvest affects the quality of the peaches. Chinese consumers generally prefer peaches with a softer texture, and peaches harvested too early lack flavor, which reduces their market acceptance. Therefore, it is necessary to test the quality of peaches before harvesting to determine the optimal harvesting time.

[0003] Acoustic vibration testing assesses the quality of agricultural products by analyzing their vibration response to specific stimuli. It is commonly used to detect hardness, internal defects, sugar content, and ripeness, and is a relatively mature non-destructive testing technique for agricultural products. However, in most current applications, the laser Doppler vibrometer needs to be manually aligned with the target point. In a laboratory environment, the operator can place the object under test on a fixed platform for assembly-line testing. But in the pre-harvest testing of peaches, due to the varying locations and large numbers of peaches, manual alignment is extremely labor-intensive and time-consuming, making large-scale testing before harvest difficult. Summary of the Invention

[0004] To address the aforementioned problems, this invention proposes a vision-based method for locating and aligning vibration points in peaches, which solves the problem of low efficiency in detecting peach quality using acoustic vibration methods and provides an effective solution for achieving pre-harvest quality control of peaches.

[0005] This invention integrates a gimbal, a binocular camera, and a laser Doppler vibrometer, automating the measurement of the vibration response of peaches on a tree. It utilizes the YOLOv8 detection model and binocular vision technology to identify peaches and locate vibration measurement points. The Perspective-n-Point (PnP) algorithm is used for joint calibration of the camera and laser Doppler vibrometer, and the gimbal rotation is controlled to automatically align the laser Doppler vibrometer with the measurement point.

[0006] To solve the above-mentioned technical problems, the technical solution proposed by this invention is as follows: A vision-based method for locating and aligning vibration measurement points on peaches, characterized by employing an automatic acquisition device for the acoustic vibration response of peaches on trees, comprising: a laser Doppler vibrometer, a binocular camera, and a gimbal, wherein the laser Doppler vibrometer is mounted on the gimbal. The method includes the following steps: Step 1: Acquire images of peaches on trees using a binocular camera, filter the acquired images to obtain qualified peach images, label the peach images to distinguish them from the background, and then perform image enhancement processing to create an enhanced dataset. Step 2: Train a YOLOv8-based peach recognition model on a tree using the image-enhanced dataset to obtain the pixel coordinates of the center point of the peach recognition box on the tree. Step 3: Use binocular stereo vision technology to capture images of peaches on the tree using binocular cameras and calculate image parallax. Obtain image depth information through image parallax calculation. Obtain depth information of peach vibration measurement points based on the pixel coordinates of the center point of the peach recognition box on the tree and the image depth information. Calculate the spatial position of the peach vibration measurement points in the camera coordinate system using the depth information of the peach vibration measurement points. Step 4: Use the PnP algorithm to jointly calibrate the laser Doppler vibrometer and the binocular camera to obtain the transformation matrix from the binocular camera coordinate system to the laser Doppler vibrometer coordinate system; Step 5: Using the transformation matrix obtained in Step 4, the spatial positions of the peach vibration measurement points in the camera coordinate system obtained in Step 3 are converted into the spatial positions of the peach vibration measurement points in the laser Doppler vibrometer coordinate system. The PnP algorithm is used to solve the mapping relationship between the spatial positions of the peach vibration measurement points in the laser Doppler vibrometer coordinate system and the rotation angle of the laser Doppler vibrometer to be solved. The laser Doppler vibrometer is rotated by controlling the pan-tilt unit to realize the automatic alignment of the peach vibration measurement points on the tree one by one, which is used to assist in the automatic on-site acquisition of the acoustic vibration response data of the peaches on the tree.

[0007] In step 1, the images of peaches on the tree include: color images of peaches on the tree under different light conditions 103-111 days after the peak flowering period and at the ripening stage.

[0008] In step 1, the peach images on the tree are labeled using the data labeling software LabelImg.

[0009] In step 1, the image-enhanced dataset includes a training set, a validation set, and a test set of peach images on trees.

[0010] In step 1, the automated multi-scale Retinex algorithm is used to enhance the images of peaches on the tree.

[0011] In step 4, the binocular stereo vision technology includes: Step 4.1, Binocular camera calibration, using the checkerboard calibration method to determine the camera's internal and external parameters and distortion coefficients; Step 4.2: Based on the external parameters obtained during the dual-target calibration process, the Bouguet method is used for stereo correction to make the image planes coplanar and the epipolar lines aligned. Step 4.3: Use a semi-global stereo matching algorithm to calculate the disparity values ​​of the left and right views, and calculate the depth information of the image.

[0012] In step 4, the PnP algorithm uses formula (1) for calculation: in, This represents the distance between the center of the laser Doppler vibration meter and the vibration measurement point on the peach. The vertical installation distance between the center of the laser Doppler vibration meter and the center of the gimbal rotation; The horizontal rotation angle of the laser Doppler vibration meter; This represents the vertical rotation angle of the laser Doppler vibration meter. , This is the rotation and translation matrix for transforming the camera coordinate system to the laser Doppler vibration meter coordinate system.

[0013] A further preferred method is a vision-based method for locating and aligning vibration measurement points on peaches, comprising the following steps: Step 1: Acquire images of peaches on the trees using an image acquisition device, including color images of peaches on the trees under different light conditions 103-111 days after full bloom and at the ripening stage. Filter the acquired images to obtain qualified peach image data. Use the data annotation software LabelImg to annotate the peach images on the trees, distinguishing the peaches from the background. Divide the data into training, validation, and test sets in a 7:2:1 ratio. Step 2: Use the automated multi-scale Retinex algorithm to enhance the images of peaches on the tree to solve problems such as uneven illumination and underexposure in the images acquired on site. Specifically, this includes: using an adaptive Gaussian kernel selection strategy to estimate the incident light components and eliminating image noise during the enhancement process through a noise suppression mechanism. Step 3: Train a YOLOv8-based peach recognition model on a tree using the image-enhanced dataset to obtain the pixel coordinates of the center point of the peach recognition box on the tree. Step 4: Obtain the depth information of the target object based on binocular stereo vision, and calculate the three-dimensional coordinates of the center point of the peach recognition box; Step 4.1, Binocular camera calibration: The internal and external parameters and distortion coefficients of the camera are determined by the checkerboard calibration method, so as to perform image correction and matching; Step 4.2: Based on the extrinsic parameter matrix obtained during the dual-target calibration process, the Bouguet method is used for stereo correction to make the image planes coplanar and the epipolar lines aligned. Step 4.3: The semi-global stereo matching algorithm is used to calculate the disparity values ​​of the left and right views, and the image depth information is calculated. The spatial position in the camera coordinate system is then calculated using the image depth information. Step 5: Construct an automatic acquisition device for the acoustic vibration response of peaches on a tree using a laser Doppler vibrometer, a binocular camera, and a gimbal. Perform joint calibration of the laser Doppler vibrometer and binocular camera based on the PnP algorithm to obtain the transformation matrix from the binocular camera coordinate system to the laser Doppler vibrometer coordinate system. The PnP calculation model can be described by formula (1): in, This represents the distance between the center of the laser Doppler vibration meter and the vibration measurement point on the peach. The vertical installation distance between the center of the laser Doppler vibration meter and the center of the gimbal rotation; The horizontal rotation angle of the laser Doppler vibration meter; This represents the vertical rotation angle of the laser Doppler vibration meter. , This is the rotation and translation matrix for transforming the camera coordinate system to the laser Doppler vibration meter coordinate system.

[0014] Step 6: Propose an automatic laser alignment algorithm to solve the mapping relationship between the spatial position of the vibration measurement point of the peach fruit in the coordinate system of the laser Doppler vibrometer and the rotation angle of the laser Doppler vibrometer. By controlling the rotation of the laser Doppler vibrometer through a gimbal, the vibration measurement points of the peach fruit on the tree are automatically aligned one by one, which is used to assist in the automatic on-site acquisition of acoustic vibration response data of the peach fruit on the tree.

[0015] As a further improvement of the present invention, a bandpass filter is used for filtering during signal preprocessing, and the power spectral density of the vibration response is calculated as the model input.

[0016] Compared with the prior art, the advantages of the present invention are as follows: The biomimetic peach vibration finite element model of this invention fully considers the complex shape, internal structure, and material properties of peaches. An accurate three-dimensional geometric model of the peach is constructed through 3D scanning and reverse modeling. The elastic moduli of the peach peel, pulp, pit, and kernel are experimentally determined as material properties for the finite element method, improving the accuracy of the finite element simulation. This invention uses the finite element simulation method to obtain simulation datasets for transfer learning, enabling the inversion of the hardness of peaches on the tree based on the acoustic vibration method even with a small sample size. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a specific implementation of the present invention.

[0018] Figure 2 An automatic acquisition device for the acoustic vibration response of peaches on a tree.

[0019] Figure 3 This is a schematic diagram of a laser alignment test conducted in an indoor environment.

[0020] Figure 4 This is a schematic diagram of a laser alignment experiment conducted in an orchard setting. Detailed Implementation

[0021] The automatic acquisition device for acoustic vibration response of peaches on trees according to the present invention will be further described below with reference to the accompanying drawings.

[0022] like Figure 1 As shown, the automatic acquisition device for the acoustic vibration response of peaches on a tree includes: a laser Doppler vibration meter (LDV) 1, a pan-tilt unit 2, a computer 3, a mobile chassis 4, a binocular camera 5, a data acquisition card 6, and an LDV controller 7. The laser Doppler vibration meter 1 is mounted on the pan-tilt unit 2.

[0023] like Figure 2 As shown, a vision-based method for locating and aligning vibration measurement points on peaches includes the following steps: Step 1: Acquire images of peaches on the trees using an image acquisition device, including color images of peaches on the trees under different light conditions 103-111 days after full bloom and at the ripening stage. Filter the acquired images to obtain qualified peach image data. Use the data annotation software LabelImg to annotate the peach images on the trees, distinguishing the peaches from the background. Divide the data into training, validation, and test sets in a 7:2:1 ratio. Step 2: Use the automated multi-scale Retinex algorithm to enhance the images of peaches on the tree to solve problems such as uneven illumination and underexposure in the images acquired on site. Specifically, this includes: using an adaptive Gaussian kernel selection strategy to estimate the incident light components and eliminating image noise during the enhancement process through a noise suppression mechanism. Step 3: Train a YOLOv8-based peach recognition model on a tree using the image-enhanced dataset to obtain the pixel coordinates of the center point of the peach recognition box on the tree. Step 4: Obtain the depth information of the target object based on binocular stereo vision, and calculate the three-dimensional coordinates of the center point of the peach recognition box; Step 4.1, Binocular camera calibration: The internal and external parameters and distortion coefficients of the camera are determined by the checkerboard calibration method, so as to perform image correction and matching; Step 4.2: Based on the extrinsic parameter matrix obtained during the dual-target calibration process, the Bouguet method is used for stereo correction to make the image planes coplanar and the epipolar lines aligned. Step 4.3: Calculate the disparity values ​​of the left and right views using a semi-global stereo matching algorithm, and obtain the depth map; Step 5: Construct an automatic acquisition device for the acoustic vibration response of peaches on a tree using a laser Doppler vibrometer, a binocular camera, and a gimbal. Perform joint calibration of the laser Doppler vibrometer and binocular camera based on the PnP algorithm to obtain the transformation matrix from the binocular camera coordinate system to the laser Doppler vibrometer coordinate system. The PnP calculation model can be described by formula (1): in, This represents the distance between the center of the laser Doppler vibration meter and the vibration measurement point on the peach. The vertical installation distance between the center of the laser Doppler vibration meter and the center of the gimbal rotation; The horizontal rotation angle of the laser Doppler vibration meter; This represents the vertical rotation angle of the laser Doppler vibration meter. , This is the rotation and translation matrix for transforming the camera coordinate system to the laser Doppler vibration meter coordinate system.

[0024] Step 6: Propose an automatic laser alignment algorithm to solve the mapping relationship between the spatial position of the vibration measurement point of the peach fruit in the coordinate system of the laser Doppler vibrometer and the rotation angle of the laser Doppler vibrometer. By controlling the rotation of the laser Doppler vibrometer through a gimbal, the vibration measurement points of the peach fruit on the tree are automatically aligned one by one, which is used to assist in the automatic on-site acquisition of acoustic vibration response data of the peach fruit on the tree.

[0025] like Figure 3 As shown, to verify the effectiveness of the present invention and to evaluate the effectiveness of the vision-based peach vibration point localization method and alignment control, experiments were conducted. First, to verify the effect of the LDV-binocular camera joint calibration and the accuracy of laser alignment, a checkerboard pattern was placed at horizontal distances of 0.5, 1.0, 1.5, 2.0, 2.5, and 3.0 m, respectively. The positioning error of laser alignment at different distances was calculated, and the results are shown in Table 1. The value represents the positioning error. As can be seen from the table, within the range of 0.5 to 3.0 m, the alignment error is controlled within 10 mm, with an average error of 5.25 mm, indicating that this method can achieve high alignment accuracy indoors.

[0026] Table 1 like Figure 4 As shown, to evaluate the measurement performance of the device in an agricultural field environment, a field measurement experiment was conducted in an orchard at the Yangdu Experimental Base of the Zhejiang Academy of Agricultural Sciences in Jiaxing City, Zhejiang Province. The experimental results after eliminating incorrect identification cases are shown in Table 2. The gimbal rotation angle when manually aligning with the center point of the peach identification frame is taken as the standard value. , These represent the average horizontal and vertical rotation angle errors obtained by the automatic alignment method of the peach vibration measurement point, respectively; the standard value is the spatial coordinates in the camera coordinate system when manually aligning the center point of the peach recognition frame. , These represent the average positioning errors in the horizontal and vertical directions, respectively. The value represents the total positioning error. The results show that the alignment error of the peaches on the tree is 9.74 mm, and the diameter of the peaches used in the experiment is about 6~9 cm. Therefore, the device can complete the automatic acquisition of the acoustic vibration response of the peaches well.

[0027] Table 2

Claims

1. A vision-based method for locating and aligning vibration measurement points on peaches, characterized in that, An automatic acquisition device for the acoustic and vibration response of peaches on trees is adopted, including: a laser Doppler vibrometer, a binocular camera and a pan-tilt unit, wherein the laser Doppler vibrometer is mounted on the pan-tilt unit; The method includes the following steps: Step 1: Acquire images of peaches on trees using a binocular camera, filter the acquired images to obtain qualified peach images, label the peach images to distinguish them from the background, and then perform image enhancement processing to create an enhanced dataset. Step 2: Train a YOLOv8-based peach recognition model on a tree using the image-enhanced dataset to obtain the pixel coordinates of the center point of the peach recognition box on the tree. Step 3: Use binocular stereo vision technology to capture images of peaches on the tree using binocular cameras and calculate image parallax. Obtain image depth information through image parallax calculation. Obtain depth information of peach vibration measurement points based on the pixel coordinates of the center point of the peach recognition box on the tree and the image depth information. Calculate the spatial position of the peach vibration measurement points in the camera coordinate system using the depth information of the peach vibration measurement points. Step 4: Use the PnP algorithm to jointly calibrate the laser Doppler vibrometer and the binocular camera to obtain the transformation matrix from the binocular camera coordinate system to the laser Doppler vibrometer coordinate system; Step 5: Using the transformation matrix obtained in Step 4, the spatial positions of the peach vibration measurement points in the camera coordinate system obtained in Step 3 are converted into the spatial positions of the peach vibration measurement points in the laser Doppler vibrometer coordinate system. The PnP algorithm is used to solve the mapping relationship between the spatial positions of the peach vibration measurement points in the laser Doppler vibrometer coordinate system and the rotation angle of the laser Doppler vibrometer to be solved. The laser Doppler vibrometer is rotated by controlling the pan-tilt unit to realize the automatic alignment of the peach vibration measurement points on the tree one by one, which is used to assist in the automatic on-site acquisition of the acoustic vibration response data of the peaches on the tree.

2. The vision-based method for locating and aligning vibration points on peaches according to claim 1, characterized in that, In step 1, the images of peaches on the tree include: color images of peaches on the tree under different light conditions 103-111 days after the peak flowering period and at the ripening stage.

3. The vision-based method for locating and aligning vibration measurement points in peaches according to claim 1, characterized in that, In step 1, the peach images on the tree are labeled using the data labeling software LabelImg.

4. The vision-based method for locating and aligning vibration points on peaches according to claim 1, characterized in that, In step 1, the image-enhanced dataset includes a training set, a validation set, and a test set of peach images on trees.

5. The vision-based method for locating and aligning vibration measurement points in peaches according to claim 1, characterized in that, In step 1, the automated multi-scale Retinex algorithm is used to enhance the images of peaches on the tree.

6. The vision-based method for locating and aligning vibration points on peaches according to claim 1, characterized in that, In step 4, the binocular stereo vision technology includes: Step 4.1, Binocular camera calibration, using the checkerboard calibration method to determine the camera's internal and external parameters and distortion coefficients; Step 4.2: Based on the external parameters obtained during the dual-target calibration process, the Bouguet method is used for stereo correction to make the image planes coplanar and the epipolar lines aligned. Step 4.3: Use a semi-global stereo matching algorithm to calculate the disparity values ​​of the left and right views, and calculate the depth information of the image.

7. The vision-based method for locating and aligning vibration measurement points of peaches according to claim 1, characterized in that, In step 4, the PnP algorithm uses formula (1) for calculation: in, This represents the distance between the center of the laser Doppler vibration meter and the vibration measurement point on the peach. The vertical installation distance between the center of the laser Doppler vibration meter and the center of the gimbal rotation; The horizontal rotation angle of the laser Doppler vibration meter; This represents the vertical rotation angle of the laser Doppler vibration meter. , This is the rotation and translation matrix for transforming the camera coordinate system to the laser Doppler vibration meter coordinate system.

Images