A Method and System for Detecting Phenotypic Parameters of Litchi Based on Image Processing and 3D Point Clouds

Abstract

This invention discloses a method and system for detecting phenotypic parameters of litchi based on image processing and 3D point clouds. The method includes: acquiring two-dimensional images of the litchi from multiple perspectives; mapping and processing the two-dimensional images of the litchi from multiple perspectives to obtain 3D point cloud data of the litchi; calculating the point cloud curvature and normal vector of the 3D point cloud data of the litchi; statistically determining the curvature threshold; identifying mechanical damage areas of the litchi; performing color model space transformation on the obtained RGB images of the litchi; extracting red pixels from the litchi image under the HSI model through a red pixel threshold to obtain the red coloring rate of the litchi; obtaining a depth image including the thickness of the litchi based on the depth image and the background depth image; constructing 3D models of the top and bottom of the litchi using the depth image; and calculating the volume of the litchi. This invention can quickly and accurately detect the phenotypic parameters of litchi, providing a scientific basis for litchi quality grading.

Description

Technical Field

[0001] This invention belongs to the field of fruit phenotypic parameter detection technology, specifically relating to a method and system for detecting litchi phenotypic parameters based on image processing and three-dimensional point clouds. Background Technology

[0002] Post-harvest grading of lychees is a crucial step before their preservation, packaging, sales, and further processing, and it is also key to increasing the added value of lychees. Currently, my country's post-harvest grading process for lychees is still relatively underdeveloped, mainly relying on manual sorting. However, manual grading of lychees suffers from high labor costs, low grading accuracy, and low efficiency. Furthermore, due to the thin skin and high water content of lychees, they are easily damaged during manual grading, which greatly affects their market value.

[0003] With the continuous development of machine vision technology in recent years, it has gradually been applied to the quality inspection of fruit products. Researchers have been collecting three-dimensional point cloud data of lychees and extracting their phenotypic parameters from this data to achieve non-destructive post-harvest grading. However, due to the small size of lychees, the accuracy of the collected three-dimensional point cloud data is relatively low, which greatly affects the accuracy of the phenotypic parameter detection results. In addition, the current phenotypic parameter detection of lychees still relies solely on the fruit diameter as the grading standard, resulting in a single grading criterion and causing uneven ripeness among lychees of the same grade. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a method for detecting phenotypic parameters of litchi based on image processing and three-dimensional point clouds. This method uses the weight, mechanical damage, and coloring rate of litchi as the grading standards for litchi quality. It acquires RGB and depth images of the litchi to be tested using a depth camera-based acquisition platform. Then, through point cloud curvature algorithm, RGB color space model conversion processing, and calculation of the thickness value of the depth image, the phenotypic parameters such as mechanical damage, coloring rate, and volume of the litchi to be tested are accurately extracted, providing a scientific basis for litchi quality grading.

[0005] Another objective of this invention is to provide a litchi phenotypic parameter detection system based on image processing and three-dimensional point clouds.

[0006] To achieve the above objectives, the present invention can be accomplished by adopting the following technical solution:

[0007] A method for detecting litchi phenotypic parameters based on image processing and 3D point clouds includes the following steps:

[0008] S1. Obtain two-dimensional images of the lychee to be tested from multiple perspectives, wherein the two-dimensional images include depth images and RGB images;

[0009] S2. Map the two-dimensional images of the litchi under multiple perspectives to obtain the original three-dimensional point cloud data of the litchi under test. Perform background segmentation and noise reduction filtering on the original three-dimensional point cloud data to obtain the three-dimensional point cloud data of the litchi under test. Calculate the point cloud curvature and normal vector of the three-dimensional point cloud data of the litchi under test, and statistically determine the curvature threshold used to judge the surface loss area of ​​the litchi. In this way, the point cloud of the litchi damage area is segmented, and the mechanical damage area of ​​the litchi under test is identified.

[0010] S3. Convert the RGB images of the lychee under multiple different viewpoints obtained in step S1 to the HSI color model space to obtain multiple lychee images under the HSI model from different viewpoints. Extract the red pixels in the multiple lychee images under the HSI model from different viewpoints by using a preset red pixel threshold and take the average value. Use the average value as the red coloring rate of the lychee under test.

[0011] S4. Based on the depth images f(x,y) of the lychee under multiple different viewpoints and the background depth image b(x,y), a depth image h(x,y) containing the thickness of the lychee is obtained. The depth image h(x,y) containing the thickness of the lychee is preprocessed. A 3D model of the top and bottom of the lychee under test is established using the preprocessed depth image h(x,y). The volume value of the lychee under test is calculated based on the 3D model of the top and bottom of the lychee under test.

[0012] Preferably, the multiple perspectives mentioned in step S1 include four perspectives: left view, right view, top view, and bottom view.

[0013] Preferably, the derivation process of the point cloud curvature and normal vector of the three-dimensional point cloud data of the litchi to be tested in step S2 is as follows:

[0014] Since each point in a point cloud has a neighborhood point cloud that is approximated using a specific surface, the curvature of a point can be represented by the curvature of the local surface fitted to that point and its neighbors. Therefore, taking pi as the center point and uniformly selecting k points near pi, the equation of the quadratic surface can be expressed as:

[0015] z(x,y)=ax 2 +bxy+cy 2

[0016] According to the least squares principle, z i The sum of squares can be expressed as:

[0017]

[0018] From this, the values ​​of the coefficients a, b, and c of the quadratic surface equation can be obtained:

[0019]

[0020] Then the curve on the surface can be represented as:

[0021] r = (x(t), y(t))

[0022] The unit normal vector at point Pi can be expressed as:

[0023]

[0024] If we take the normal vector at point Pi as the normal vector of the local surface in the neighborhood, then the covariance matrix of each point in the neighborhood is:

[0025]

[0026] C·X j =λ j ·X j j = 1, 2, 3

[0027] Where P0 is the centroid of the neighborhood points, k is the number of neighborhood points, and λ j and X j Let represent the eigenvalues ​​and eigenvectors of C, respectively.

[0028] The eigenvector corresponding to the smallest eigenvalue of matrix C is point P. i Given the normal vector, construct the covariance matrix. Calculate the eigenvalues ​​of the matrix through eigenvalue decomposition. When the eigenvalues ​​satisfy the condition λ0≤λ1≤λ2, the curvature of the neighborhood points can be expressed as:

[0029]

[0030] Ultimately, the neighborhood curvature k i This refers to the curvature of the neighborhood surface.

[0031] Preferably, the curvature threshold used to determine the lost area on the surface of the litchi in step S2 is 1.81.

[0032] Preferably, the specific process of step S3 is as follows:

[0033] First, the R, G, and B components of the RGB images obtained from multiple different viewpoints in step S3 are normalized. The color of a pixel located at spatial position (x,y) in the RGB color space is represented by three values: R(x,y), G(x,y), and B(x,y). Then, the litchi region in the normalized RGB image is converted to the HSI color space. The H(x,y), S(x,y), and I(x,y) components of a pixel in the HSI color space are represented as follows:

[0034]

[0035]

[0036]

[0037] in,

[0038] Finally, based on the above formula, the lychee images under the HSI model corresponding to the RGB images of the lychee under multiple different viewpoints were obtained;

[0039] Then, according to the preset red pixel threshold, the red pixel values ​​of each of the multiple lychee images under the HSI model from different perspectives are extracted and the average value is calculated. The average value is used as the red coloring rate of the lychee to be tested. The red coloring rate is the ratio of the red pixel value of the entire surface of the lychee to the total area of ​​all pixels of the lychee to be tested. The red pixel value is the red H component of the lychee surface in the lychee image under the HSI model.

[0040] Preferably, the preset red pixel threshold value ranges from 35° to 40°.

[0041] Preferably, in step S4, the preprocessing method for the depth image h(x,y) containing the thickness of the lychee is erosion, dilation, and Gaussian smoothing.

[0042] Preferably, the depth image f(x,y) and the background depth image b(x,y) in step S4 are captured in the same environment and at the same height.

[0043] A litchi phenotypic parameter detection system based on image processing and 3D point cloud is used in the litchi phenotypic parameter detection method based on image processing and 3D point cloud. The system includes a litchi image acquisition platform and a cloud server.

[0044] The lychee image acquisition platform includes a dark box, an LED light source, a lychee placement plate, a depth camera, and a camera bracket. The LED light source is installed at the top of the dark box, the depth camera is installed below the LED light source via the camera bracket, and the lychee placement plate is installed at the bottom of the dark box, with the lens of the depth camera facing the lychee placement plate. The lychee placement plate is used to place the lychees to be tested, and the depth camera is communicatively connected to the cloud server.

[0045] The cloud server includes a processor and a memory; the memory stores non-transitory computer instructions, which, when executed by the processor, perform the above-described method for detecting litchi phenotypic parameters based on image processing and 3D point clouds.

[0046] Preferably, the depth camera is an Intel Realsense D405 depth camera.

[0047] The present invention has the following advantages over the prior art:

[0048] (1) The litchi phenotypic parameter detection method based on image processing and three-dimensional point cloud of the present invention obtains the point cloud data of the litchi under test by acquiring RGB images and depth images of the litchi under test from four perspectives, and obtains the curvature threshold of the litchi surface loss area based on the point cloud normal vector and curvature algorithm, thereby quickly and accurately identifying the mechanical damage of litchi. Moreover, it effectively protects the litchi during the detection process and does not damage the litchi skin, effectively improving the economic value of litchi. Furthermore, according to multiple experimental statistics, the average recognition accuracy of the litchi mechanical damage area reaches 94%, thereby improving the accuracy of litchi phenotypic parameter detection.

[0049] (2) The litchi phenotypic parameter detection method based on image processing and three-dimensional point cloud of the present invention performs RGB-HSI color model space conversion on RGB images of litchi under multiple different perspectives to obtain litchi images under the HSI model, extracts red pixels in the litchi images under the HSI model by using a preset red pixel threshold, and finally calculates the red coloring rate of litchi. Compared with the conventional method of reconstructing litchi three-dimensional point cloud based on MVS-SFM algorithm to extract litchi point cloud coloring rate in the prior art, the present invention improves the speed by 90%, thereby greatly enhancing the efficiency of litchi phenotypic parameter detection.

[0050] (3) The litchi phenotypic parameter detection system based on image processing and three-dimensional point cloud of the present invention obtains the litchi to be tested and acquires litchi depth images and RGB images from four perspectives through the clever cooperation of the dark box, LED light source, litchi placement plate, depth camera and camera bracket. The acquired litchi depth images and RGB images from four perspectives are mapped to obtain the three-dimensional point cloud data of the litchi to be tested, thereby making the three-dimensional point cloud data of the litchi to be tested more accurate and improving the accuracy of litchi phenotypic parameter detection. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the litchi phenotypic parameter detection method based on image processing and three-dimensional point cloud provided in Embodiment 1 of the present invention;

[0052] Figure 2 These are RGB images, depth images, and point cloud maps of lychee from multiple perspectives provided in Embodiment 1 of the present invention;

[0053] Figure 3 This is a schematic diagram comparing the original three-dimensional point cloud and the processed point cloud of lychee provided in Embodiment 1 of the present invention;

[0054] Figure 4 This is a schematic diagram comparing the RGB image and HSI model image of a lychee provided in Embodiment 1 of the present invention;

[0055] Figure 5 This is a visualization result of the extraction of red pixels from lychee provided in Embodiment 1 of the present invention;

[0056] Figure 6 This is a schematic diagram of the depth images of the top and bottom of a lychee provided in Embodiment 1 of the present invention;

[0057] Figure 7 This is a schematic diagram comparing the lychee depth image h(x,y) before and after processing, provided in Embodiment 1 of the present invention.

[0058] Figure 8 This is a schematic diagram of the top model of a lychee provided in Embodiment 1 of the present invention;

[0059] Figure 9 This is a schematic diagram of the bottom model of a lychee provided in Embodiment 1 of the present invention;

[0060] Figure 10 This is a diagram showing the experimental analysis results of mechanical damage to litchi provided in Embodiment 1 of the present invention;

[0061] Figure 11 This is a schematic diagram of the process for extracting the shading rate of litchi point cloud based on the MVS-SFM algorithm for reconstructing the three-dimensional point cloud of litchi in Embodiment 1 of the present invention.

[0062] Figure 12This is a linear result diagram of the litchi coloring rate extracted based on the litchi RGB image and color space model and the manually measured coloring rate, provided in Embodiment 1 of the present invention.

[0063] Figure 13 This is a linear result diagram of the litchi point cloud coloring rate extracted and manually measured coloring rate extracted based on the MVS-SFM algorithm for reconstructing the three-dimensional point cloud of litchi according to Embodiment 1 of the present invention.

[0064] Figure 14 This is a linear graph showing the predicted volume of litchi and the manually measured volume provided in Embodiment 1 of the present invention;

[0065] Figure 15 This is a schematic diagram of the structure of the litchi phenotypic parameter detection system based on image processing and three-dimensional point cloud provided in Embodiment 2 of the present invention.

[0066] Among them, 1 is the dark box, 2 is the LED light source, 3 is the camera bracket, 4 is the depth camera, 5 is the lychee to be tested, 6 is the cloud server, and 7 is the lychee placement board. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0069] In the description of this invention, it should be understood that the terms "upper", "lower", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0070] Example 1

[0071] like Figure 1 As shown, a method for detecting litchi phenotypic parameters based on image processing and 3D point clouds includes the following steps:

[0072] S1. Obtain two-dimensional images of the lychee under test from multiple perspectives. The two-dimensional images include depth images and RGB images. Specifically, the multiple perspectives include four perspectives: left view, right view, top view, and bottom view.

[0073] Specifically, in this embodiment, an Intel Realsense D405 depth camera 4 is used to acquire depth images and RGB images of the lychee under test from multiple perspectives. Each pixel in the depth image represents the distance between the pixel and the plane of the depth camera 4, and the unit of distance is mm.

[0074] S2. Map the two-dimensional images of the litchi under multiple perspectives to obtain the original three-dimensional point cloud data of the litchi under test. Perform background segmentation and noise reduction filtering on the original three-dimensional point cloud data to obtain the three-dimensional point cloud data of the litchi under test. Calculate the point cloud curvature and normal vector of the three-dimensional point cloud data of the litchi under test, and statistically determine the curvature threshold used to judge the surface loss area of ​​the litchi. In this way, the point cloud of the litchi damage area is segmented, and the mechanical damage area of ​​the litchi under test is identified.

[0075] Specifically, the process of acquiring the original three-dimensional point cloud data of the litchi to be tested is as follows:

[0076] First, the depth images and RGB images of the litchi under test acquired by the depth camera 4 from multiple perspectives in step S1 are transmitted to the cloud server 6. In the cloud server 6, the depth images and RGB images of the litchi under test from multiple perspectives are processed by using Python software in conjunction with the corresponding programming driver of the depth camera 4, so that the depth images and RGB images are combined. Then, the two-dimensional pixel data is converted into points in three-dimensional space through the calibration parameters of the depth camera 4, and the color data in the RGB image is mapped to the corresponding three-dimensional points, so that the generated point cloud has real color information. Finally, the original three-dimensional point cloud of the litchi under test is obtained through the mapping relationship.

[0077] Specifically, such as Figure 3 As shown, the original 3D point cloud data of the litchi to be tested acquired by the depth camera 4 contains a large number of scattered points and isolated points. Furthermore, during the acquisition of point cloud data, due to the influence of equipment accuracy, operator experience, environmental factors, electromagnetic wave diffraction, changes in the surface properties of the measured object, and errors in the data stitching and registration process, some unavoidable noise and other point clouds unrelated to litchi will appear in the point cloud data. Therefore, in order to obtain better 3D point cloud data, this embodiment uses cloudcompare software to perform background segmentation on the acquired original 3D point cloud data of the litchi to be tested, and performs noise reduction filtering on the litchi point cloud based on the bounding box algorithm and the statistical outlier removal (SOR) method, finally obtaining a clean and complete litchi point cloud.

[0078] The surface curvature of a point cloud is a characteristic value used to describe the degree of change in the point cloud surface. Mechanical damage to litchi during growth, harvesting, and post-harvest grading causes depressions or protrusions on the litchi surface. These depressions or protrusions alter the original shape of the fruit surface, causing changes in the point cloud curvature in that area, typically manifesting as abnormally high or low values. Therefore, by calculating the surface curvature of the litchi point cloud, a threshold can be statistically determined. The point cloud area with curvature within this threshold range is the area where the litchi surface curvature changes; this area is the identified mechanical damage area on the litchi surface. This threshold range is the change curvature threshold of the litchi point cloud. This threshold can then be used to identify the mechanical damage area of ​​the litchi to be tested.

[0079] Specifically, the calculation principle of the point cloud curvature and normal vector of the three-dimensional point cloud data of the litchi to be tested is as follows:

[0080] Since surface curvature is described by eigenvalues ​​to represent the degree of variation in a point cloud, the curvature of a point can be obtained by estimating the normal vector of each point in the point cloud. The normal vector can be calculated using the tangent normal to the surface at that point. According to the Least Squares Method (LSM), a quadratic surface can be used to characterize the local region, and the normal vector can be estimated using a local region surface fitting method.

[0081] Since each point in a point cloud has a neighborhood point cloud that is approximated using a specific surface, the curvature of a point can be represented by the curvature of the local surface fitted to that point and its neighbors. Therefore, taking pi as the center point and uniformly selecting k points near pi, the equation of the quadratic surface can be expressed as:

[0082] z(x,y)=ax 2 +bxy+cy 2

[0083] According to the least squares principle, z i The sum of squares can be expressed as:

[0084]

[0085] From this, the values ​​of the coefficients a, b, and c of the quadratic surface equation can be obtained:

[0086]

[0087] The curve on the surface can then be represented as:

[0088] r = (x(t), y(t))

[0089] The unit normal vector at point Pi can be expressed as:

[0090]

[0091] If we take the normal vector at point Pi as the normal vector of the local surface in the neighborhood, then the covariance matrix of each point in the neighborhood is:

[0092]

[0093] C·X j =λ j ·X j j = 1, 2, 3

[0094] Where P0 is the centroid of the neighborhood points, k is the number of neighborhood points, and λ j and X j Let represent the eigenvalues ​​and eigenvectors of C, respectively.

[0095] The eigenvector corresponding to the smallest eigenvalue of matrix C is point P. i Given the normal vector, construct the covariance matrix. Calculate the eigenvalues ​​of the matrix through eigenvalue decomposition. When the eigenvalues ​​satisfy the condition λ0≤λ1≤λ2, the curvature of the neighborhood points can be expressed as:

[0096]

[0097] Finally, the obtained neighborhood curvature k i This refers to the curvature of the point cloud in the surface region.

[0098] Specifically, in the Cloudcompare software, after iterative analysis of the point cloud curvature of different lychee point cloud samples, it was finally concluded that the surface loss curvature of the lychee was measured with the highest accuracy when the curvature threshold was set to 1.81.

[0099] S3. Convert the RGB images of the lychee under multiple different viewpoints obtained in step S1 to the HSI color model space to obtain multiple lychee images under the HSI model from different viewpoints. Extract the red pixels in the multiple lychee images under the HSI model from different viewpoints by using a preset red pixel threshold and take the average value. Use the average value as the red coloring rate of the lychee under test.

[0100] Specifically, the process of step S3 is as follows:

[0101] First, a Python program with OpenCV is used to obtain the R, G, and B component values ​​of each pixel in the RGB images obtained from multiple different viewpoints in step S3. Then, the R, G, and B component values ​​of each pixel are normalized to obtain the color of the pixel located at spatial position (x,y) in the RGB color space, represented by the three values ​​of the R component R(x,y), G component G(x,y), and B component B(x,y). Next, the litchi region in the normalized RGB image is converted to the HSI color space. The H component H(x,y), S component S(x,y), and I component I(x,y) of a pixel in the HSI color space are represented as follows:

[0102]

[0103]

[0104]

[0105] in,

[0106] The RGB color space is based on a three-dimensional rectangular coordinate system and includes three original spectral components: R, G, and B. The HSI model is set according to the human visual system and consists of three components: H, S, and I. The H component represents the color of an object, the I component represents the brightness, the S component represents the saturation of the color, and θ(x,y) is the angle value in the H component conversion formula.

[0107] like Figure 4 As shown, the lychee images under the HSI model corresponding to the RGB images of the lychee under multiple different viewpoints are finally obtained according to the above formula;

[0108] Since a smaller H value for a pixel in the HSI color space indicates a redder hue, pixels with an H component less than the preset red pixel threshold are marked as red pixels. Then, the red pixel values ​​in the multiple lychee images from different HSI perspectives are calculated and averaged. This average value is used as the red coloring rate of the lychee under test. The red coloring rate is the ratio of the total area of ​​red pixels on the surface of the lychee to the total area of ​​all pixels on the lychee under test.

[0109] The method for obtaining the preset red pixel threshold is as follows:

[0110] First, different lychee image samples under the HSI model are obtained. Then, the red coloring rate corresponding to the sample is obtained by adjusting the red pixel threshold size. The obtained red coloring rate is verified by the lychee coloring rate standard. Finally, the range of red pixel threshold values ​​with the most accurate verification result is taken as the preset red pixel threshold.

[0111] Specifically, after conducting multiple experiments to measure the red coloring rate of lychee samples, the results obtained when the preset red pixel threshold value ranged from 35° to 40° were the most accurate.

[0112] S4. Based on the depth images f(x,y) of the lychee under multiple different viewpoints and the background depth image b(x,y), a depth image h(x,y) containing the thickness of the lychee is obtained. The depth image h(x,y) containing the thickness of the lychee is preprocessed. A 3D model of the top and bottom of the lychee under test is established using the preprocessed depth image h(x,y). The volume value of the lychee under test is calculated based on the 3D model of the top and bottom of the lychee under test.

[0113] The specific process of step S4 is as follows:

[0114] Since a depth image is an image or image channel containing information about the distances to the surfaces of scene objects at the viewpoint, also known as a distance image, it directly reflects the geometry of the visible surfaces of the target being photographed. Each non-zero grayscale pixel in the depth image represents the height of the lychee surface within that pixel region; therefore, the thickness of the lychee can be defined as the distance from the surface of the lychee to the platform on which it is placed.

[0115] Since the depth image f(x,y) of the lychee represents the distance between the lychee surface and the depth camera 4, this distance does not directly reflect the thickness of the lychee. Therefore, it is necessary to maintain the background depth image b(x,y) of the depth camera 4 at the same height and under the same shooting environment, and to construct a depth image h(x,y) containing the thickness of the lychee based on the depth image f(x,y) and the background depth image b(x,y). The depth image h(x,y) containing the thickness of the lychee is represented as follows:

[0116]

[0117] Then, a program was written using Python and OpenCV to perform erosion, dilation, and Gaussian smoothing operations on the depth image h(x,y) to remove small noise and fill small holes, resulting in the following: Figure 7 As shown

[0118] The preprocessed depth image h(x,y) was then imported into Matlab software to create 3D models of the top and bottom of the lychee to be tested. At this point, the non-zero pixel values ​​in image h(x,y) represent the thickness of the lychee at that point. Since the default pixel value unit for depth images directly saved after camera capture is sub-millimeter, the unit was converted to mm. Simultaneously, the number of non-zero pixel values ​​represents the surface area S of the lychee projected into the two-dimensional image space. Calculations showed that the actual area of ​​each pixel is equal to 0.089 mm. 2 .

[0119] Specifically, the process for obtaining the actual volume of each pixel is as follows:

[0120] First, using the same camera height and angle as when capturing lychee data, a 100 square centimeter area of ​​green cardstock was photographed vertically downwards. Then, a program written in Python and OpenCV was used to segment the green cardstock area, calculate the number of pixels in that area in the image, and finally divide the actual area of ​​the cardstock by the number of pixels to obtain the actual area of ​​each pixel. It should be noted that this value represents the actual area represented by each pixel in the depth image generated by that depth camera, and does not change with external environmental conditions; therefore, it can be reused in subsequent lychee detection.

[0121] The volume of a lychee is defined as the sum of the volumes of every pixel within its surface area. Since a depth image h(x,y) only contains half the lychee's thickness information, the volume of the lychee is obtained by subtracting the smallest non-zero pixel value from each non-zero pixel value in h(x,y). Finally, the volume of the entire lychee can be calculated using the 3D surface models of its top and bottom.

[0122] The volume of a lychee is calculated as follows:

[0123]

[0124] V 总 =V 顶 +V 底

[0125] Conclusions and Analysis

[0126] To provide efficient technical means and data support for post-harvest grading of litchi, the following experimental method is used to analyze the litchi phenotypic parameter detection method based on image processing and 3D point cloud described in this embodiment.

[0127] This experiment used the Fei Zi Xiao variety of lychee as the sample, which was collected from Zengcheng District, Guangzhou City, Guangdong Province.

[0128] (1) Analysis of mechanical injury data

[0129] To verify the accuracy of point cloud loss region identification for litchi, 50 litchis with different degrees of mechanical damage were prepared for the experiment, and the accuracy of mechanical damage identification of litchis under different number of viewpoints was tested.

[0130] Table 1 below shows the damage detection accuracy of litchi from single, two, three, and four viewing angles. Curvature calculation was performed after synthesizing the point cloud to obtain the detection results of mechanical damage on the litchi surface. The results in Table 1 show that the detection rate of mechanical damage on litchi from a single viewing angle is low, with a large number of litchis being missed. As the number of detection angles increases, the accuracy of mechanical damage detection significantly improves, reaching 94% for litchis from four detection angles. The detection results indicate that the method of calculating the curvature change value of the point cloud can quickly and effectively detect mechanical damage on litchi under certain error tolerance conditions, demonstrating high detection accuracy.

[0131] Perspective (individual) Total number of samples Correct number of tests (pieces) Detection accuracy (%) 1 50 30 60 2 50 38 76 3 50 42 84 4 50 47 94

[0132] Table 1 Results of mechanical damage detection in litchi

[0133] (2) Data analysis of litchi coloring rate extraction

[0134] To verify the extraction results of litchi coloring rate, this experiment used manual judgment of litchi coloring rate as real data for verification. In addition, the experiment compared the litchi point cloud coloring rate extracted by reconstructing litchi 3D point cloud based on MVS-SFM algorithm in the existing technology with the present invention.

[0135] Specifically, by Figure 12 It can be seen that the results obtained from the litchi coloring rate extraction step provided by the present invention and the litchi coloring rate judged manually are significantly linearly correlated. The correlation coefficient R2 and root mean square error (RMSE) of litchi coloring rate in RGB image are 0.9574 and 0.0809%, respectively, the mean absolute error (MAE) is 6.33%, and the mean relative error (MRE) is 4.17%.

[0136] Specifically, such as Figure 11 As shown, the process of extracting the shading rate of litchi point cloud from the reconstructed 3D point cloud based on the MVS-SFM algorithm is as follows:

[0137] First, the litchi images were reconstructed using the MVS-SFM algorithm to obtain litchi point clouds. Then, the litchi point clouds were segmented into planes based on the Random Sample Consensus (RANSAC) algorithm, and a complete and clean litchi point cloud was obtained through statistical filtering and other processing. Finally, the red coloring rate of the litchi surface was statistically obtained based on the determined litchi red threshold range.

[0138] like Figure 13 As shown, the litchi point cloud coloring rate extracted from the reconstructed 3D point cloud based on the MVS-SFM algorithm also exhibits a clear linear distribution compared to the manually judged litchi coloring rate, with R² and RMSE of 0.9205 and 0.0563%, respectively, MAE of 4.37%, and MRE of 6.01%. The results indicate that the litchi coloring rate obtained through computer vision processing has a good correlation with the manually measured coloring rate.

[0139] The results show that although the MVS-SFM algorithm can obtain a relatively complete litchi point cloud and is easy to operate, the sparse and dense reconstruction process of a single litchi is time-consuming and has high equipment requirements. On average, it takes 10 minutes to reconstruct the point cloud of a litchi and extract the color rate. However, the litchi color rate extraction provided by this invention takes less than 1 minute, which improves the overall speed by 90%.

[0140] (3) Data analysis of litchi volume extraction

[0141] To verify the results of this invention, which uses depth images of lychees to create 3D models of lychees and extract their volume, this experiment uses manual volume measurement to verify the lychee detection results. Specifically, the volume of the lychee is measured using the water displacement method.

[0142] like Figure 14 As shown, after multiple experiments, the predicted volume of the litchi using the depth image of the litchi and the volume extracted by the present invention showed an obvious linear distribution with the manually measured volume. The R2 and RMSE of the predicted litchi volume were 0.8901 and 2.4733 cm3, respectively, the MAE was 1.59 cm3, and the MRE was 7.94%.

[0143] The detection results show that the 3D model established by the present invention using the lychee depth image and the result of estimating the lychee volume based on the pixel value thickness of the lychee 3D model have a good correlation with the results of manual measurement, and the detection accuracy reaches 89%.

[0144] In summary, this invention acquires RGB color images and depth images of lychees from four different perspectives, obtains three-dimensional point clouds of lychees through mapping, and utilizes image processing and three-dimensional point cloud technology to process the lychee images and point clouds, achieving non-destructive and rapid extraction of three phenotypic parameters: mechanical damage, coloring rate, and volume of lychees. This enhances the economic value of lychees and provides a scientific basis for lychee quality grading.

[0145] Example 2:

[0146] like Figure 15 As shown, a litchi phenotypic parameter detection system based on image processing and three-dimensional point cloud is used in the litchi phenotypic parameter detection method based on image processing and three-dimensional point cloud. The system includes a litchi image acquisition platform and a cloud server 6.

[0147] like Figure 14 As shown, the lychee image acquisition platform includes a dark box 1, an LED light source 2, a lychee placement plate 7, a depth camera 4, and a camera bracket 3. The LED light source 2 is installed at the top of the dark box 1, the depth camera 4 is installed below the LED light source 2 via the camera bracket 3, and the lychee placement plate 7 is installed at the bottom of the dark box 1, with the lens of the depth camera 4 facing the lychee placement plate 7. The lychee placement plate 7 is used to place the lychees to be tested, and the depth camera 4 is communicatively connected to the cloud server 6.

[0148] The workflow of the lychee image acquisition platform is as follows:

[0149] During testing, the lychee to be tested is first placed directly below the depth camera 4, with the camera lens horizontally facing the center of the lychee. Then, the lychee is rotated evenly by hand, and depth images and RGB images of the lychee from four perspectives: left, right, top, and bottom. The depth camera 4 processes the acquired depth images and RGB images from the four perspectives to obtain the original three-dimensional point cloud data of the lychee, and transmits the depth images, RGB images, and original three-dimensional point cloud data of the lychee from the four perspectives to the cloud server 6.

[0150] The cloud server 6 includes a processor and a memory; the memory stores non-transitory computer instructions, which, when executed by the processor, perform the litchi phenotypic parameter detection method based on image processing and three-dimensional point cloud as described in Embodiment 1.

[0151] The depth camera 4 is an Intel Realsense D405 depth camera.

[0152] Specifically, the Intel Realsense D405 depth camera is a close-up depth camera with a frame rate of 90 frames per second, a depth range of 7cm-50cm, a depth field of view of 87° (horizontal) × 58° (vertical), a frame rate of 90 frames per second, a resolution of 1280×720 in color camera (RGB mode), and a maximum output resolution of 1280×720 in depth camera mode.

[0153] In the description of this invention, it should be noted that, unless otherwise explicitly specified and agreed, the terms "set," "install," and "connect" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0154] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for detecting litchi phenotypic parameters based on image processing and 3D point clouds, characterized in that, Includes the following steps: S1. Obtain two-dimensional images of the lychee to be tested from multiple perspectives, wherein the two-dimensional images include depth images and RGB images; S2. Map the two-dimensional images of the litchi under multiple perspectives to obtain the original three-dimensional point cloud data of the litchi under test. Perform background segmentation and noise reduction filtering on the original three-dimensional point cloud data to obtain the three-dimensional point cloud data of the litchi under test. Calculate the point cloud curvature and normal vector of the three-dimensional point cloud data of the litchi under test, and statistically determine the curvature threshold used to judge the surface loss area of ​​the litchi. In this way, the point cloud of the litchi damage area is segmented, and the mechanical damage area of ​​the litchi under test is identified. S3. Convert the RGB images of the lychee under multiple different viewpoints obtained in step S1 to the HSI color model space to obtain multiple lychee images under the HSI model from different viewpoints. Extract the red pixels in the multiple lychee images under the HSI model from different viewpoints by using a preset red pixel threshold and take the average value. Use the average value as the red coloring rate of the lychee under test. S4. Based on the depth images f(x,y) of the lychee under multiple different viewpoints and the background depth image b(x,y), a depth image h(x,y) containing the thickness of the lychee is obtained. The depth image h(x,y) containing the thickness of the lychee is preprocessed. A 3D model of the top and bottom of the lychee under test is established using the preprocessed depth image h(x,y). The volume value of the lychee under test is calculated based on the 3D model of the top and bottom of the lychee under test.

2. The method for detecting litchi phenotypic parameters based on image processing and 3D point clouds according to claim 1, characterized in that, The multiple perspectives mentioned in step S1 include four perspectives: left view, right view, top view, and bottom view.

3. The method for detecting litchi phenotypic parameters based on image processing and 3D point clouds according to claim 1, characterized in that, The derivation process of the point cloud curvature and normal vector of the three-dimensional point cloud data of the litchi to be tested in step S2 is as follows: Since each point in a point cloud has a neighborhood point cloud that is approximated using a specific surface, the curvature of a point can be represented by the curvature of the local surface fitted to that point and its neighbors. Therefore, taking pi as the center point and uniformly selecting k points near pi, the equation of the quadratic surface can be expressed as: z(x,y)=ax 2 +bxy+cy 2 According to the least squares principle, z i The sum of squares can be expressed as: From this, the values ​​of the coefficients a, b, and c of the quadratic surface equation can be obtained: Then the curve on the surface can be represented as: r = (x(t), y(t)) The unit normal vector at point Pi can be expressed as: If we take the normal vector at point Pi as the normal vector of the local surface in the neighborhood, then the covariance matrix of each point in the neighborhood is: C·X j =λ j ·X j ,j=1,2,3 Where P0 is the centroid of the neighborhood points, k is the number of neighborhood points, and λ j and X j Let C represent the eigenvalues ​​and eigenvectors, respectively. The eigenvector corresponding to the smallest eigenvalue of matrix C is point P. i Given the normal vector, construct the covariance matrix. Calculate the eigenvalues ​​of the matrix through eigenvalue decomposition. When the eigenvalues ​​satisfy the condition λ0≤λ1≤λ2, the curvature of the neighborhood points can be expressed as: Ultimately, the neighborhood curvature k i This refers to the curvature of the neighborhood surface.

4. The method for detecting litchi phenotypic parameters based on image processing and 3D point clouds according to claim 1, characterized in that, The curvature threshold for determining the lost area on the surface of the litchi in step S2 is 1.

81.

5. The method for detecting litchi phenotypic parameters based on image processing and 3D point clouds according to claim 1, characterized in that, The specific process of step S3 is as follows: First, the R, G, and B components of the RGB images obtained from multiple different viewpoints in step S3 are normalized. The color of a pixel located at spatial position (x,y) in the RGB color space is represented by three values: R(x,y), G(x,y), and B(x,y). Then, the litchi region in the normalized RGB image is converted to the HSI color space. The H(x,y), S(x,y), and I(x,y) components of a pixel in the HSI color space are represented as follows: in, Finally, based on the above formula, the lychee images under the HSI model corresponding to the RGB images of the lychee under multiple different viewpoints were obtained; Then, according to the preset red pixel threshold, the red pixel values ​​of each of the multiple lychee images under the HSI model from different perspectives are extracted and the average value is calculated. The average value is used as the red coloring rate of the lychee to be tested. The red coloring rate is the ratio of the red pixel value of the entire surface of the lychee to the total area of ​​all pixels of the lychee to be tested. The red pixel value is the red H component of the lychee surface in the lychee image under the HSI model.

6. The method for detecting litchi phenotypic parameters based on image processing and 3D point clouds according to claim 5, characterized in that, The preset red pixel threshold value ranges from 35° to 40°.

7. The method for detecting litchi phenotypic parameters based on image processing and 3D point clouds according to claim 1, characterized in that, In step S4, the preprocessing method for the depth image h(x,y) containing the thickness of the lychee is erosion, dilation and Gaussian smoothing.

8. The method for detecting litchi phenotypic parameters based on image processing and 3D point clouds according to claim 1, characterized in that, The depth image f(x,y) and the background depth image b(x,y) mentioned in step S4 are captured in the same environment and at the same height.

9. A litchi phenotypic parameter detection system based on image processing and 3D point cloud, used to implement the litchi phenotypic parameter detection method based on image processing and 3D point cloud as described in any one of claims 1-8, characterized in that, The system includes a litchi image acquisition platform and a cloud server; The lychee image acquisition platform includes a dark box, an LED light source, a lychee placement plate, a depth camera, and a camera bracket. The LED light source is installed at the top of the dark box, the depth camera is installed below the LED light source via the camera bracket, and the lychee placement plate is installed at the bottom of the dark box, with the lens of the depth camera facing the lychee placement plate. The lychee placement plate is used to place the lychees to be tested, and the depth camera is communicatively connected to the cloud server. The cloud server includes a processor and a memory; the memory stores non-transitory computer instructions, which, when executed by the processor, perform the litchi phenotypic parameter detection method based on image processing and three-dimensional point cloud as described in any one of claims 1-8.

10. The litchi phenotypic parameter detection system based on image processing and three-dimensional point cloud according to claim 9, characterized in that, The depth camera used is an Intel Realsense D405 depth camera.

Images