Plant segmentation method and device based on fusion of point cloud projection and image segmentation

Abstract

The application provides a plant segmentation method and device based on point cloud projection and image segmentation fusion. A camera and a three-dimensional laser are used to collect corresponding plant images and plant point cloud data in the same field of view. The plant point cloud data can be projected based on the observation direction of the three-dimensional laser, and a projection map is obtained. In order to correct the errors of the projection map, the projection map is mapped through the optimal homography matrix according to the re-projection error between the projection map and the plant image, and an accurate mapping map is obtained. Then, the plant image is segmented to obtain a plant organ image segmentation result. The plant organ image segmentation result is backfilled into the plant point cloud data according to the mapping map, and an accurate three-dimensional plant organ point cloud segmentation result is obtained. Various phenotype data of the plant organ can be determined according to the plant organ point cloud segmentation result, so as to facilitate the phenotype research and breeding analysis of the plant organ.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a plant segmentation method and device based on the fusion of point cloud projection and image segmentation. Background Technology

[0002] With the development of agricultural informatization and plant phenotyping, obtaining the structural characteristics, morphological parameters and growth status of plants (e.g., crops) using digital means has become a research hotspot.

[0003] Among these, three-dimensional structural traits of plants, such as length, angle, and spatial distribution, are of great value for screening various plant properties. Currently, commonly used methods for obtaining plant traits are mainly based on manual measurement or image analysis. However, these methods generally suffer from low efficiency, high subjectivity, and the inability to obtain three-dimensional structural information, making it difficult to meet the needs of large-scale, high-throughput field plant phenotypic studies. Summary of the Invention

[0004] In view of this, the purpose of this application is to propose a plant segmentation method and device based on the fusion of point cloud projection and image segmentation, so as to solve or partially solve the above-mentioned technical problems.

[0005] To achieve the above objectives, this application provides a plant segmentation method based on the fusion of point cloud projection and image segmentation, comprising:

[0006] Using cameras and 3D lasers, plant images and point cloud data were collected from the same field of view of the plant distribution area.

[0007] Determine the observation direction of the three-dimensional laser, and project the plant point cloud data onto the observation direction to obtain a projection map;

[0008] The optimal homography matrix is ​​determined based on the reprojection error between the projection map and the plant image;

[0009] Based on the optimal homography matrix, the projection map is mapped onto the plant image to obtain a mapping map;

[0010] The plant organs in the plant image are segmented to obtain the plant organ image segmentation result;

[0011] Based on the mapping map, the plant organ image segmentation results are backfilled into the plant point cloud data to obtain the plant organ point cloud segmentation results;

[0012] Determine the mean point in the point cloud segmentation result of the plant organ, and construct a covariance matrix based on the mean point;

[0013] Based on the covariance matrix, eigenvalue decomposition is performed to obtain multiple vectors representing the growth direction of organs. The maximum value among the multiple vectors representing the growth direction of organs is determined, and the maximum value is taken as the main growth direction vector of plant organs.

[0014] The phenotypic data of the plant organs are determined by combining the main growth direction vector of the plant organs with the point cloud segmentation results of the plant organs.

[0015] Based on the same inventive concept, this disclosure also provides a plant segmentation system based on the fusion of point cloud projection and image segmentation, comprising:

[0016] The acquisition module is configured to use a camera and a 3D laser to acquire corresponding plant images and plant point cloud data based on the same field of view of the plant distribution area.

[0017] The point cloud projection processing module is configured to determine the observation direction of the three-dimensional laser, project the plant point cloud data onto the observation direction, and obtain a projection map.

[0018] The homography matrix determination module is configured to determine the optimal homography matrix based on the reprojection error between the projection image and the plant image;

[0019] The mapping processing module is configured to map the projection map onto the plant image according to the optimal homography matrix to obtain a mapping map;

[0020] The organ segmentation processing module is configured to segment plant organs in the plant image to obtain plant organ image segmentation results.

[0021] The organ segmentation result backfilling module is configured to backfill the plant organ image segmentation result into the plant point cloud data based on the mapping map, so as to obtain the plant organ point cloud segmentation result;

[0022] The covariance processing module is configured to determine the mean point in the point cloud segmentation result of the plant organ and construct a covariance matrix based on the mean point.

[0023] The organ growth main direction determination module is configured to perform eigenvalue decomposition based on the covariance matrix to obtain multiple vectors representing organ growth directions, determine the maximum value among the multiple vectors representing organ growth directions, and use the maximum value as the plant organ growth main direction vector.

[0024] The phenotypic data determination module is configured to combine the main growth direction vector of the plant organ with the point cloud segmentation result of the plant organ to determine the phenotypic data of the plant organ.

[0025] Based on the same inventive concept, this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method described above when executing the computer program.

[0026] Based on the same inventive concept, this disclosure also provides a non-transitory computer-readable storage medium that stores computer instructions for causing a computer to perform the method described above.

[0027] As described above, the plant segmentation method and device based on point cloud projection and image segmentation fusion provided in this application utilizes a camera and a 3D laser to acquire corresponding plant images and plant point cloud data in the same field of view. The plant point cloud data can then be projected using the observation direction of the 3D laser as a reference, resulting in a projection image. However, there is an error between the projection image and the plant image. To correct this error, an optimal homography matrix is ​​determined based on the reprojection error between the projection image and the plant image. This optimal homography matrix is ​​then used to map the projection image, resulting in an accurate mapping image that better matches the plant image. Next, the plant image is segmented to extract plant organs, yielding accurate plant organ image segmentation results. To obtain accurate 3D plant organ images, the 2D plant... The organ image segmentation results are backfilled into the plant point cloud data to obtain accurate 3D plant organ point cloud segmentation results. Based on this segmentation, the mean point can be determined and a covariance matrix constructed. Then, through eigenvalue decomposition of the covariance matrix, multiple vectors representing organ growth directions are obtained. However, the smaller values ​​in these vectors are inaccurate, so the largest value is used as the principal growth direction vector. This allows the plant organ's growth direction to be determined. Combining this growth direction with the obtained 3D plant organ point cloud segmentation results, various phenotypic data of the plant organ can be accurately determined. Since both the obtained plant organ growth direction and the 3D plant organ point cloud segmentation results are relatively accurate, the determined phenotypic data of the plant organ will also be relatively accurate, facilitating phenotypic research and breeding analysis of plant organs. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart illustrating the plant segmentation method based on point cloud projection and image segmentation fusion according to an embodiment of this application.

[0030] Figure 2 This is a schematic diagram of the plant segmentation system based on point cloud projection and image segmentation fusion according to an embodiment of this application.

[0031] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0033] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0034] RGB: An industry color standard that uses variations in the three color channels—red (R), green (G), and blue (B)—and their superposition to obtain a variety of colors. RGB represents the colors of the three channels: red, green, and blue.

[0035] UNet: A U-shaped convolutional neural network architecture for image segmentation, proposed by Olaf Ronneberger et al. in 2015, and widely used in fields such as medical image segmentation.

[0036] K-Net is a model that provides a unified, concise, and efficient framework for image segmentation tasks, demonstrating outstanding performance in semantic, instance, and panoptic segmentation.

[0037] TTL: Transistor-to-Transistor Logic.

[0038] DeepLab is a series of semantic segmentation models developed by Google. It achieves high-precision pixel-level segmentation through dilated convolution and the ASPP (Atrous Spatial Pyramid Pooling) module.

[0039] In related technologies, 3D point cloud technology is increasingly being applied to plant (e.g., crop) structure modeling, where 3D geometric information of plants can be acquired using devices such as LiDAR and depth cameras. However, directly segmenting target data from point cloud data still has the following shortcomings:

[0040] (1) Point cloud data is sparse and irregular in structure, making feature expression difficult. Training a three-dimensional semantic segmentation model is complex and computationally expensive.

[0041] (2) The cost of point cloud annotation is much higher than that of image annotation. Manual point-by-point annotation is difficult, and point cloud noise and severe occlusion are common in plant scenes, resulting in unstable annotation quality.

[0042] (3) The detection accuracy for some small organs of plants (e.g., wheat ears) is limited, which affects the integrity and accuracy of phenotypic characteristics.

[0043] Compared with point clouds, RGB images have advantages such as clear texture, simple annotation, and the ability to utilize mature semantic segmentation models. However, two-dimensional images lack depth and spatial structure information, making them difficult to use directly for the extraction of three-dimensional wheat ear traits.

[0044] The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0045] The plant segmentation method based on point cloud projection and image segmentation fusion proposed in this application embodiment is preferably a crop, and the crop is preferably wheat.

[0046] like Figure 1 As shown, the method includes:

[0047] Step 101: Using a camera and a 3D laser, collect corresponding plant images and plant point cloud data for the plant distribution area based on the same field of view.

[0048] In practice, the camera and the 3D laser are placed within the plant distribution area, or on the same side of the plant distribution area, ensuring that the camera and the 3D laser can capture images of the plant distribution area from the same field of view. The camera captures the RGB pixels corresponding to the plant distribution area, obtaining a plant image (e.g., a two-dimensional RGB image); the 3D laser captures the depth point cloud of the plants corresponding to the plant distribution area, obtaining plant point cloud data.

[0049] Step 102: Determine the observation direction of the three-dimensional laser, and project the plant point cloud data onto the observation direction to obtain a projection map.

[0050] In practice, since the 3D laser and the camera are located in the same field of view, the plant point cloud data collected by the 3D laser can be projected onto the observation direction to obtain a two-dimensional projection map. However, there are still some deviations between the projection map and the plant image. The projection map cannot directly represent the correspondence between the plant point cloud data and the plant image.

[0051] Step 103: Determine the optimal homography matrix based on the reprojection error between the projection map and the plant image.

[0052] In practice, reprojection error represents the degree of pixel deviation between two images. Projection mapping is performed using various homography matrices based on the projected image, and the reprojection error between the mapped image and the corresponding plant image is calculated. The optimal homography matrix, which minimizes the reprojection error, is then selected.

[0053] Step 104: Based on the optimal homography matrix, map the projection map onto the plant image to obtain the mapping map.

[0054] In practice, after obtaining the optimal homography matrix, the projection image can be mapped to the plant image according to the optimal homography matrix, thereby obtaining the mapping image that best matches the plant image.

[0055] Step 105: Segment the plant organs in the plant image to obtain the plant organ image segmentation result.

[0056] In practice, semantic segmentation models capable of image segmentation (e.g., UNet, K-Net, DeepLab, which are trained) can be used to segment plant images and separate the plant organs, thus obtaining accurate plant organ image segmentation results.

[0057] The plant organs include at least one of the following: roots, stems, leaves, flowers, and fruits (preferably, the fruits of crops, such as the ears of wheat).

[0058] Step 106: Based on the mapping map, the plant organ image segmentation results are backfilled into the plant point cloud data to obtain the plant organ point cloud segmentation results.

[0059] In practice, the plant organ image segmentation is two-dimensional and cannot accurately analyze the phenotypic traits of plant organs. Therefore, it is necessary to use the above-mentioned mapping map to perform depth backfilling processing on the two-dimensional plant organ image segmentation result according to the mapping relationship between the plant point cloud data and the plant image represented by the mapping map, so as to obtain accurate plant organ point cloud segmentation results.

[0060] Step 107: Determine the mean point in the point cloud segmentation result of the plant organ, and construct a covariance matrix based on the mean point.

[0061] In practice, the depth value corresponding to each point cloud in the plant organ point cloud segmentation result is averaged to determine the mean point, which represents the center position of the plant organ point cloud segmentation result. The covariance matrix is ​​then constructed based on this mean point.

[0062] Step 108: Perform eigenvalue decomposition based on the covariance matrix to obtain multiple vectors representing organ growth directions, determine the maximum value among the multiple vectors representing organ growth directions, and use the maximum value as the main growth direction vector of the plant organ.

[0063] In practice, the covariance matrix is ​​decomposed into eigenvalues, and vector solutions are obtained based on each eigenvalue to obtain the corresponding non-zero vector solutions. These non-zero vector solutions represent the vectors of organ growth direction. However, the smaller values ​​represent the wrong growth direction (for example, these vectors of wheat ears may mostly represent the growth direction of the awns of wheat grains). Therefore, the maximum value is selected as the main growth direction vector of plant organs to characterize the main growth direction of plant organs.

[0064] Step 109: Combine the main growth direction vector of the plant organ with the point cloud segmentation result of the plant organ to determine the phenotypic data of the plant organ.

[0065] In practice, the main growth direction vector of plant organs can be combined with the point cloud segmentation results of plant organs to form a set of data, and the phenotypic data can be determined based on it. Since the main growth direction vector of plant organs and the point cloud segmentation results of plant organs are relatively accurate, the phenotypic data of plant organs determined will also be relatively accurate.

[0066] The above scheme utilizes a camera and a 3D laser to acquire corresponding plant images and plant point cloud data within the same field of view. The plant point cloud data can then be projected using the observation direction of the 3D laser as a reference, resulting in a projection map. However, an error exists between this projection map and the plant image. To correct this error, an optimal homography matrix is ​​determined based on the reprojection error between the projection map and the plant image. This optimal homography matrix is ​​then used to map the projection map, resulting in an accurate mapping map that better matches the plant image. Next, the plant image is segmented to extract plant organs, yielding accurate plant organ image segmentation results. To obtain accurate 3D plant organ images, the 2D plant organ image segmentation results are backfilled into the plant point cloud data based on the obtained mapping map. According to the method, accurate three-dimensional plant organ point cloud segmentation results are obtained. Based on these results, the mean point can be determined and a covariance matrix can be constructed. Then, through eigenvalue decomposition of the covariance matrix, multiple vectors representing organ growth directions are obtained. However, the smaller values ​​of these vectors are inaccurate, so the maximum value is taken as the principal growth direction vector of the plant organ. This allows the plant organ growth direction to be determined based on the principal growth direction vector. Combining the plant organ growth direction with the obtained three-dimensional plant organ point cloud segmentation results, various phenotypic data of the plant organ can be accurately determined. Since both the obtained plant organ growth direction and the three-dimensional plant organ point cloud segmentation results are relatively accurate, the determined phenotypic data of the plant organ will also be relatively accurate, facilitating phenotypic research and breeding analysis of plant organs.

[0067] In some embodiments, the camera and the 3D laser can be rigidly fixed to the same frame, ensuring that their poses remain constant. To ensure the accuracy of the plant point cloud data projected onto the plant image, the extrinsic parameter matrix between the 3D laser and the camera needs to be obtained. During calibration, a checkerboard-based joint calibration method, a reflector method, or a vision-laser hybrid calibration method can be used to ensure that the extrinsic parameter matrix can be reliably obtained under various lighting conditions.

[0068] During data collection in plant distribution areas (e.g., fields or greenhouses), a high frame rate synchronous acquisition method is used to reduce the impact of wind-induced plant swaying, ensuring high temporal consistency between plant images and plant point cloud data. Alternatively, cameras with hardware triggering capabilities and 3D lasers can be selected to achieve strict synchronization in sampling time via TTL signals.

[0069] The transposed 3D coordinates corresponding to the plant image are obtained by multiplying the extrinsic parameter matrix with the transposed plant point cloud data.

[0070] Since cameras typically use a pinhole imaging model, after obtaining the transformed 3D coordinates, the pinhole imaging model can be used for projection processing. The corresponding projection formula is:

[0071] ; ;in, The x-coordinate in the coordinate system corresponding to the plant image. This represents the ordinate in the coordinate system corresponding to the plant image. The horizontal axis represents the imaging scale of the camera. The vertical axis represents the imaging scale of the camera. Let x be the x-coordinate of the i-th point in the plant point cloud data in the coordinate system corresponding to the camera. Let be the ordinate of the i-th point in the plant point cloud data in the coordinate system corresponding to the camera. Let be the vertical coordinate of the i-th point in the plant point cloud data in the coordinate system corresponding to the camera. This represents the x-coordinate of the pixel corresponding to the geometric center of the camera's image sensor. This is the ordinate of the pixel corresponding to the geometric center of the camera's image sensor.

[0072] To ensure more accurate projection results, in practice, it is usually necessary to first correct the distortion of the plant image and ensure the validity of the point cloud on the imaging plane, i.e., only when... >0 and A point is considered a valid projection point only if it falls within the size range of the plant image.

[0073] The mapping matrix M is constructed as follows:

[0074] .

[0075] The mapping matrix M records the mapping of "point cloud point number → pixel coordinates" of plant point cloud data. It is the core data structure of this embodiment and can be stored using a hash table, compressed sparse table or two-dimensional index array.

[0076] However, the relative position between the camera and the 3D laser may not be strictly fixed in general, and the mapping matrix obtained by the above method is not accurate enough. Therefore, the following method is adopted:

[0077] In some embodiments, step 103 includes:

[0078] Step 1031: Determine the homography matrix H.

[0079] Step 1032: Determine a predetermined number of first feature point pairs in the projection map, and perform mapping prediction on the predetermined number of first feature point pairs based on the homography matrix to obtain a predetermined number of predicted feature point pairs.

[0080] In practice, since the relative posture of the camera and the 3D laser is unknown or not fixed during the acquisition process, it is necessary to obtain the mapping relationship from the projection map to the plant image through image registration.

[0081] Step 1033: Obtain a predetermined number of second feature point pairs in the plant image that correspond to a predetermined number of first feature point pairs.

[0082] Step 1034: Determine the geometric error between a predetermined number of predicted feature point pairs and a predetermined number of second feature point pairs.

[0083] Step 1034 includes:

[0084] Step 10341: Determine two second feature points in the second feature point pair and two predicted feature points in the predicted feature point pair.

[0085] in, and For the two second feature points in the j-th pair of second feature points, and These are the two predicted feature points in the j-th predicted feature point pair.

[0086] Step 10342: According to the correspondence between the two second feature points and the two predicted feature points, the difference is calculated and then squared to obtain two squared results. The two squared results are then summed to obtain the summation result.

[0087] In practice, the formula for the summation result is: .

[0088] Step 10343: A predetermined number of summation results are obtained between a predetermined number of predicted feature point pairs and a predetermined number of second feature point pairs. The predetermined number of summation results are accumulated to obtain the geometric error.

[0089] In practical implementation, the formula for geometric error is:

[0090] ,in, Let K be the geometric error of the current homography matrix H, and K be a predetermined number.

[0091] Step 1035: Minimize the geometric error and adjust the homography matrix to obtain the adjusted homography matrix. The adjusted homography matrix corresponding to the minimum geometric error is taken as the optimal homography matrix.

[0092] By using the above scheme, after minimizing the geometric error, the adjusted homography matrix corresponding to the smallest geometric error can best represent the transformation relationship from the projection image to the plant image.

[0093] In some embodiments, step 104 includes:

[0094] Step 1041: Determine the plant image pixel matrix corresponding to the plant image. .

[0095] Step 1042: Multiply the optimal homography matrix by the plant image pixel matrix, and then transpose the result to obtain the mapping map.

[0096] In practical implementation, the formula relating the mapping map to the pixel matrix of the plant image is as follows:

[0097] ,in, Let H be the pixel matrix of the projection image, and H be the optimal homography matrix.

[0098] Using the above method, even if there are some deviations in the poses of the camera and the 3D laser, a mapping diagram reflecting the projection mapping relationship between plant point cloud data and plant images can be obtained.

[0099] As some embodiments, step 105 includes:

[0100] Pre-collect some plant sample images with accurate labels of real plant organ masks, and build deep learning models (e.g., using UNet, K-Net, or DeepLab).

[0101] Plant sample images are input into a deep learning model for training. The corresponding formula during training is as follows:

[0102] , among which, I rgb Images of plant samples. is a trainable parameter, and S is the output plant organ mask result.

[0103] The loss function L, which combines cross-entropy and Dice entropy, is given by the following formula: ,in, For cross-entropy, For Dice entropy, As weight.

[0104] The deep learning model is adjusted based on the hybrid loss function to complete the training, and the semantic segmentation model is obtained after the deep learning model is trained.

[0105] During the training process, plant sample images can be enhanced using at least one of the following methods: random rotation, color perturbation, illumination change simulation, random occlusion, etc.

[0106] Once the semantic segmentation model is trained, plant images can be input into the model for processing, and the corresponding plant organ image segmentation results can be output.

[0107] As some embodiments, step 106 includes:

[0108] After obtaining the plant organ image segmentation results, the plant organ image segmentation results are backfilled into the plant point cloud data according to the mapping matrix M or the aforementioned mapping diagram to obtain the plant organ point cloud segmentation results. The specific rules are as follows:

[0109] ;in, For the i-th point in the point cloud, Represents the pixel corresponding to the i-th point cloud point. This region corresponds to the segmentation result of the plant organ image.

[0110] To prevent point cloud noise caused by single-point misjudgment, the backfilled point cloud was further fine-tuned using region growing or voxel filtering to ensure continuous and natural-looking plant organ point cloud segmentation results. The final plant organ point cloud segmentation results are shown below. It is a subset of the original plant point cloud data, and its spatial accuracy depends on the camera resolution, point cloud density, and calibration accuracy.

[0111] In some embodiments, step 109 includes:

[0112] Step 1091: Project the plant organ point cloud segmentation result according to the main growth direction vector of the plant organ to obtain the plant organ projection result of the plant organ point cloud segmentation result on the main growth direction vector of the plant organ.

[0113] In practice, the formula corresponding to the projection results of plant organs is: ,in, Let be the projection coordinates of the i-th point cloud in the plant organ projection result. For the i-th point cloud in the point cloud segmentation result of plant organs, The mean point in the point cloud segmentation results of plant organs. Let T be the principal direction vector of plant organ growth, and let T denote the transpose.

[0114] Step 1092: Determine the maximum and minimum projection coordinate values ​​in the projection results of the plant organ, and take the difference between the maximum and minimum projection coordinate values ​​as the length of the plant organ.

[0115] In practice, the maximum projected coordinate value is And the minimum projected coordinates are .

[0116] The corresponding plant organ length L is: .

[0117] Step 1093: Determine the angle between the main growth direction vector of the plant organ and the vertical direction, and use the angle as the plant growth tilt angle.

[0118] In practice, ,in The angle at which the plant grows. Indicates the vertical direction.

[0119] Step 1094: The main growth direction vector of the plant organ, the length of the plant organ, and the plant growth tilt angle are used as phenotypic data of the plant organ.

[0120] The above scheme can obtain the plant organ growth main direction vector that accurately represents the main direction of organ growth, the plant organ length that can accurately characterize the size of the plant organ, and the plant growth tilt angle that can accurately determine the plant growth tilt. These phenotypic data of plant organs can accurately reflect some phenotypic traits of plant organs and determine the growth status of plant organs.

[0121] In some embodiments, after step 1094, the method further includes:

[0122] Step 1095: Using the mean point as the geometric center point of the plant organ, determine the three-dimensional coordinates of the geometric center point in the three-dimensional space corresponding to the three-dimensional laser, and use the three-dimensional coordinates as the basic positioning coordinates of the plant organ. .

[0123] Step 1096: The product of the minimum projected coordinate value and the principal direction vector of plant organ growth, plus the basic positioning coordinates, is used as the base position of the plant organ closest to the plant trunk. .

[0124] In practical implementation, the base location The corresponding formula is: .

[0125] Step 1097: The product of the maximum projected coordinate value and the principal direction vector of plant organ growth, plus the basic positioning coordinates, is used as the apex position of the plant organ. .

[0126] In practical implementation, the top position The corresponding formula is: .

[0127] Step 1098: Add the base position and the apex position to the phenotypic data output, or send the base position and the apex position to the agricultural machinery equipment so that the agricultural machinery equipment can grasp and harvest the plant organs according to the base position and the apex position.

[0128] The above method can accurately determine the base and apex positions of plant organs, allowing for the study of fruit setting based on these positions. Furthermore, agricultural machinery can accurately grasp plant organs based on these base and apex positions, facilitating the harvesting of plant organs.

[0129] In some embodiments, after step 1098, the method further includes:

[0130] Step 1099: Combine the basic positioning coordinates with the principal direction vector of plant organ growth to form plant organ pose data. The formula is: .

[0131] Step 10910: Determine the yaw angle and roll angle of the plant organ based on the plant organ pose data.

[0132] In practice, yaw angle The formula is: ,in, for The horizontal axis component, for The vertical axis component.

[0133] Roll angle The formula is: ,in, for Vertical coordinate components.

[0134] Step 10911: Combine the plant growth tilt angle with the yaw angle and the roll angle to form the spatial attitude parameters of the plant organ. .

[0135] Step 10912: Add the spatial attitude parameters to the phenotypic data for output, or send the spatial attitude parameters to the agricultural machinery equipment so that the agricultural machinery equipment can combine the spatial attitude parameters with the base position and the top position to determine the grasping path and grasping direction of the plant organ, and grasp and harvest the plant organ based on the grasping path and grasping direction.

[0136] Through the above scheme, the spatial attitude parameter can accurately characterize the orientation and posture of plant organs in space. After the accurate spatial attitude parameter corresponding to the plant organ is sent to the agricultural machinery, the agricultural machinery can accurately combine the spatial attitude parameter with the base position and apex position obtained above, and then plan the accurate grasping path and grasping direction of the plant organ. This makes the agricultural machinery more accurate in grasping and harvesting plant organs, improves the harvesting efficiency of plant organs, and the spatial attitude parameter also helps in the phenotypic study of plant organs.

[0137] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0138] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0139] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides a plant segmentation system based on point cloud projection and image segmentation fusion.

[0140] refer to Figure 2 The plant segmentation system based on point cloud projection and image segmentation fusion includes:

[0141] The acquisition module 201 is configured to use a camera and a 3D laser to acquire corresponding plant images and plant point cloud data based on the same field of view of the plant distribution area.

[0142] The point cloud projection processing module 202 is configured to determine the observation direction of the three-dimensional laser, project the plant point cloud data onto the observation direction, and obtain a projection map.

[0143] The homography matrix determination module 203 is configured to determine the optimal homography matrix based on the reprojection error between the projection map and the plant image;

[0144] The mapping processing module 204 is configured to map the projection map onto the plant image according to the optimal homography matrix to obtain a mapping map;

[0145] The organ segmentation processing module 205 is configured to segment the plant organs in the plant image to obtain the plant organ image segmentation result.

[0146] The organ segmentation result backfilling module 206 is configured to backfill the plant organ image segmentation result into the plant point cloud data based on the mapping map, so as to obtain the plant organ point cloud segmentation result;

[0147] The covariance processing module 207 is configured to determine the mean point in the point cloud segmentation result of the plant organ and construct a covariance matrix based on the mean point.

[0148] The organ growth main direction determination module 208 is configured to perform eigenvalue decomposition based on the covariance matrix to obtain multiple vectors representing organ growth directions, determine the maximum value among the multiple vectors representing organ growth directions, and use the maximum value as the plant organ growth main direction vector.

[0149] The phenotypic data determination module 209 is configured to combine the main growth direction vector of the plant organ with the point cloud segmentation result of the plant organ to determine the phenotypic data of the plant organ.

[0150] In some embodiments, the homography matrix determination module 203 is specifically configured as follows:

[0151] Determine the homography matrix;

[0152] A predetermined number of first feature point pairs are determined in the projection map, and the predetermined number of first feature point pairs are mapped and predicted based on the homography matrix to obtain a predetermined number of predicted feature point pairs.

[0153] Obtain a predetermined number of second feature point pairs in the plant image, corresponding to a predetermined number of first feature point pairs;

[0154] Determine the geometric error between a predetermined number of predicted feature point pairs and a predetermined number of second feature point pairs;

[0155] The geometric error is minimized, and the homography matrix is ​​adjusted to obtain the adjusted homography matrix. The adjusted homography matrix corresponding to the minimum geometric error is taken as the optimal homography matrix.

[0156] In some embodiments, the homography matrix determination module 203 is further configured to:

[0157] Determine two second feature points in the second feature point pair and two predicted feature points in the predicted feature point pair;

[0158] Based on the correspondence between the two second feature points and the two predicted feature points, the difference is calculated and then squared to obtain two squared results. The two squared results are then summed to obtain the summation result.

[0159] A predetermined number of predicted feature point pairs are summed with a predetermined number of second feature point pairs to obtain a predetermined number of summed results. The predetermined number of summed results are then accumulated to obtain the geometric error.

[0160] In some embodiments, the mapping processing module 204 is specifically configured to:

[0161] Determine the plant image pixel matrix corresponding to the plant image;

[0162] The optimal homography matrix is ​​multiplied by the pixel matrix of the plant image, and then transposed to obtain the mapping map.

[0163] In some embodiments, the phenotypic data determination module 209 is specifically configured as follows:

[0164] The plant organ point cloud segmentation result is projected according to the main growth direction vector of the plant organ to obtain the plant organ projection result of the plant organ point cloud segmentation result on the main growth direction vector of the plant organ.

[0165] Determine the maximum and minimum projection coordinate values ​​in the projection results of the plant organ, and take the difference between the maximum and minimum projection coordinate values ​​as the length of the plant organ;

[0166] Determine the angle between the main growth direction vector of the plant organ and the vertical direction, and use the angle as the plant growth tilt angle;

[0167] The main growth direction vector of the plant organ, the length of the plant organ, and the tilt angle of the plant growth are used as the phenotypic data of the plant organ.

[0168] In some embodiments, the phenotypic data determination module 209 is further configured to:

[0169] The mean point is used as the geometric center point of the plant organ, and the three-dimensional coordinates of the geometric center point in the three-dimensional space corresponding to the three-dimensional laser are determined. The three-dimensional coordinates are used as the basic positioning coordinates of the plant organ.

[0170] The product of the minimum projected coordinate value and the main growth direction vector of the plant organ, plus the basic positioning coordinates, is used as the base position of the plant organ closest to the plant trunk.

[0171] The product of the maximum projected coordinate value and the main growth direction vector of the plant organ, plus the basic positioning coordinates, is used as the top position of the plant organ.

[0172] The base position and the apex position can be added to the phenotypic data for output, or the base position and the apex position can be sent to agricultural machinery so that the agricultural machinery can grasp and harvest the plant organs according to the base position and the apex position.

[0173] In some embodiments, the phenotypic data determination module 209 is further configured to:

[0174] The basic positioning coordinates are combined with the main growth direction vector of the plant organ to form plant organ pose data;

[0175] The yaw angle and roll angle of the plant organ are determined based on the plant organ pose data;

[0176] The plant growth tilt angle is combined with the yaw angle and the roll angle to form the spatial attitude parameters of the plant organ.

[0177] The spatial attitude parameters are added to the phenotypic data for output, or the spatial attitude parameters are sent to agricultural machinery equipment so that the agricultural machinery equipment can combine the spatial attitude parameters with the base position and the top position to determine the grasping path and grasping direction of the plant organ, and grasp and harvest the plant organ based on the grasping path and grasping direction.

[0178] For ease of description, the above system is described by dividing it into various modules based on their functions. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0179] The system described in the above embodiments is used to implement the corresponding method in any of the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0180] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the methods described in any of the above embodiments.

[0181] Figure 3 This embodiment illustrates a more specific hardware structure of an electronic device. The device may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0182] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0183] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0184] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0185] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0186] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0187] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0188] The electronic devices described above are used to implement the corresponding methods in any of the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0189] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to perform the methods described in any of the above embodiments.

[0190] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0191] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to perform the methods described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0192] Based on the same concept, corresponding to any of the above embodiments, this application also provides a computer program product, including computer program instructions, which, when run on a computer, cause the computer to perform the method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0193] It is understood that before using the technical solutions of the various embodiments in this disclosure, users will be informed of the type, scope of use, and usage scenarios of the personal information involved in an appropriate manner, and user authorization will be obtained.

[0194] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose, based on the prompt message, whether to provide personal information to the software or hardware such as electronic devices, applications, servers, or storage media performing the operations of this disclosed technical solution.

[0195] As an optional but not limited implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0196] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0197] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application is limited to these examples; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in detail for the sake of brevity.

[0198] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0199] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0200] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the claims of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A plant segmentation method based on the fusion of point cloud projection and image segmentation, characterized in that, include: Using cameras and 3D lasers, plant images and point cloud data were collected from the same field of view of the plant distribution area. Determine the observation direction of the three-dimensional laser, and project the plant point cloud data onto the observation direction to obtain a projection map; The optimal homography matrix is ​​determined based on the reprojection error between the projection map and the plant image; Based on the optimal homography matrix, the projection map is mapped onto the plant image to obtain a mapping map; The plant organs in the plant image are segmented to obtain the plant organ image segmentation result; Based on the mapping map, the plant organ image segmentation results are backfilled into the plant point cloud data to obtain the plant organ point cloud segmentation results; Determine the mean point in the point cloud segmentation result of the plant organ, and construct a covariance matrix based on the mean point; Based on the covariance matrix, eigenvalue decomposition is performed to obtain multiple vectors representing the growth direction of organs. The maximum value among the multiple vectors representing the growth direction of organs is determined, and the maximum value is taken as the main growth direction vector of plant organs. The phenotypic data of the plant organs are determined by combining the main growth direction vector of the plant organs with the point cloud segmentation results of the plant organs.

2. The method according to claim 1, characterized in that, Determining the optimal homography matrix based on the reprojection error between the projection image and the plant image includes: Determine the homography matrix; A predetermined number of first feature point pairs are determined in the projection map, and the predetermined number of first feature point pairs are mapped and predicted based on the homography matrix to obtain a predetermined number of predicted feature point pairs. Obtain a predetermined number of second feature point pairs in the plant image, corresponding to a predetermined number of first feature point pairs; Determine the geometric error between a predetermined number of predicted feature point pairs and a predetermined number of second feature point pairs; The geometric error is minimized, and the homography matrix is ​​adjusted to obtain the adjusted homography matrix. The adjusted homography matrix corresponding to the minimum geometric error is taken as the optimal homography matrix.

3. The method according to claim 2, characterized in that, The geometric error between the predetermined number of predicted feature point pairs and the predetermined number of second feature point pairs includes: Determine two second feature points in the second feature point pair and two predicted feature points in the predicted feature point pair; Based on the correspondence between the two second feature points and the two predicted feature points, the difference is calculated and then squared to obtain two squared results. The two squared results are then summed to obtain the summation result. A predetermined number of predicted feature point pairs are summed with a predetermined number of second feature point pairs to obtain a predetermined number of summed results. The predetermined number of summed results are then accumulated to obtain the geometric error.

4. The method according to claim 2, characterized in that, The step of mapping the projection map onto the plant image based on the optimal homography matrix to obtain the mapping map includes: Determine the plant image pixel matrix corresponding to the plant image; The optimal homography matrix is ​​multiplied by the pixel matrix of the plant image, and then transposed to obtain the mapping map.

5. The method according to claim 1, characterized in that, The step of combining the principal growth direction vector of the plant organ with the point cloud segmentation result of the plant organ to determine the phenotypic data of the plant organ includes: The plant organ point cloud segmentation result is projected according to the main growth direction vector of the plant organ to obtain the plant organ projection result of the plant organ point cloud segmentation result on the main growth direction vector of the plant organ. Determine the maximum and minimum projection coordinate values ​​in the projection results of the plant organ, and take the difference between the maximum and minimum projection coordinate values ​​as the length of the plant organ; Determine the angle between the main growth direction vector of the plant organ and the vertical direction, and use the angle as the plant growth tilt angle; The main growth direction vector of the plant organ, the length of the plant organ, and the tilt angle of the plant growth are used as the phenotypic data of the plant organ.

6. The method according to claim 5, characterized in that, After using the principal growth direction vector of the plant organ, the length of the plant organ, and the plant growth tilt angle as phenotypic data of the plant organ, the method further includes: The mean point is used as the geometric center point of the plant organ, and the three-dimensional coordinates of the geometric center point in the three-dimensional space corresponding to the three-dimensional laser are determined. The three-dimensional coordinates are used as the basic positioning coordinates of the plant organ. The product of the minimum projected coordinate value and the main growth direction vector of the plant organ, plus the basic positioning coordinates, is used as the base position of the plant organ closest to the plant trunk. The product of the maximum projected coordinate value and the main growth direction vector of the plant organ, plus the basic positioning coordinates, is used as the top position of the plant organ. The base position and the apex position can be added to the phenotypic data for output, or the base position and the apex position can be sent to agricultural machinery so that the agricultural machinery can grasp and harvest the plant organs according to the base position and the apex position.

7. The method according to claim 6, characterized in that, After adding the base position and the apex position to the phenotypic data output, or sending the base position and the apex position to agricultural machinery, the method further includes: The basic positioning coordinates are combined with the main growth direction vector of the plant organ to form plant organ pose data; The yaw angle and roll angle of the plant organ are determined based on the plant organ pose data; The plant growth tilt angle is combined with the yaw angle and the roll angle to form the spatial attitude parameters of the plant organ. The spatial attitude parameters are added to the phenotypic data for output, or the spatial attitude parameters are sent to agricultural machinery equipment so that the agricultural machinery equipment can combine the spatial attitude parameters with the base position and the top position to determine the grasping path and grasping direction of the plant organ, and grasp and harvest the plant organ based on the grasping path and grasping direction.

8. A plant segmentation system based on the fusion of point cloud projection and image segmentation, characterized in that, include: The acquisition module is configured to use a camera and a 3D laser to acquire corresponding plant images and plant point cloud data based on the same field of view of the plant distribution area. The point cloud projection processing module is configured to determine the observation direction of the three-dimensional laser, project the plant point cloud data onto the observation direction, and obtain a projection map. The homography matrix determination module is configured to determine the optimal homography matrix based on the reprojection error between the projection image and the plant image; The mapping processing module is configured to map the projection map onto the plant image according to the optimal homography matrix to obtain a mapping map; The organ segmentation processing module is configured to segment plant organs in the plant image to obtain plant organ image segmentation results. The organ segmentation result backfilling module is configured to backfill the plant organ image segmentation result into the plant point cloud data based on the mapping map, so as to obtain the plant organ point cloud segmentation result; The covariance processing module is configured to determine the mean point in the point cloud segmentation result of the plant organ and construct a covariance matrix based on the mean point. The organ growth main direction determination module is configured to perform eigenvalue decomposition based on the covariance matrix to obtain multiple vectors representing organ growth directions, determine the maximum value among the multiple vectors representing organ growth directions, and use the maximum value as the plant organ growth main direction vector. The phenotypic data determination module is configured to combine the main growth direction vector of the plant organ with the point cloud segmentation result of the plant organ to determine the phenotypic data of the plant organ.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 7.

Images