Automated plant phenotype collection method, device and electronic equipment

Abstract

The present invention provides an automated plant phenotype collection method, device and electronic equipment, which relates to the field of non-contact plant collection, and includes the following steps: S100, establishing an obstacle model; S200, generating a plant fruit recognition model; S300, determining the position information of the fruit and the camera shooting position; S400, enabling the mobile robot to arrive at the camera shooting position without collision according to the path; S500, obtaining the point cloud information of the fruit; S600, reconstructing a three-dimensional model representing the fruit phenotype. The present invention uses laser radar, multiple cameras and mechanical arms to fuse and transform multiple perspectives, trains sweet peppers and realizes sweet pepper recognition through the yolov5 scheme, and realizes path planning through obstacle avoidance of the RRT algorithm, fully automating the non-contact crop phenotype lossless collection, point cloud construction, and automated mechanical operation, thereby improving collection efficiency.

Description

Technical Field

[0001] The present invention relates to the field of non-contact plant collection, and in particular to an automated plant phenotype collection method, device and electronic equipment. Background Art

[0002] The mechanized collection of phenotypic information encountered some difficulties and challenges during the research and development process. First, because collision-free contact collection was not considered, crops were easily damaged during the collection operation, which brought difficulties to the accuracy and reliability of the data. Damage to crops may lead to distortion of phenotypic data and affect researchers' accurate understanding of the growth and development process of crops.

[0003] Another problem is the single-camera, single-viewing acquisition method. The limitation of this method is that the viewing angle is limited and it is impossible to fully observe and capture the phenotypic information of crops. The limited viewing angle leads to incomplete data, which may miss important details and features, limiting the comprehensive analysis of the growth and development process of crops.

[0004] In addition, the obstruction of leaves during crop growth also poses a problem for phenotypic collection. Due to the obstruction of leaves, some crop phenotypic characteristics cannot be accurately recorded and measured, resulting in a loss of information. This limits researchers' comprehensive understanding of important characteristics during crop growth and affects the accuracy of crop trait evaluation and variety improvement.

[0005] These difficulties and challenges indicate that current phenotypic collection methods still face room for improvement. Solving these problems requires innovative technologies and methods to improve the efficiency and accuracy of data collection and ensure comprehensive observation and measurement of crop phenotypes. This will help to conduct more in-depth research on the growth and development process of crops, improve the accuracy of quality assessment and genetic breeding, and bring greater impetus to the development of agricultural science and the implementation of modern agriculture. Summary of the invention

[0006] The technical problem to be solved by the present invention is that the mechanized collection of plant phenotypic information does not take into account collision-free contact collection, and crops are damaged during the collection operation; single camera, single-angle collection, limited viewing angle, incomplete crop phenotypic information, and missing information; leaves block the crop process, affecting phenotypic collection. The purpose is to provide an automated plant phenotypic collection method, device and electronic equipment to solve the above problems.

[0007] An automated plant phenotype collection method comprises the following steps:

[0008] S100, scanning plant information through laser radar, taking the plants as obstacles, and establishing an obstacle model;

[0009] S200, generating a plant fruit recognition model according to the obstacle model, the plant information, and the first model;

[0010] S300, determining the position information of the fruit and the camera shooting position according to the plant fruit recognition model;

[0011] S400, planning a moving path according to the position information of the fruit, the camera shooting position and the second model, and combining with the first camera to enable the mobile robot to arrive at the camera shooting position without collision along the path;

[0012] S500, photographing the fruit of the plant by a second camera to obtain point cloud information of the fruit;

[0013] S600, reconstructing a three-dimensional model representing the phenotype of the fruit by fusing point cloud information of a plurality of the fruits according to a third model.

[0014] Furthermore, the laser radar is arranged on the mobile robot.

[0015] Furthermore, the second camera is at least one depth camera, and the at least one depth camera is arranged on the mobile robot.

[0016] Further, wherein the second camera includes at least one structured light camera, step 500 includes:

[0017] Step 501, setting the at least one structured light camera around the fruit;

[0018] Step 502, projecting a structured light source onto the surface of the fruit;

[0019] Step 503, capturing a reflected image: the at least one structured light camera captures the image reflected by the light source and records the phenotypic information of the fruit.

[0020] Furthermore, the first model is a YOLOv5 model.

[0021] Furthermore, the second model is an RRT model.

[0022] Furthermore, the third model is an ICP model.

[0023] Furthermore, steps 100 to 600 involve conversion of a lidar coordinate system, a structured light camera coordinate system, a robotic arm coordinate system, and a depth camera coordinate system, and conversion between coordinate systems is performed using at least one of external calibration, hand-eye calibration, a geometric constraint-based method, or a visual odometer.

[0024] Another embodiment of the present invention provides an automated plant phenotype collection device, comprising:

[0025] A mobile robot, comprising:

[0026] LiDAR, used to collect map information and convert point clouds into octree system maps;

[0027] At least one depth camera, used to identify plant fruits in real time and obtain depth information;

[0028] The robotic arm, combined with the RRT algorithm, is used to avoid obstacles and achieve contactless fruit collection;

[0029] At least one structured light camera module, used for collecting fruit phenotypic information;

[0030] The controller includes realizing point cloud fusion registration and 3D reconstruction of fruits through ICP algorithm, and reducing noise in point cloud through filtering algorithm.

[0031] Another aspect of the present invention provides an electronic device, including:

[0032] A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the above method is implemented when the processor executes the program.

[0033] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0034] 1. The present invention provides an automated plant phenotype collection method, which can fully automatically solve the non-contact and lossless collection of crop phenotypes, point cloud construction, and automated mechanical operations, thereby improving collection efficiency;

[0035] 2. The present invention discloses an automated plant phenotype collection device, which integrates and transforms multiple perspectives by using laser radar, multiple cameras and mechanical arms, trains bell peppers and realizes bell pepper recognition through the yolov5 program, and realizes path planning through obstacle avoidance using the RRT algorithm, fully automates the non-contact collection of crop phenotypes, point cloud construction, and automated mechanical operations, thereby improving collection efficiency;

[0036] 3. The electronic device of the present invention can fully automatically solve the non-contact and lossless collection of crop phenotypes, point cloud construction, and automated mechanical operations, thereby improving collection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings are used to better understand the present invention and do not constitute a limitation of the present invention.

[0038] Figure 1 is a schematic diagram of a flow chart according to an embodiment of the present invention;

[0039] Figure 2 is another flow chart according to an embodiment of the present invention;

[0040] Figure 3 is a schematic diagram of coordinate system conversion according to an embodiment of the present invention;

[0041] Figure 4 is a schematic diagram of an automated plant phenotype collection device according to an embodiment of the present invention;

[0042] Figure 5 is a schematic diagram of an electronic device according to an embodiment of the present invention.

[0043] Reference numerals:

[0044] 700-automated plant phenotype collection device, 701-mobile robot, 7011-laser radar, 7012-depth camera, 7013-robotic arm, 7014-structured light camera, 702-controller, 800-electronic device, 801-processor, 802-memory. DETAILED DESCRIPTION

[0045] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0046] like Figure 1 As shown, the present invention provides an automated plant phenotype collection method, comprising the following steps:

[0047] S100, scanning plant information through the laser radar 7011, taking the plant as an obstacle, and establishing an obstacle model;

[0048] S200, generating a plant fruit recognition model according to the obstacle model, the plant information, and the first model;

[0049] S300, determining the position information of the fruit and the camera shooting position according to the plant fruit recognition model;

[0050] S400, planning a moving path according to the position information of the fruit, the camera shooting position and the second model, and combining with the first camera to enable the mobile robot 701 to arrive at the camera shooting position without collision along the path;

[0051] S500, photographing the fruit of the plant by a second camera to obtain point cloud information of the fruit;

[0052] S600, reconstructing a three-dimensional model representing the phenotype of the fruit by fusing point cloud information of a plurality of the fruits according to a third model.

[0053] In order to solve the problems existing in the prior art, the applicant has carefully analyzed different problems and formed a complete set of technical solutions to these problems to comprehensively solve the technical problems.

[0054] For mechanized collection of plant phenotyping information: In order to reduce crop damage, a series of mechanized tools and equipment, such as robots or automated systems, are used to achieve collision-free contact collection. These devices avoid direct contact with crops through precise control and positioning, thereby reducing the risk of crop damage.

[0055] In order to overcome the limitations of single-camera, single-viewing angle acquisition, a multi-camera system can solve the problem and simultaneously shoot crops from different angles to obtain more comprehensive phenotypic information. These cameras can be placed at different positions and angles to cover all sides and angles of the crop, thus overcoming the problems of limited viewing angle and incomplete phenotypic information.

[0056] In order to solve the impact of leaf occlusion during cropping, some technical means are used. For example, by optimizing the camera position and angle, adjusting the position and intensity of the light source, and using special image processing algorithms, the impact of leaf occlusion on phenotypic collection can be minimized or eliminated. In addition, sensor technology and machine learning algorithms can be combined to perform real-time leaf detection and segmentation to accurately measure and record the obscured parts.

[0057] The application of these technical solutions makes phenotypic collection more accurate and comprehensive. Through mechanized collection, multi-camera multi-view collection and the handling of leaf occlusion problems, the accuracy and completeness of the data can be improved, providing a more reliable data basis for the study of crop growth and development processes. In addition, combined with the progress in image processing, sensor technology and machine learning, the efficiency and automation of phenotypic collection can be further improved, providing stronger support for agricultural science and crop improvement.

[0058] Existing non-contact solutions still have difficulty meeting existing technical problems: For example, near-infrared spectroscopy can be used to non-destructively detect the texture of bell pepper fruits, including internal quality information such as sugar content. The advantage of this method is that it can perform qualitative and quantitative analysis of the fruit, but the accuracy needs to be improved. In addition, by analyzing the bell pepper point cloud, the posture of the bell pepper can be estimated, regardless of whether the stem is visible.

[0059] In this technical solution, the basic principles and characteristics of near-infrared spectroscopy technology: Near-infrared light refers to electromagnetic waves with a wavelength range of 780-2526nm, which are usually divided into near-infrared short-wave and long-wave. Near-infrared methods for detecting fruit sugar include near-infrared reflection detection, near-infrared transmission detection, and near-infrared diffuse reflection detection. The near-infrared reflection detection method is more suitable for experimental research on detecting surface information of the peel. The near-infrared transmission method basically reflects the internal quality information of the fruit, but it requires specific conditions. Bell pepper posture estimation: By comparing the analysis of the bell pepper point cloud and the experimental results of the simulated point cloud, the error based on the bell pepper point cloud is greater than the average value based on the simulated data. Despite the error, the proposed symmetry axis detection method can still estimate the posture of the bell pepper, regardless of whether the pepper stalk is visible.

[0060] Nondestructive detection of sugar content: An intelligent near-infrared diffuse reflectance spectroscopy (INIS) method based on near-infrared spectroscopy analysis technology was designed for nondestructive detection of sugar content in fruits and vegetables. In the experimental part, cherry tomatoes were selected as representative samples, and a BP network model was established to study the sugar content. Spectral features were extracted from the spectral data of the experiment, using smoothing and principal component analysis (PCA). An intelligent near-infrared diffuse reflectance spectroscopy scheme (INIS) was proposed for nondestructive detection and prediction of sugar content in representative fruits. BP neural network technology has broad application prospects in near-infrared nondestructive testing, and the prediction results are more accurate. In fruit testing, fruits can be qualitatively and quantitatively analyzed, including evaluation of fruit type, regional classification, fruit sugar content, etc. However, there is still room for improvement in the accuracy of the results, and there are many fruits to be studied. Therefore, this paper combines near-infrared spectroscopy and BP neural network to study the sugar content of cherry tomatoes to improve the accuracy of the results.

[0061] The difficulty still lies in the model construction of multi-view fusion, multi-coordinate system transformation and non-contact collection of plant phenotypes. The present invention takes the phenotype of sweet pepper fruit as an example, but it should not be limited to this.

[0062] The main methods of operation are as follows Figure 1 and Figure 2 These methods will be described in detail below.

[0063] Among them, the bell pepper fruit phenotypic information may include the following:

[0064] 1. Shape characteristics: including geometric shape parameters such as fruit size, aspect ratio, height, volume, etc., which are used to describe the overall shape and size characteristics of the fruit.

[0065] 2. Surface texture: describes the texture, gloss, and other characteristics of the fruit's outer surface, such as smooth, wrinkled, bumpy, etc., which is used to evaluate the maturity and quality of the fruit.

[0066] 3. Color characteristics: Describe the color attributes of the fruit, such as red, yellow, green, etc., which are used to distinguish fruits of different varieties or maturity.

[0067] 4. Defect detection: Detect defects on the surface of the fruit, such as lesions, scars, rot, etc., to assess the health and quality of the fruit.

[0068] 5. Uniformity: Evaluate the uniformity of the fruit's interior, such as the density and uniformity of the flesh, to determine the internal quality of the fruit.

[0069] 6. Fruit distribution: describes the distribution of fruits on the plant, such as the position and density of the fruits, which is used to analyze the spatial distribution patterns of fruit growth and development.

[0070] 7. Growth rate: Record the growth rate and development process of the fruit, such as the fruit growth curve, growth period, etc., to study the development laws and growth mechanisms of the fruit.

[0071] 8. Other characteristics: Based on research needs, other characteristics such as fruit weight, sugar content, aroma, and nutritional components can also be included to evaluate the nutritional value and edible quality of the fruit.

[0072] By collecting and analyzing the phenotypic information of bell pepper fruits, we can understand the morphological characteristics, quality status, growth and development of the fruits, as well as the correlation with factors such as environment, variety, and cultivation management. This information is of great reference value for fruit quality evaluation, variety improvement, agricultural production management, and scientific research.

[0073] In some embodiments, the laser radar 7011 is set on the mobile robot 701.

[0074] In this embodiment, step 100 may be the establishment of an obstacle model: the laser radar 7011 collects map information and converts the point cloud into an octree system map, treating the bell pepper plants as obstacles.

[0075] The process of laser radar 7011 collecting map information is as follows:

[0076] Step 101, laser radar 7011 scanning: The laser radar 7011 device scans the environment by emitting a laser beam and measuring its return time. It records the coordinates (X, Y, Z) and reflection intensity of each point.

[0077] Step 102, point cloud data processing: converting the raw data collected by the laser radar 7011 into a point cloud data set. This data contains the three-dimensional coordinate information of a large number of discrete points.

[0078] Step 103, point cloud filtering: filtering the point cloud data to remove noise and invalid data points. Common filtering methods include voxel filtering, statistical filtering, normal filtering, etc.

[0079] Step 104, point cloud segmentation: segment the point cloud data into different regions or objects. The segmentation method can be based on features such as color, shape, density, etc. Common algorithms include Euclidean clustering, normal-based segmentation, etc.

[0080] Step 105, octree map construction: convert the point cloud data into an octree system map. The octree is a hierarchical data structure that can divide the three-dimensional space into different voxel units. Each voxel unit can represent an area and record the point cloud information in the area.

[0081] Step 106, octree map optimization: Optimize the octree map to improve the accuracy and efficiency of the map. The optimization methods include voxel merging, voxel compression, data indexing, etc.

[0082] Through the above steps, the point cloud data collected by the LiDAR 7011 can be converted into an octree system map, which can be used for applications such as environmental perception, navigation planning, and obstacle detection.

[0083] The second camera is at least one depth camera 7012 , and the at least one depth camera 7012 is arranged on the mobile robot 701 .

[0084] In some embodiments, a Kinect depth camera 7012 is used as a hardware acquisition device for the bell pepper recognition model. This is a depth perception device developed by Microsoft based on the principle of structured light. It includes an RGB color camera and an infrared depth sensor, which can simultaneously obtain the color image and depth information of the scene through structured light projection and infrared imaging. Its working principle is: Structured light projection: The Kinect depth camera 7012 will project a structured light pattern with a specific texture through an infrared laser, and these textures will be projected onto the surface of objects in the scene. Infrared imaging: The infrared sensor in the camera will receive the reflected structured light and record their position and shape in the image. Depth calculation: By analyzing the deformation of the structured light on the surface of the object, the camera can calculate the distance between each pixel and the camera, thereby obtaining a depth map of the scene.

[0085] Step 300 may be to determine the target point: using the established bell pepper recognition model, using the depth camera 7012 to recognize bell peppers in real time, and outputting the bell pepper depth information, with the upper left vertex of the recognition frame as the initial target point, the upper right vertex as the second target point, the lower right vertex as the third target point, the lower left vertex as the fourth target point, and the center point of the recognition frame as the last fixed point. The robot arm 7013 may move to these target points for detection.

[0086] Get depth image: The depth image collected by Kinect depth camera 7012 contains the depth information of objects in the scene. These depth values ​​represent the distance between each pixel and the camera.

[0087] Target box selection: In the depth image, the target detection algorithm can identify the location of the bell pepper and select its bounding box. The upper left vertex of the bounding box is used as the initial target point, the upper right vertex is used as the second target point, the lower right vertex is used as the third target point, and the lower left vertex is used as the fourth target point, thus forming a box selection area consisting of 4 fixed points surrounding the bell pepper.

[0088] Fixed point determination: Within the bell pepper’s bounding box, the center point of the box can be calculated and used as the last fixed point. The position of the point in three-dimensional space can be determined by converting the pixel coordinates in the depth image into spatial coordinates.

[0089] Depth information analysis: Using the depth values ​​of the pixels within the bell pepper bounding box in the depth image, the distance of each key point relative to the camera can be calculated.

[0090] Target point positioning: By combining the depth information and the pixel coordinates of the target point, the accurate position of the bell pepper in three-dimensional space can be obtained, including the four vertices and center point of its bounding box.

[0091] Wherein the second camera includes at least one structured light camera 7014, the step 500 includes:

[0092] Step 501, setting the at least one structured light camera 7014 around the fruit;

[0093] Step 502, projecting a structured light source onto the surface of the fruit;

[0094] Step 503, capturing a reflected image: the at least one structured light camera 7014 captures an image reflected by a light source, and records the phenotypic information of the fruit.

[0095] In some implementations, step 500 may include the following steps:

[0096] Step 501, setting multiple structured light cameras 7014: setting multiple structured light cameras 7014 around the fruit to cover multiple viewing angles and directions.

[0097] Step 502, projecting a structured light source: each camera projects a structured light source (such as stripes or grid points) onto the surface of the fruit.

[0098] Step 503, capturing reflected image: the camera captures the image reflected by the light source, and records the texture and shape information of the fruit surface.

[0099] Step 504, data fusion: fusing images from multiple perspectives and orientations to obtain more complete and accurate fruit phenotypic information.

[0100] Step 505, data processing and analysis: Process and analyze the collected image data, extract the characteristic parameters and morphological information of the fruit, and form point cloud information from the perspective of the structured light camera 7014.

[0101] The first model is the YOLOv5 model.

[0102] Furthermore, in some embodiments, step 200 implements a bell pepper recognition function: collecting bell pepper fruit pictures to create a bell pepper fruit dataset, using a YOLOV5 algorithm to train the bell pepper fruit dataset, and establishing a bell pepper recognition model.

[0103] Among them, YOLOv5 is a target detection algorithm. It is the latest version of the YOLO (You Only Look Once) series of algorithms. The main features and working principles of this algorithm are: Single-stage detection: YOLOv5 is a single-stage target detection algorithm that performs target detection directly on the image through a single neural network model without the need for a complex candidate region generation process. Anchor-based target detection: YOLOv5 uses a series of prior boxes (Anchors) to predict the location and category of the target. Each prior box is responsible for detecting targets of different sizes and aspect ratios. Feature pyramid structure: YOLOv5 uses a feature pyramid structure to detect targets of different sizes through multiple feature maps of different scales, thereby improving the accuracy of detection and multi-scale adaptability. Multi-level detection head: YOLOv5 uses a multi-level detection head to predict the bounding box, category and confidence of targets of different scales to enhance detection performance. Training strategy: YOLOv5 uses an adaptive training strategy that automatically adjusts the weight of the loss function according to the size and difficulty of the target to achieve better target detection results. YOLOv5 has achieved good performance in object detection tasks, achieving a balance between speed and accuracy.

[0104] In some embodiments, the second model is a RRT model.

[0105] Step 400 may be path planning: using the RRT algorithm, a path including all target points is planned from the initial position. During the planning process, collision detection is performed to determine whether there is a collision, and if there is a collision, re-planning is performed.

[0106] The RRT algorithm, full name of which is Rapidly-exploring Random Tree, is a path planning algorithm used to plan the motion path of a machine or a robotic arm 7013 in a complex environment to avoid obstacles and optimize the motion trajectory.

[0107] In some embodiments, the specific steps are as follows:

[0108] Step 401, initial position and target point setting: according to the initial position and target point in the bell pepper depth information, the starting point and target point of the RRT algorithm are set.

[0109] Step 402, path planning: using the RRT algorithm, random sampling is performed around the starting point, and the path is gradually generated by expanding the tree. During each expansion, the current node is connected to the sampling point, and it is determined whether it collides with an obstacle.

[0110] Step 403, collision detection and re-planning: During the path planning process, collision detection is performed on each extended path. If it is found that the path collides with an obstacle, the path needs to be re-planned. This can be achieved by adjusting the sampling points, the expansion direction of the nodes, or applying other collision avoidance strategies.

[0111] Step 404, until the target point is reached: continue to expand the tree and detect collisions of the path until the generated path connects the starting position and the target point without collision.

[0112] In other embodiments, the third model is an ICP model.

[0113] Specifically, step 600 may be a three-dimensional reconstruction: the ICP algorithm is used to fuse and register the point clouds of multiple cameras and multiple perspectives to reconstruct the three-dimensional model of the fruit point cloud. A filtering algorithm is used to reduce the noise in the point cloud. The ICP algorithm, which is called the Iterative Closest Point algorithm, is used to register the data collected by different perspectives or cameras (i.e., align the data from different perspectives) and generate a more accurate overall model.

[0114] In some embodiments, the specific steps are as follows:

[0115] Step 601, by extracting feature points from images taken from different perspectives, we can obtain point cloud data for each perspective. These point cloud data may contain some noise and incomplete information. In order to reduce the impact of noise, we can apply a filtering algorithm (such as Gaussian filtering or statistical filtering) to smooth the point cloud data and remove outliers.

[0116] Step 602, use the ICP algorithm to register these filtered point cloud data. The ICP algorithm aligns point clouds from different perspectives to a target point cloud in a reference coordinate system through iterative optimization. In each iteration, the algorithm aligns the target point cloud with the source point cloud by finding the best rigid transformation (such as translation and rotation). This process will be repeated until the convergence condition is reached, and the optimal registration result is obtained.

[0117] Step 603, after ICP algorithm registration, we can fuse the point cloud data from multiple perspectives together to form a complete fruit point cloud 3D model. This 3D model can provide detailed fruit morphological information, including surface geometry, texture information, etc. By analyzing and measuring the model, we can obtain important parameters such as the size, volume, and shape of the fruit, and further study the characteristics and development laws of the fruit.

[0118] In this way, by using the ICP algorithm for point cloud registration and the filtering algorithm for noise reduction, multi-camera and multi-view point cloud fusion and reconstruction can be achieved to obtain a high-quality fruit point cloud three-dimensional model.

[0119] In one embodiment, steps 100 to 600 involve conversion of the laser radar 7011 coordinate system, the structured light camera 7014 coordinate system, the robotic arm 7013 coordinate system, and the depth camera 7012 coordinate system, and conversion between coordinate systems is performed using at least one of external calibration, hand-eye calibration, a geometric constraint-based method, or a visual odometer.

[0120] In steps 100 to 600, if Figure 3 As shown, the problem involves the conversion of multiple coordinate systems, including the conversion of the laser radar 7011 coordinate system, the structured light camera 7014 coordinate system, the robot arm 7013 coordinate system, and the depth camera 7012 coordinate system. Therefore, one or more of the following solutions need to be used to solve the problem:

[0121] Method 1: External calibration: By placing calibration objects with known positions (such as a chessboard or a spherical calibration plate) in the actual scene and measuring their positional relationship in different coordinate systems, external calibration can be performed. In this way, the relative relationship between the coordinate systems can be determined, so as to perform conversion between coordinate systems.

[0122] Method 2: Hand-eye calibration: When it comes to the coordinate system conversion between the robot arm 7013 and the camera, the relative position and posture relationship between the robot arm 7013 and the camera can be determined by the hand-eye calibration method. This requires collecting the position and posture of the end effector of the robot arm 7013 and the position and posture of the calibration object captured by the camera, and determining the relationship between the two through calculation.

[0123] Method 3: Method based on geometric constraints: If the geometric relationship between certain coordinate systems is known, such as the end effector of the robot 7013 is fixed above the camera, the coordinate system transformation can be performed through the method of geometric constraints. By measuring the fixed relationship between the two coordinate systems, such as relative position, rotation matrix, etc., the conversion between the coordinate systems can be performed.

[0124] Method 4: Visual odometer: Use sensors such as cameras or LiDAR 7011 to obtain features in the environment, and infer the position and posture of the robot or device in different coordinate systems by calculating the relative changes in motion. This method is suitable for coordinate system transformation in real-time or dynamic environments.

[0125] In an exemplary technical solution, in some embodiments, the present invention combines a multi-perspective greenhouse sweet pepper fruit phenotype collision-free path acquisition and reconstruction method with machine vision, adopts multi-perspective, multi-camera fusion combined with the robotic arm 7013 using the RRT algorithm to avoid obstacles, thereby achieving the purpose of contactless collection of fruit phenotypes. At the same time, the ICP algorithm is used to align the collected data and reconstruct three-dimensionally.

[0126] Among them, multi-view and multi-camera fusion refers to using multiple viewpoints and cameras to observe the target object at the same time, and obtaining more comprehensive and accurate information by fusing the data of these viewpoints and cameras.

[0127] In another aspect of the present invention, an automated plant phenotype collection device 700 is provided, comprising:

[0128] A mobile robot 701, wherein the mobile robot 701 comprises:

[0129] LiDAR 7011, used to collect map information and convert point clouds into octree system maps;

[0130] At least one depth camera 7012, used to identify plant fruits in real time and obtain depth information;

[0131] The robot arm 7013, with the RRT algorithm, is used to avoid obstacles and achieve contactless fruit collection;

[0132] At least one structured light camera 7014 module, used to collect fruit phenotypic information;

[0133] The controller 702 includes an ICP algorithm to realize point cloud fusion registration and three-dimensional reconstruction of the fruit, and a filtering algorithm to reduce noise in the point cloud.

[0134] In some embodiments, Figure 4 and Figure 5 As shown, the controller 702 may include a processor 801 and a memory 802, wherein the memory 802 stores computer program code, and when the computer program code is called by the processor 801, it can realize point cloud fusion registration and three-dimensional reconstruction of the fruit through the ICP algorithm, and reduce the noise in the point cloud through the filtering algorithm.

[0135] The inventive point of this embodiment has been fully explained in the above embodiments for the technical solution of automated plant phenotype collection, which will not be repeated here.

[0136] In yet another aspect of the present invention, an electronic device 800 is provided, comprising:

[0137] The memory 802, the processor 801 and the computer program stored in the memory 802 and executable on the processor 801, when the processor 801 executes the program, the automated plant phenotype collection method as described above is implemented.

[0138] Specifically, the electronic device 800 can be used to implement an automated plant phenotype collection method.

[0139] It should be noted that if Figure 5 As shown, the electronic device 800 in this example is in the form of a general computing device. The components of the electronic device 800 may include but are not limited to: one or more processors 801 or processing units, a system memory 802, and a bus connecting different system components (including the system memory 802 and the processing unit).

[0140] The electronic device 800 includes a computing unit, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory, read-only memory 802) or a computer program loaded from a storage unit to a RAM (Random Access Memory, random access / access memory 802). In the RAM, various programs and data required for the operation of the device can also be stored. The computing unit, ROM and RAM are connected to each other through a bus. The I / O (Input / Output, input / output) interface is also connected to the bus.

[0141] Multiple components in the electronic device 800 are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0142] The electronic device 800 used to implement the example of the embodiment of the present invention is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 800 can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown in the present invention, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present invention described and / or required in the present invention.

[0143] The processor 801 is a computing unit, which may be various general and / or special processing components with processing and computing capabilities. Some examples of the processor 801 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor, Digital Signal Processor 801), and any appropriate processor 801, controller 702, microcontroller 702, etc. The computing unit performs the various methods and processes described above, such as an imaging method based on focal plane fitting. For example, in some embodiments, the imaging method based on focal plane fitting may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the computing unit, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the processor 801 may be configured to execute the aforementioned method in any other appropriate manner (for example, by means of firmware).

[0144] In the context of the present invention, memory 802 may be a machine-readable medium, which may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may also be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a RAM, a ROM, an EPROM (Electrically Programmable Read-Only-Memory, Erasable Programmable Read-Only Memory 802) or a flash memory 802, an optical fiber, a CD-ROM (CompactDisc Read-Only Memory, Portable Compact Disk Read-Only Memory 802), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0145] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to the processor 801 or controller 702 of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor 801 or controller 702, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0146] In view of the above, the present invention has the following advantages and beneficial effects compared with the prior art:

[0147] 1. The present invention provides an automated plant phenotype collection method, which uses laser radar, multiple cameras and robotic arms to fuse and transform multiple perspectives, trains bell peppers and realizes bell pepper recognition through the yolov5 program, and realizes path planning through obstacle avoidance using the RRT algorithm, fully automates the non-contact collection of crop phenotypes, point cloud construction, and automated mechanical operations, thereby improving collection efficiency;

[0148] 2. The present invention provides an automated plant phenotype collection device, which integrates and transforms multiple perspectives by using laser radar, multiple cameras and robotic arms, trains bell peppers and realizes bell pepper recognition through the yolov5 program, and realizes path planning through obstacle avoidance using the RRT algorithm, fully automates the non-contact collection of crop phenotypes, point cloud construction, and automated mechanical operations, thereby improving collection efficiency;

[0149] 3. An electronic device provided by the present invention integrates and transforms multiple perspectives by applying laser radar, multiple cameras and a robotic arm, trains and recognizes sweet peppers through the yolov5 scheme, and realizes path planning through obstacle avoidance using the RRT algorithm, thereby fully automatically solving the non-contact lossless collection of crop phenotypes, point cloud construction, and automated mechanical operations, thereby improving collection efficiency.

[0150] In the several embodiments provided by the present invention, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0151] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0152] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

Claims

1. An automated plant phenotype collection method, characterized in that: The following steps are involved: S100, scanning plant information through laser radar (7011), taking the plant as an obstacle, and establishing an obstacle model; S200, generating a plant fruit recognition model according to the obstacle model, the plant information, and the first model, wherein the first model is a YOLOv5 model; S300, determining the position information of the fruit and the camera shooting position according to the plant fruit recognition model; S400, planning a moving path according to the position information of the fruit, the camera shooting position and the second model, and combining with the first camera to enable the mobile robot (701) to arrive at the camera shooting position without collision according to the path, wherein the second model is an RRT model; S500, photographing the fruit of the plant by a second camera to obtain point cloud information of the fruit; Wherein the second camera includes at least one structured light camera (7014), the step S500 includes: Step 501, setting the at least one structured light camera around the fruit (7014); Step 502, projecting a structured light source onto the surface of the fruit; Step 503, capturing a reflected image: the at least one structured light camera (7014) captures an image reflected by a light source and records the phenotypic information of the fruit; Step 504, data fusion: fusing images from multiple viewing angles and orientations to obtain and update the phenotypic information; Step 505, data processing and analysis: processing and analyzing the collected image, extracting characteristic parameters and morphological information of the fruit, and forming point cloud information of the perspective of the structured light camera (7014); Step 600, based on the third model, the point cloud information of multiple fruits is integrated to reconstruct a three-dimensional model representing the fruit phenotype, wherein the third model is an ICP model, specifically comprising: extracting feature points from images taken from different perspectives to obtain point cloud data of each perspective; registering the filtered point cloud data to align the point clouds of different perspectives to a target point cloud in a reference coordinate system; fusing the point cloud data of multiple perspectives to form a complete three-dimensional point cloud model of the fruit; wherein steps 100 to 600 involve conversion of the laser radar (7011) coordinate system, the structured light camera (7014) coordinate system, the robot arm (7013) coordinate system and the depth camera (7012) coordinate system, and using at least one of external calibration, hand-eye calibration, a method based on geometric constraints or a visual odometer to perform conversion between coordinate systems.

2. The method for collecting plant phenotypes automatically according to claim 1, characterized in that: The laser radar (7011) is arranged on the mobile robot (701).

3. The method for collecting automated plant phenotypes according to claim 1, characterized in that: The second camera is at least one depth camera (7012), and the at least one depth camera (7012) is arranged on the mobile robot (701).

4. An automated plant phenotype collection device (700), characterized in that: Used to implement the method according to any one of claims 1 to 3, comprising: A mobile robot (701), the mobile robot (701) comprising: LiDAR (7011), used to collect map information and convert point clouds into octree system maps; At least one depth camera (7012) for real-time identification of plant fruits and acquisition of depth information; The robotic arm (7013) is used with the RRT algorithm to avoid obstacles and achieve contactless fruit collection; At least one structured light camera (7014) module, used to collect fruit phenotypic information; The controller (702) includes implementing point cloud fusion registration and three-dimensional reconstruction of the fruit through an ICP algorithm, and reducing noise in the point cloud through a filtering algorithm.

5. An electronic device (800), characterized in that: include: A memory (802), a processor (801), and a computer program stored in the memory (802) and executable on the processor (801), wherein the processor (801) implements the method according to any one of claims 1 to 4 when executing the program.

Images