A method and device for locating tea bud and leaf picking points

Abstract

The present invention provides a method and device for locating tea bud picking points. A tea bud instance segmentation model in a target unstructured environment is constructed through collected tea bud image data in an unstructured tea garden environment. The tea bud instance segmentation model in the target unstructured environment is then used to determine the visual recognition result of the tea bud in a small field of view corresponding to the tea bud image data to be tested, and the corresponding three-dimensional spatial coordinates of the tea bud are determined based on the result to determine the picking point and its three-dimensional coordinates.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and a device for locating tea bud and leaf picking points. Background Art

[0002] Tea bud picking faces the dilemma of heavy manual labor, high costs and worker shortage. Especially for famous and high-quality teas that require high tenderness and integrity, they rely heavily on manual picking, which accounts for 60% of the workload in the tea production process. Faced with an aging population and young people's career choices, the automation of famous and high-quality tea picking has become an inevitable trend. Although traditional tea picking machines improve efficiency and reduce costs, the "one-size-fits-all" approach damages the quality of tea and cannot match the high standards of famous and high-quality tea. Therefore, the research and development of automated picking technology that can imitate manual labor and ensure the integrity of tea buds is of great significance to promoting the sustainable development of the tea industry.

[0003] In the visual system for picking tea buds and leaves, accurate positioning of the picking point is a crucial link. Taking one bud and one leaf as an example, the ideal picking point should be accurately locked in the stem area between the first leaf and the second leaf close to the tea bud. Usually, this specific area only accounts for 2% of the total stem length. However, due to the high growth density of tea trees, large leaf area index, complex crown structure and covered tea tree picking points, the existing tea picking machinery often ignores the unique morphology and structure of the famous tea buds and leaves when picking them, making it difficult to accurately position the picking point. The picking point is often positioned at a uniform height, resulting in varying degrees of damage to the picked famous tea buds and leaves, affecting the quality of the tea. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the purpose of the present invention is to provide a method and device for locating the tea bud picking point. The method is based on the constructed tea bud instance segmentation model in the target unstructured environment, predicts the visual recognition result of the tea bud under the small field of view of the tea bud image data to be tested, and then determines the picking point and the three-dimensional coordinates of the picking point. The method accurately locates the tea bud picking point to ensure that the picked tea bud is intact.

[0005] In a first aspect, the present invention provides a method for locating a tea bud picking point, comprising:

[0006] Collect tea bud and leaf image data in an unstructured tea garden environment;

[0007] Labeling the tea bud and leaf image data to construct a tea bud and leaf image dataset;

[0008] Constructing an initialized tea bud and leaf instance segmentation model in an unstructured environment, and using the tea bud and leaf image dataset for training to obtain a tea bud and leaf instance segmentation model in a target unstructured environment;

[0009] Inputting the image data of tea buds to be tested into the tea bud instance segmentation model in the target unstructured environment to obtain the visual recognition result of tea buds in a small field of view;

[0010] Determine the corresponding three-dimensional spatial coordinates of the tea buds according to the image data of the tea buds to be tested and the visual recognition results of the tea buds under the small field of view;

[0011] Based on the visual recognition results of the tea buds and leaves in the small field of view and the three-dimensional spatial coordinates of the tea buds and leaves, the picking points and the three-dimensional coordinates of the picking points are determined.

[0012] In a preferred technical solution of the present invention, the tea bud leaf image data is tea bud leaf RGB image data; labeling the tea bud leaf image data to construct a tea bud leaf image data set includes:

[0013] Determining the category label corresponding to the tea bud leaf RGB image data according to the visibility of the stems;

[0014] The category labels are used to perform image segmentation and annotation on the tea bud leaf RGB image data to construct the tea bud leaf image dataset.

[0015] In a preferred technical solution of the present invention, the image data of tea buds to be tested includes: RGB image data of tea buds to be tested and depth image data of tea buds to be tested; according to the image data of tea buds to be tested and the visual recognition result of tea buds under the small field of view, the corresponding three-dimensional spatial coordinates of tea buds are determined, including:

[0016] Generate a corresponding point cloud set according to the tea bud leaf depth image data to be tested;

[0017] The RGB image data of the tea buds to be tested and the depth image data of the tea buds to be tested are registered, and the three-dimensional coordinates in the point cloud set corresponding to the point coordinates on the RGB image data of the tea buds to be tested are obtained by combining the preset depth camera parameter formula;

[0018] The point coordinates corresponding to the visual recognition results of the tea buds and leaves under the small field of view are mapped to the point cloud set to obtain the three-dimensional spatial coordinates of the tea buds and leaves.

[0019] In a preferred technical solution of the present invention, based on the visual recognition result of the tea buds and leaves in the small field of view and the three-dimensional spatial coordinates of the tea buds and leaves, the picking point and the three-dimensional coordinates of the picking point are determined, including:

[0020] When the visual recognition result of the tea buds and leaves under the small field of view shows that the tea buds and leaves stems and the picking point are clearly visible, the intersection of the detection frame corresponding to the visual recognition result of the tea buds and leaves under the small field of view and the tea buds and leaves stems is taken as the picking point;

[0021] Based on the picking point, the three-dimensional spatial coordinates of the tea buds and leaves are spatially transformed to obtain the three-dimensional coordinates of the picking point.

[0022] In a preferred technical solution of the present invention, based on the visual recognition result of the tea buds and leaves in the small field of view and the three-dimensional spatial coordinates of the tea buds and leaves, the picking point and the three-dimensional coordinates of the picking point are determined, including:

[0023] When the visual recognition result of the tea buds and leaves under the small field of view is that the tea buds and leaves overlap with the stems, a principal component analysis method is used to reduce the dimension of the three-dimensional spatial coordinates of the tea buds and leaves to obtain a corresponding feature vector;

[0024] A feature space is constructed on the eigenvector corresponding to the maximum eigenvalue and a principal axis fitting is performed to obtain the main axis of the stem; based on the main axis of the stem, the picking point and the three-dimensional coordinates of the picking point are determined.

[0025] In a preferred technical solution of the present invention, based on the visual recognition result of the tea buds and leaves in the small field of view and the three-dimensional spatial coordinates of the tea buds and leaves, the picking points and the three-dimensional coordinates of the picking points are determined, including:

[0026] When the visual recognition result of the tea buds and leaves in the small field of view is that the tea buds and tea leaves are visible but the stems are blocked, performing a mean removal calculation on the three-dimensional spatial coordinates of the tea buds and leaves;

[0027] Calculating the pixel area of ​​the covered area by using the pre-established morphological parameter model and the tea bud leaf morphological information corresponding to the tea bud leaf image data to be tested;

[0028] The picking point and the three-dimensional coordinates of the picking point are obtained by inferring the three-dimensional spatial coordinates of the tea buds and leaves after removing the mean, the pixel area and the morphological information of the tea buds and leaves.

[0029] In a second aspect, the present invention provides a tea bud picking point positioning device, comprising:

[0030] The acquisition module is used to collect tea bud and leaf image data in an unstructured tea garden environment;

[0031] A labeling module, used to label the tea bud and leaf image data and construct a tea bud and leaf image data set;

[0032] A training module is used to construct an initialized tea bud and leaf instance segmentation model in an unstructured environment, and use the tea bud and leaf image dataset for training to obtain a tea bud and leaf instance segmentation model in a target unstructured environment;

[0033] A prediction module is used to input the image data of tea buds to be tested into the tea bud instance segmentation model in the target unstructured environment to obtain the visual recognition result of tea buds in a small field of view;

[0034] A spatial coordinate determination module, used to determine the corresponding three-dimensional spatial coordinates of the tea buds and leaves according to the image data of the tea buds and leaves to be tested and the visual recognition results of the tea buds and leaves under the small field of view;

[0035] The picking point information determination module is used to determine the picking point and the three-dimensional coordinates of the picking point based on the visual recognition results of the tea buds and leaves in the small field of view and the three-dimensional spatial coordinates of the tea buds and leaves.

[0036] In a third aspect, the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the method provided in the first aspect are performed.

[0037] In a fourth aspect, the present invention provides a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, runs the steps of the method provided in the first aspect.

[0038] In a fifth aspect, the present disclosure provides a computer program product, comprising a computer program / instructions, wherein the computer program is executed by a processor to perform the steps in the method provided in the first aspect.

[0039] The beneficial effects of the present invention are:

[0040] The present invention provides a method and device for locating tea bud picking points, the method comprising: collecting tea bud image data in an unstructured tea garden environment; labeling the tea bud image data to construct a tea bud image data set; constructing an initialized tea bud instance segmentation model in an unstructured environment, and using the tea bud image data set for training to obtain a tea bud instance segmentation model in a target unstructured environment; inputting tea bud image data to be tested into the tea bud instance segmentation model in the target unstructured environment to obtain a visual recognition result of the tea bud in a small field of view; determining the corresponding three-dimensional spatial coordinates of the tea bud according to the tea bud image data to be tested and the visual recognition result of the tea bud in the small field of view; determining the picking point and the three-dimensional coordinates of the picking point based on the visual recognition result of the tea bud in the small field of view and the three-dimensional spatial coordinates of the tea bud. Through the collected tea bud and leaf image data in the unstructured tea garden environment, a tea bud and leaf instance segmentation model in the target unstructured environment is constructed, and then the tea bud and leaf instance segmentation model in the target unstructured environment is used to determine the tea bud and leaf visual recognition result in a small field of view corresponding to the tea bud and leaf image data to be tested, and the corresponding three-dimensional spatial coordinates of the tea bud and leaf are determined based on the result to determine the picking point and its three-dimensional coordinates of the picking point. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1 This is a flow chart of a first embodiment of a method for locating a tea bud picking point according to the present invention;

[0043] Figure 2 This is a flow chart of a second embodiment of a method for locating a tea bud picking point according to the present invention;

[0044] Figure 3 The present invention is a structural block diagram of a tea bud picking point positioning device embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, and to fully understand and implement how the present disclosure applies technical means to solve technical problems and achieve the corresponding technical effects, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only embodiments of a part of the present disclosure, not all of the embodiments. The embodiments of the present disclosure and the various features in the embodiments can be combined with each other without conflict, and the technical solutions formed are all within the scope of protection of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present disclosure.

[0046] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0047] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0048] Embodiment 1

[0049] See also Figure 1 , Figure 1 This is a flow chart of a first embodiment of a method for locating a tea bud picking point according to the present invention, the method comprising:

[0050] Step S101, collecting tea bud and leaf image data in an unstructured tea garden environment;

[0051] In the embodiment of the present invention, a handheld mobile camera sensing device is used to adjust viewing angles in different directions to collect tea bud and leaf image data in an unstructured tea garden environment.

[0052] Step S102, labeling the tea bud and leaf image data to construct a tea bud and leaf image dataset;

[0053] In the embodiment of the present invention, according to the visibility of the tea bud leaf stems in the tea bud leaf image data, the tea bud leaf image data is labeled to obtain a tea bud leaf image data set, which is used as subsequent model training data. For example, the visibility of the tea bud leaf stems includes that the tea bud leaf stems and picking points are clearly visible, the tea bud leaves overlap with the stems, and the tea buds and tea leaves are visible but the stems are blocked.

[0054] Step S103, constructing an initialized tea bud and leaf instance segmentation model in an unstructured environment, and using the tea bud and leaf image dataset for training to obtain a tea bud and leaf instance segmentation model in a target unstructured environment;

[0055] In the embodiment of the present invention, after constructing the initial tea bud instance segmentation model in the unstructured environment, the tea bud instance segmentation model in the unstructured environment is trained by the image data and its labels in the tea bud image data set to obtain the tea bud instance segmentation model in the target unstructured environment. The tea bud instance segmentation model in the target unstructured environment can segment the tea bud from the complex background and obtain the occlusion of the tea bud picking point.

[0056] Step S104, inputting the image data of the tea buds to be tested into the tea bud instance segmentation model in the target unstructured environment to obtain the visual recognition result of the tea buds in a small field of view;

[0057] In an embodiment of the present invention, if there is tea bud and leaf image data to be tested that needs to be identified, the tea bud and leaf image data to be tested can be input into the tea bud and leaf instance segmentation model in the target unstructured environment, and the model can output the corresponding tea bud and leaf visual recognition results under a small field of view.

[0058] Step S105, determining the corresponding three-dimensional spatial coordinates of the tea buds according to the image data of the tea buds to be tested and the visual recognition result of the tea buds in the small field of view;

[0059] In the embodiment of the present invention, before determining the three-dimensional coordinates of the picking point, it is necessary to complete the acquisition of the spatial coordinates of the tea buds and leaves in the visual recognition results of the tea buds and leaves under a small field of view.

[0060] Step S106, determining the picking point and the three-dimensional coordinates of the picking point based on the visual recognition result of the tea buds and leaves in the small field of view and the three-dimensional spatial coordinates of the tea buds and leaves.

[0061] In an embodiment of the present invention, after determining the three-dimensional spatial coordinates of the tea buds and leaves, the picking points at different levels of visibility of the stems in the image data of the tea buds and leaves to be tested can be determined according to the visual recognition results of the tea buds and leaves under different small fields of view, and then the three-dimensional coordinates of the picking points can be determined based on the picking points.

[0062] A method for locating tea bud picking points provided in an embodiment of the present invention includes: collecting tea bud image data in an unstructured tea garden environment; labeling the tea bud image data to construct a tea bud image data set; constructing an initialized tea bud instance segmentation model in an unstructured environment, and using the tea bud image data set for training to obtain a tea bud instance segmentation model in a target unstructured environment; inputting the tea bud image data to be tested into the tea bud instance segmentation model in the target unstructured environment to obtain a visual recognition result of the tea bud in a small field of view; determining the corresponding three-dimensional spatial coordinates of the tea bud according to the tea bud image data to be tested and the visual recognition result of the tea bud in the small field of view; and determining the picking point and the three-dimensional coordinates of the picking point based on the visual recognition result of the tea bud in the small field of view and the three-dimensional spatial coordinates of the tea bud. Through the collected tea bud and leaf image data in the unstructured tea garden environment, a tea bud and leaf instance segmentation model in the target unstructured environment is constructed, and then the tea bud and leaf instance segmentation model in the target unstructured environment is used to determine the tea bud and leaf visual recognition result in a small field of view corresponding to the tea bud and leaf image data to be tested, and the corresponding three-dimensional spatial coordinates of the tea bud and leaf are determined based on the result to determine the picking point and its three-dimensional coordinates of the picking point.

[0063] Embodiment 2

[0064] See also Figure 2 , Figure 2 This is a flow chart of a second embodiment of a method for locating a tea bud picking point according to the present invention, and the steps include:

[0065] S201, collecting tea bud leaf image data in an unstructured tea garden environment; the tea bud leaf image data is tea bud leaf RGB image data;

[0066] S202, determining a category label corresponding to the tea bud leaf RGB image data according to the visibility of the stems;

[0067] S203, performing image segmentation and annotation on the tea bud leaf RGB image data using the category label to construct the tea bud leaf image dataset;

[0068] S204, constructing an initialized tea bud and leaf instance segmentation model in an unstructured environment, and using the tea bud and leaf image dataset for training to obtain a tea bud and leaf instance segmentation model in a target unstructured environment;

[0069] In the embodiment of the present invention, the tea bud leaf instance segmentation model in an unstructured environment specifically adopts the Transform architecture.

[0070] S205, inputting the image data of tea buds and leaves to be tested into the tea buds and leaves instance segmentation model in the target unstructured environment to obtain the visual recognition result of tea buds and leaves under a small field of view; the image data of tea buds and leaves to be tested includes: RGB image data of tea buds and leaves to be tested and depth image data of tea buds and leaves to be tested;

[0071] S206, generating a corresponding point cloud set according to the depth image data of the tea buds and leaves to be tested;

[0072] S207, registering the RGB image data of the tea buds and leaves to be tested and the depth image data of the tea buds and leaves to be tested, and combining with a preset depth camera parameter formula to obtain the three-dimensional coordinates in the point cloud set corresponding to the point coordinates on the RGB image data of the tea buds and leaves to be tested;

[0073] S208, mapping the point coordinates corresponding to the visual recognition result of the tea buds and leaves in the small field of view to the point cloud set to obtain the three-dimensional spatial coordinates of the tea buds and leaves;

[0074] In an embodiment of the present invention, the depth image data of the tea buds to be tested taken by the D435i depth camera through the infrared camera is transmitted to the computer, and at the same time, the RGB image data of the tea buds to be tested is also transmitted to the computer. Subsequently, a corresponding point cloud set is generated according to the depth image data of the tea buds to be tested, and the RGB image data of the tea buds to be tested and the depth image data of the tea buds to be tested are registered, and the three-dimensional coordinates of any point on the RGB image data of the tea buds to be tested in the point cloud set are obtained in combination with the depth camera parameter formula. Finally, the segmentation result of the tea buds, that is, the coordinates of the tea buds in the visual recognition result of the tea buds under a small field of view are mapped to the point cloud set to obtain the three-dimensional spatial coordinates of the tea buds.

[0075] Specifically, the depth camera parameter formula is:

[0076] ;

[0077] in, is the actual space coordinate of point P, is the focal length of the depth camera, is the coordinate of point P in the RGB image.

[0078] S209, determining the picking point and the three-dimensional coordinates of the picking point based on the visual recognition result of the tea buds and leaves in the small field of view and the three-dimensional spatial coordinates of the tea buds and leaves.

[0079] When the visual recognition result of the tea buds and leaves under the small field of view shows that the tea buds and leaves stems and the picking point are clearly visible, the intersection of the detection frame corresponding to the visual recognition result of the tea buds and leaves under the small field of view and the tea buds and leaves stems is taken as the picking point;

[0080] Based on the picking point, the three-dimensional spatial coordinates of the tea buds and leaves are spatially transformed to obtain the three-dimensional coordinates of the picking point.

[0081] When the visual recognition result of the tea buds and leaves under the small field of view is that the tea buds and leaves overlap with the stems, a principal component analysis method is used to reduce the dimension of the three-dimensional spatial coordinates of the tea buds and leaves to obtain a corresponding feature vector;

[0082] A feature space is constructed on the eigenvector corresponding to the maximum eigenvalue and a principal axis fitting is performed to obtain the main axis of the stem; based on the main axis of the stem, the picking point and the three-dimensional coordinates of the picking point are determined.

[0083] In this embodiment, taking the bottom of the main axis of the stem as a reference, a position is determined on the main axis of the stem according to a preset ratio of the length of the main axis of the stem, and the position is the picking point.

[0084] When the visual recognition result of the tea buds and leaves in the small field of view is that the tea buds and tea leaves are visible but the stems are blocked, performing a mean removal calculation on the three-dimensional spatial coordinates of the tea buds and leaves;

[0085] Calculating the pixel area of ​​the covered area by using the pre-established morphological parameter model and the tea bud leaf morphological information corresponding to the tea bud leaf image data to be tested;

[0086] The picking point and the three-dimensional coordinates of the picking point are obtained by inferring the three-dimensional spatial coordinates of the tea buds and leaves after removing the mean, the pixel area and the morphological information of the tea buds and leaves.

[0087] In the embodiment of the present invention, the tea bud leaf instance segmentation model in the target unstructured environment has three tea bud leaf visual recognition results under small fields of view. The tea bud leaf visual recognition results under small fields of view reflect the visibility of the tea bud leaf stems, which are "tea bud leaf Y ", "tea I ", "tea V ”, where “tea Y ” means the tea buds, leaves, stems and picking points are clearly visible, “tea I " indicates that the tea buds and stems overlap, "tea V ” means the tea buds and tea leaves are visible but the stems are obscured.

[0088] In one embodiment, when the visual recognition result of tea buds and leaves in the small field of view shows that the tea buds and leaves stems and picking points are clearly visible, the detection box in the visual recognition result of tea buds and leaves in the small field of view output by the tea buds and leaves instance segmentation model in the target unstructured environment is used, and the intersection with the tea buds and leaves stems is used as the picking point, and then the three-dimensional coordinates of the picking point are output in combination with the "hand-eye" space coordinate transformation. This method is called the "point" strategy.

[0089] The "hand-eye" space coordinate conversion technology is often used in the field of industrial automation and robotics to describe how to map the coordinates in the camera vision to the coordinate system of the robot's end effector (such as a manipulator). Specifically, this technology involves: Hand (End-effector): The robot's end effector, usually a fixture, gripper or other tool, is used to perform actual physical operations, such as picking tea buds. Eye (Eye): refers to a camera or sensor used to obtain visual information of objects in the environment, such as RGB images and depth information.

[0090] When the visual recognition result of tea buds and leaves under the small field of view is that the tea buds and leaves overlap with the stems, the principal component analysis method is used to reduce the dimensionality of the three-dimensional spatial coordinates of the tea buds and leaves and fit the main axis of the stems. The picking point is defined as 2% above the bottom of the main axis of the stems, and the three-dimensional coordinates of the picking point are output. The mean value operation is performed on all the three-dimensional spatial coordinates of the tea buds and leaves to make the generated data samples more consistent with the established morphological parameter model. The mathematical model constructed by combining leaf area and morphological parameters is then used to determine the bud vertex parameters and other characteristic coordinate points. Then, through qualitative and quantitative analysis and comparison, the blocked area is better restored to determine the picking point.

[0091] This method is called the "line" strategy. In its specific implementation, the steps include: (1) spatial data foil: spatial data filtering of the three-dimensional spatial coordinates of tea buds and leaves to remove noise and unnecessary data points; (2) variance statistics: statistics of the variance on each coordinate axis, and select the axis with the largest variance as the reference axis; (3) reference plane determination: the plane perpendicular to the reference axis is taken as the reference plane, and the space is divided into two subspaces through the median point on the reference axis. The dividing point is taken as the root node, and the left node corresponds to the dividing point with a coordinate smaller than the reference axis, and the right node corresponds to the dividing point with a coordinate larger than the reference axis. points; (4) Selection of new reference axis and reference plane: In each subspace, the maximum variance of the axes other than the reference axis is counted, and the axis with the largest variance is selected as the new reference axis, and its perpendicular plane is selected as the new reference plane; (5) Principal component analysis: Principal component analysis is performed on the three-dimensional point cloud data of tea stems; (6) Principal axis fitting: Among the eigenvectors obtained by principal component analysis, the eigenvector corresponding to the maximum eigenvalue is selected, and a feature space is constructed on it to achieve feature compression. Then, principal axis fitting is performed in the feature space to obtain the main axis of the stem. The specific formula for principal axis fitting is:

[0092] ;

[0093] in, is the slope of the fitting line of the stem, is the intercept of the fitted line.

[0094] The slope of the fitted line for the stem is determined by the following formula:

[0095] ;

[0096] The intercept of the fitted line is determined by the following formula:

[0097] ;

[0098] in, is the stem pixel in the image coordinate system Direction coordinates, For stem pixels Direction coordinates, for The average value of yes The average value of is the number of stem pixels.

[0099] In one embodiment, when the visual recognition result of tea buds and leaves in the small field of view is that the tea buds and tea leaves are visible but the stems are blocked, the three-dimensional spatial coordinates of the tea buds and leaves are de-meaned, and the pixel area of ​​the presented mask is calculated. The blocked picking points are inferred based on the tea bud and leaf morphological information, and the three-dimensional coordinates of the picking points are output.

[0100] This method is called the "face" strategy. In its specific implementation, the steps include: (1) De-meaning calculation: De-meaning the three-dimensional spatial coordinates of tea buds and leaves to make the data more consistent with the morphological parameter model; (2) Pixel area calculation: Using the pre-established morphological parameter model and the tea bud morphological information corresponding to the image data of the tea buds and leaves to be tested, the mask presented by the tea buds and leaves in the image, that is, the pixel area of ​​the covered area, is calculated; (3) Picking point reasoning: Combining the morphological characteristics of the tea buds and leaves with the output of the mathematical model to infer the three-dimensional coordinates of the picking points in the blocked area.

[0101] Among them, the calculation formula for removing the mean value of the three-dimensional spatial coordinates of tea buds and leaves is:

[0102] ;

[0103] ;

[0104] ;

[0105] in, is the three-dimensional coordinate of tea buds and leaves, is the three-dimensional coordinate of the tea buds after removing the mean, To calculate the number of point clouds.

[0106] In this embodiment, a mean removal operation is performed on all point cloud data, that is, the data sample is subtracted ( ), so that the generated data samples are more consistent with the established morphological parameter model. The mathematical model constructed by combining leaf area and morphological parameters , determine the bud vertex parameters and other characteristic coordinate points. Through qualitative and quantitative analysis and comparison, the occluded area is better restored to determine the location of the picking point. This method is called the "face" strategy.

[0107] The embodiment of the present invention discloses a method and device for locating a tea bud picking point, the method comprising: collecting tea bud image data in an unstructured tea garden environment; labeling the tea bud image data to construct a tea bud image data set; constructing an initialized tea bud instance segmentation model in an unstructured environment, and using the tea bud image data set for training to obtain a tea bud instance segmentation model in a target unstructured environment; inputting the tea bud image data to be tested into the tea bud instance segmentation model in the target unstructured environment to obtain a visual recognition result of the tea bud in a small field of view; determining the corresponding three-dimensional spatial coordinates of the tea bud according to the tea bud image data to be tested and the visual recognition result of the tea bud in the small field of view; determining the picking point and the three-dimensional coordinates of the picking point based on the visual recognition result of the tea bud in the small field of view and the three-dimensional spatial coordinates of the tea bud. Through the collected tea bud image data in the unstructured tea garden environment, the tea bud instance segmentation model in the target unstructured environment is constructed, and then the tea bud instance segmentation model in the target unstructured environment is used to determine the visual recognition result of the tea bud under the small field of view corresponding to the image data of the tea bud to be tested, and the corresponding three-dimensional space coordinates of the tea bud are determined based on the result. Finally, based on the three-dimensional space coordinates of the tea bud, different picking point determination strategies are selected according to the visual recognition results of the tea bud under different small fields of view to determine the picking point and its three-dimensional coordinates. This improves the accuracy of picking and ensures that the picked tea buds are intact.

[0108] Embodiment 3

[0109] See also Figure 3 , Figure 3 This is a structural block diagram of an embodiment of a tea bud leaf picking point positioning device of the present invention, the device comprises:

[0110] The acquisition module 301 is used to acquire tea bud and leaf image data in an unstructured tea garden environment;

[0111] A labeling module 302 is used to label the tea bud and leaf image data to construct a tea bud and leaf image data set;

[0112] The training module 303 is used to construct an initialized tea bud and leaf instance segmentation model in an unstructured environment, and use the tea bud and leaf image dataset for training to obtain a tea bud and leaf instance segmentation model in a target unstructured environment;

[0113] Prediction module 304, used for inputting the image data of tea buds to be tested into the tea bud instance segmentation model in the target unstructured environment to obtain the visual recognition result of tea buds in a small field of view;

[0114] A spatial coordinate determination module 305 is used to determine the corresponding three-dimensional spatial coordinates of the tea buds according to the image data of the tea buds to be tested and the visual recognition results of the tea buds under the small field of view;

[0115] The picking point information determination module 306 is used to determine the picking point and the three-dimensional coordinates of the picking point based on the visual recognition results of the tea buds and leaves in the small field of view and the three-dimensional spatial coordinates of the tea buds and leaves.

[0116] In an optional embodiment, the tea bud leaf image data is tea bud leaf RGB image data; the labeling module 302 includes:

[0117] A category label determination submodule, used to determine the category label corresponding to the tea bud leaf RGB image data according to the visibility of the stem;

[0118] The data set construction submodule is used to use the category label to perform image segmentation and annotation on the tea bud leaf RGB image data to construct the tea bud leaf image data set.

[0119] In an optional embodiment, the image data of the tea buds to be tested includes: RGB image data of the tea buds to be tested and depth image data of the tea buds to be tested; the spatial coordinate determination module 305 includes:

[0120] A point cloud set generation submodule, used to generate a corresponding point cloud set according to the depth image data of the tea buds and leaves to be tested;

[0121] A registration submodule, for registering the RGB image data of the tea buds and leaves to be tested and the depth image data of the tea buds and leaves to be tested, and obtaining the three-dimensional coordinates in the point cloud set corresponding to the point coordinates on the RGB image data of the tea buds and leaves to be tested in combination with a preset depth camera parameter formula;

[0122] The mapping submodule is used to map the point coordinates corresponding to the visual recognition results of the tea buds and leaves under the small field of view to the point cloud set to obtain the three-dimensional spatial coordinates of the tea buds and leaves.

[0123] In an optional embodiment, the picking point information determination module 306 includes:

[0124] The first picking point determination submodule is used for, when the visual recognition result of the tea buds and leaves under the small field of view is that the tea buds and leaves stems and the picking point are clearly visible, taking the detection frame corresponding to the visual recognition result of the tea buds and leaves under the small field of view and the intersection of the tea buds and leaves stems as the picking point;

[0125] The picking point three-dimensional coordinate determination submodule is used to perform spatial transformation on the three-dimensional spatial coordinates of the tea bud leaves based on the picking point to obtain the three-dimensional coordinates of the picking point.

[0126] In an optional embodiment, the picking point information determination module 306 includes:

[0127] A dimensionality reduction submodule, for reducing the dimensionality of the three-dimensional spatial coordinates of the tea buds and leaves by using a principal component analysis method to obtain a corresponding feature vector when the visual recognition result of the tea buds and leaves under the small field of view is that the tea buds and leaves overlap with the stems;

[0128] A stem main axis determination submodule is used to construct a feature space on the eigenvector corresponding to the maximum eigenvalue and perform main axis fitting to obtain the stem main axis;

[0129] The second picking point determination submodule is used to determine the picking point according to the main axis of the stem.

[0130] In an optional embodiment, the picking point information determination module 306 includes:

[0131] A de-meaning calculation submodule is used for performing de-meaning calculation on the three-dimensional spatial coordinates of the tea buds and leaves when the visual recognition result of the tea buds and leaves in the small field of view is that the tea buds and tea leaves are visible but the stems are blocked;

[0132] A pixel area calculation submodule, for calculating the pixel area of ​​the covered area by using a pre-established morphological parameter model and the tea bud and leaf morphological information corresponding to the tea bud and leaf image data to be tested;

[0133] The third picking point determination submodule is used to infer the picking point based on the three-dimensional spatial coordinates of the tea buds and leaves after removing the mean, the pixel area and the morphological information of the tea buds and leaves.

[0134] Embodiment 4

[0135] An embodiment of the present invention also provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of a method for locating a tea bud picking point in any embodiment.

[0136] Embodiment 5

[0137] An embodiment of the present invention further provides a computer storage medium having a computer program stored thereon, and when the computer program is executed by the processor, the steps of a method for locating a tea bud picking point in any embodiment are implemented.

[0138] Embodiment 6

[0139] This embodiment also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implements the steps of a method for locating a tea bud picking point in the above embodiment.

[0140] The processor may include, but is not limited to, one or more processors or microprocessors, etc. Each processor may be an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components to execute the methods in the above embodiments.

[0141] The computer-readable storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, and the computer-readable storage medium may include but is not limited to, for example, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (e.g., hard disk, floppy disk, solid-state drive, removable disk, CD-ROM, DVD-ROM, Blu-ray disc, etc.).

[0142] The computer-readable storage medium may also store at least one computer executable program / instruction, which may be, for example, a computer-readable instruction. The computer-readable storage medium includes, but is not limited to, for example, a volatile memory and / or a non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The computer-readable storage medium may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. For example, a non-transitory computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device runs the computer-readable instructions stored on the computer-readable storage medium, the various methods described above may be performed.

[0143] In addition, the computer device may also include (but not limited to) a data bus, an input / output (I / O) bus, a display, and input / output devices (eg, keyboard, mouse, speaker, etc.), etc.

[0144] The processor may communicate with external devices via an I / O bus via a wired or wireless network.

[0145] In one embodiment, the at least one computer executable instruction may also be compiled into or constitute a software product / computer program product, wherein one or more computer executable instructions are executed by a processor to perform the various functions and / or method steps in the embodiments described in the present technology.

[0146] In the embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0147] It should be noted that in the present disclosure, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element limited by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0148] Although the embodiments disclosed in the present disclosure are as above, the above contents are only embodiments adopted for facilitating the understanding of the present disclosure and are not intended to limit the present disclosure. Any technician in the technical field to which the present disclosure belongs can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in the present disclosure, but the scope of patent protection of the present disclosure shall still be subject to the scope defined in the attached claims.

Claims

1. A method for locating tea bud picking points, characterized in that: include: Collect tea bud and leaf image data in an unstructured tea garden environment; Labeling the tea bud and leaf image data to construct a tea bud and leaf image dataset; Constructing an initialized tea bud and leaf instance segmentation model in an unstructured environment, and using the tea bud and leaf image dataset for training to obtain a tea bud and leaf instance segmentation model in a target unstructured environment; Inputting the image data of tea buds to be tested into the tea bud instance segmentation model in the target unstructured environment to obtain the visual recognition result of tea buds in a small field of view; The tea bud leaf image data to be tested includes: RGB image data of the tea bud leaf to be tested and depth image data of the tea bud leaf to be tested; Determining the corresponding three-dimensional spatial coordinates of the tea buds according to the image data of the tea buds to be tested and the visual recognition result of the tea buds under the small field of view, including: generating a corresponding point cloud set according to the depth image data of the tea buds to be tested; registering the RGB image data of the tea buds to be tested and the depth image data of the tea buds to be tested, and obtaining the three-dimensional coordinates in the point cloud corresponding to the point coordinates on the RGB image data of the tea buds to be tested in combination with a preset depth camera parameter formula; mapping the point coordinates corresponding to the visual recognition result of the tea buds under the small field of view to the point cloud set, obtaining the three-dimensional spatial coordinates of the tea buds; Based on the visual recognition result of the tea buds and leaves in the small field of view and the three-dimensional spatial coordinates of the tea buds and leaves, the picking points and the three-dimensional coordinates of the picking points are determined, including: When the visual recognition result of the tea buds and leaves under the small field of view shows that the tea buds and leaves stems and the picking point are clearly visible, the intersection of the detection frame corresponding to the visual recognition result of the tea buds and leaves under the small field of view and the tea buds and leaves stems is taken as the picking point; based on the picking point, the three-dimensional spatial coordinates of the tea buds and leaves are spatially transformed to obtain the three-dimensional coordinates of the picking point; When the visual recognition result of the tea buds and leaves under the small field of view is that the tea buds and leaves overlap with the stems, the principal component analysis method is used to reduce the dimension of the three-dimensional spatial coordinates of the tea buds and leaves to obtain the corresponding eigenvector; the feature space is constructed on the eigenvector corresponding to the maximum eigenvalue and the principal axis is fitted to obtain the main axis of the stem; the picking point and the three-dimensional coordinates of the picking point are determined according to the main axis of the stem; When the visual recognition result of tea buds and leaves in the small field of view is that the tea buds and tea leaves are visible but the stems are blocked, the three-dimensional spatial coordinates of the tea buds and leaves are de-meaned; the pixel area of ​​the covered area is calculated using the pre-established morphological parameter model and the tea bud and leaf morphological information corresponding to the image data of the tea bud and leaves to be tested; and the picking point and the three-dimensional coordinates of the picking point are inferred based on the three-dimensional spatial coordinates of the tea bud and leaves after de-meaning, the pixel area and the tea bud and leaf morphological information.

2. The method for locating tea bud picking points according to claim 1, characterized in that: The tea bud leaf image data is tea bud leaf RGB image data; labeling the tea bud leaf image data to construct a tea bud leaf image data set includes: Determining the category label corresponding to the tea bud leaf RGB image data according to the visibility of the stems; The category labels are used to perform image segmentation and annotation on the tea bud leaf RGB image data to construct the tea bud leaf image dataset.

3. A tea bud picking point positioning device, characterized in that: include: The acquisition module is used to collect tea bud and leaf image data in an unstructured tea garden environment; A labeling module, used to label the tea bud and leaf image data and construct a tea bud and leaf image data set; A training module is used to construct an initialized tea bud and leaf instance segmentation model in an unstructured environment, and use the tea bud and leaf image dataset for training to obtain a tea bud and leaf instance segmentation model in a target unstructured environment; A prediction module is used to input the image data of tea buds to be tested into the tea bud instance segmentation model in the target unstructured environment to obtain the visual recognition result of tea buds in a small field of view; The tea bud leaf image data to be tested includes: RGB image data of the tea bud leaf to be tested and depth image data of the tea bud leaf to be tested; A spatial coordinate determination module, used to determine the corresponding three-dimensional spatial coordinates of the tea buds and leaves according to the image data of the tea buds and leaves to be tested and the visual recognition results of the tea buds and leaves under the small field of view; A picking point information determination module, used to determine the picking point and the three-dimensional coordinates of the picking point based on the visual recognition result of the tea buds and leaves in the small field of view and the three-dimensional spatial coordinates of the tea buds and leaves; The spatial coordinate determination module comprises: A point cloud set generation submodule, used to generate a corresponding point cloud set according to the depth image data of the tea buds and leaves to be tested; A registration submodule, for registering the RGB image data of the tea buds and leaves to be tested and the depth image data of the tea buds and leaves to be tested, and obtaining the three-dimensional coordinates in the point cloud set corresponding to the point coordinates on the RGB image data of the tea buds and leaves to be tested in combination with a preset depth camera parameter formula; A mapping submodule, used for mapping the point coordinates corresponding to the visual recognition result of the tea buds and leaves under the small field of view to the point cloud set to obtain the three-dimensional spatial coordinates of the tea buds and leaves; The picking point information determination module includes: The first picking point determination submodule is used for, when the visual recognition result of the tea buds and leaves under the small field of view is that the tea buds and leaves stems and the picking point are clearly visible, taking the detection frame corresponding to the visual recognition result of the tea buds and leaves under the small field of view and the intersection of the tea buds and leaves stems as the picking point; A picking point three-dimensional coordinate determination submodule is used to perform spatial transformation on the three-dimensional spatial coordinates of the tea buds and leaves based on the picking point to obtain the three-dimensional coordinates of the picking point; A dimensionality reduction submodule, for reducing the dimensionality of the three-dimensional spatial coordinates of the tea buds and leaves by using a principal component analysis method to obtain a corresponding feature vector when the visual recognition result of the tea buds and leaves under the small field of view is that the tea buds and leaves overlap with the stems; A stem main axis determination submodule is used to construct a feature space on the eigenvector corresponding to the maximum eigenvalue and perform main axis fitting to obtain the stem main axis; A second picking point determination submodule is used to determine the picking point according to the main axis of the stem; A de-meaning calculation submodule is used for performing de-meaning calculation on the three-dimensional spatial coordinates of the tea buds and leaves when the visual recognition result of the tea buds and leaves in the small field of view is that the tea buds and tea leaves are visible but the stems are blocked; A pixel area calculation submodule, for calculating the pixel area of ​​the covered area by using a pre-established morphological parameter model and the tea bud morphological information corresponding to the tea bud image data to be tested; The third picking point determination submodule is used to infer the picking point based on the three-dimensional spatial coordinates of the tea buds and leaves after removing the mean, the pixel area and the morphological information of the tea buds and leaves.

4. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 2 is executed.

5. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 2 is performed.

6. A computer program product comprising a computer program / instructions, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.

Images