A method and device for identifying tea buds and determining their position and posture

Abstract

The present invention discloses a method and device for identifying tea buds and determining their position and posture, including: collecting tea images and preprocessing the tea images; constructing a data set based on the preprocessed tea images, and using an instance segmentation network for training to obtain a deep learning network model; collecting RGB images for tea to be identified, and using a depth camera to collect depth images, identifying the RGB images based on the deep learning network model, and obtaining recognition results; based on the recognition results, using an instance segmentation network to obtain tea picking points, and based on the recognition results, obtaining the center point of a tea bud bounding box, matching the tea picking points with the center point of a tea bud bounding box; matching the RGB images with the depth images, obtaining the three-dimensional coordinates of the tea picking points based on the optical center of the camera, and obtaining the tea picking direction based on the three-dimensional coordinates. The application of the method of the present invention can improve the efficiency and accuracy of machine picking of tea, and reduce the labor cost in tea picking operations.

Description

Technical Field

[0001] The invention belongs to the technical field of machine vision and artificial intelligence, and in particular relates to a method and a device for identifying young tea buds and determining their positions and postures. Background Art

[0002] my country is a major tea-growing and consuming country. Tea occupies an important position in the lives of the people due to its high drinking value and economic value. The production and processing of tea mainly includes picking, withering, rolling and drying, among which picking is the first of all processes and has an important impact on the subsequent processing of tea.

[0003] At present, tea picking in my country mainly relies on manual picking and mechanical picking. However, with the development of society, tea picking faces more and more problems. On the one hand, with the process of urbanization, the agricultural labor force has shifted, and the number of people directly engaged in agricultural production has continued to decrease. At the same time, the aging of people engaged in agricultural production has become more and more serious. The cost of manual picking is becoming increasingly unsustainable, and the labor cost is also increasing. On the other hand, the current mechanical tea picking is mostly carried out in a "one-size-fits-all" way, which is very destructive to tea leaves and tea trees, and the quality of the picked tea is not high.

[0004] Obtaining the tea bud picking point and picking posture is very necessary for picking tea. Since the tea buds to be picked do not grow completely vertically upward, some existing technologies only determine the picking point of the tea buds, then move the picking equipment to just above the picking point, and then move vertically downward to the picking point for picking. This method has poor picking effect in actual use. Summary of the invention

[0005] The purpose of the present invention is to provide a method and device for identifying tea buds and determining their position and posture, so as to solve the problems existing in the above-mentioned prior art.

[0006] On the one hand, to achieve the above-mentioned purpose, the present invention provides a method for identifying tea buds and determining their position and posture, comprising the following steps:

[0007] Collecting tea leaves images and preprocessing the tea leaves images;

[0008] A dataset is constructed based on the preprocessed tea images, and an instance segmentation network is used for training to obtain a deep learning network model.

[0009] For the tea leaves to be identified, an RGB image is collected, and a depth image is collected using a depth camera, and the RGB image is identified based on the deep learning network model to obtain an identification result;

[0010] Based on the recognition results, an instance segmentation network is used to obtain the tea picking points, and the RGB image is matched with the depth image to obtain the three-dimensional coordinates of the tea picking points based on the camera optical center.

[0011] Based on the recognition result, the center point of the tea bud bounding box is obtained, and the tea picking point is matched with the center point of the tea bud bounding box;

[0012] Get the tea picking direction based on the matching results.

[0013] Optionally, the process of preprocessing the tea leaf image includes:

[0014] Eliminate invalid images and amplify valid images. The amplification methods include rotating images, changing grayscale, and mirroring.

[0015] The whole tea buds and picking areas in the amplified tea image are labeled.

[0016] Optionally, the recognition result includes tea varieties, tea bud areas, and tea picking areas; after obtaining the tea varieties, the deep learning network model automatically identifies the tea picking areas according to the tea varieties.

[0017] Optionally, in the process of identifying the depth image based on the deep learning network model, a picking area can be specified, and the deep learning network model performs identification based on the specified picking area.

[0018] Optionally, the method of obtaining tea picking points using an instance segmentation network is as follows:

[0019] ;

[0020] Where p is the probability that a pixel in the depth image is predicted as a picking point by the deep learning network, x and y are the horizontal and vertical coordinates of the pixel in the depth image, and x c and c are the horizontal and vertical coordinate values ​​of the tea picking point, i is each numbered pixel point in the calculation process, and c represents the centroid of the picking area.

[0021] Optionally, the process of obtaining the center point of the tea bud boundary box includes:

[0022] A tea bud boundary box is obtained based on the recognition result, and the middle value of the upper left point and the lower right point of the tea bud boundary box is calculated to obtain the center point of the tea bud boundary box.

[0023] Optionally, the process of matching the tea picking point with the center point of the tea bud bounding box includes:

[0024] The first point set and the second point set are constructed for the tea picking points and the center points of the tea bud bounding box respectively, so that the points in the first point set find the nearest points in the second point set one by one and match them, and the tea picking points and the corresponding tea bud bounding box center points are matched one by one.

[0025] Optionally, the process of obtaining the tea picking direction based on the matching result includes:

[0026] Connect the center point of the tea bud boundary box with the tea picking point, extract the points where the line passes through the picking area, obtain the spatial three-dimensional coordinates of the extracted points based on the depth information of the depth camera, fit the spatial three-dimensional coordinates into a straight line, obtain the growth direction of the tea bud, and regard the opposite direction of the tea bud growth direction as the tea picking direction.

[0027] On the other hand, to achieve the above-mentioned purpose, the present invention also provides a tea bud identification and position determination device, comprising a collection module, a processing module, and an output module connected in sequence;

[0028] The acquisition module is used to synchronously acquire RGB images and depth information of tea leaves to be identified in the tea garden;

[0029] The processing module is used to process the RGB image and depth information to obtain the picking point and picking posture of the tea buds;

[0030] The output module is used to transmit the picking point and picking posture of the tea buds to the picking operation equipment.

[0031] Optionally, the processing module uses an instance segmentation network to obtain tea picking points and the center point of the tea bud bounding box, connects the center point of the tea bud bounding box and the tea picking points, extracts the points where the line passes through the picking area, and obtains the spatial three-dimensional coordinates of the extracted points based on the depth information, fits the spatial three-dimensional coordinates into a straight line, obtains the growth direction of the tea buds, and regards the opposite direction of the tea bud growth direction as the tea picking direction.

[0032] The technical effects of the present invention are:

[0033] The present invention identifies the young buds of tea leaves and their picking points by means of deep learning. By using an instance segmentation network fused with ResNet (residual network) and FPN (feature pyramid network), it is possible to accurately identify tea leaves of different sizes in different pictures. Moreover, the instance segmentation network can help users see the shape of tea buds more clearly.

[0034] The application of this technical method can improve the efficiency and accuracy of machine-picking tea leaves. The ability to obtain picking postures can help with accurate and efficient picking of tea buds, thereby reducing labor costs in tea picking operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0036] Figure 1 Schematic diagram of the process of identifying tea buds and determining their position and posture in an embodiment of the present invention.

[0037] Figure 2 Schematic diagram of the structure of the equipment for identifying tea buds and determining their position and posture in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0039] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here. Example 1

[0040] like Figure 1 As shown, this embodiment provides a method for identifying tea buds and determining their position and posture.

[0041] The present invention is mainly realized through deep learning technology and machine vision technology in artificial intelligence.

[0042] The specific steps are as follows:

[0043] (1) The first step is to collect tea images. Image acquisition equipment is used to obtain tea buds to be picked and establish a tea bud image dataset. The types of tea buds include single buds, one bud and one leaf, one bud and two leaves, and different types of tea.

[0044] (2) Process the tea image dataset, including removing invalid photos and amplifying valid tea images. The amplification methods include rotating the image, changing the grayscale, mirroring, etc. Then complete the annotation of the tea images. The annotated content includes the entire tea bud (such as single bud, one bud and one leaf, one bud and two leaves, etc.) and its picking area.

[0045] (3) Using the obtained tea image dataset for deep learning training, the present invention uses one type of image deep learning network, namely instance segmentation network, to perform deep learning training of tea images, such as Mask-RCNN network. A trained deep learning network model is obtained.

[0046] (4) Collect RGB images of the tea leaves to be identified, and use a depth camera to collect depth images. Use the trained deep learning model to identify the RGB of the tea buds to obtain the tea buds and picking areas. The identification results include three categories: tea variety, tea bud area, and tea picking area.

[0047] (5) Calculate the tea picking point based on the identified tea bud picking area. The data structure of the picking area obtained by using the instance segmentation network is the probability that each pixel point in the entire image matrix is ​​a picking point. The pixel position of the picking point in the image can be calculated as follows:

[0048] ;

[0049] Where p is the probability that a pixel in the depth image is predicted as a picking point by the deep learning network, x and y are the horizontal and vertical coordinates of the pixel in the image, and x c and c It is the horizontal and vertical coordinates of the picking point. The calculation of this point is obtained by calculating the centroid coordinates of the picking area. C is the abbreviation of centroid. There are many pixels in the picking area. The calculation is to traverse all the pixels. i is each numbered pixel in the calculation process.

[0050] After obtaining the variety of tea, the user can be automatically informed of the area to be picked and the location of the picking point for typical tea varieties. For example, the Maojian type of tea in the industry picks single buds as the target tea. When the current deep learning system recognizes that it is Maojian tea, it can automatically identify the area between the single bud and the first leaf as the area to be picked, and then calculate the picking point.

[0051] In addition to automatically identifying the type of tea and automatically identifying the picking area and picking point of that type of tea, the user can also independently specify the picking area of ​​the tea buds. For example, if the user inputs that he needs to pick one bud and one leaf, the system will automatically identify the area between the first leaf and the second leaf as the picking area based on the user input.

[0052] (6) By matching the RGB image with the depth image, the three-dimensional coordinates of the picking point based on the camera optical center can be obtained.

[0053] (7) After the picking point is confirmed, the growth direction of the tea buds is calculated. First, the center point of the tea bud boundary box (the output result of the tea bud identified in step (4) includes the boundary box of the tea bud) is calculated (the center point is calculated by finding the middle value of the upper left point and the lower right point of the boundary box), and the center is connected to the picking point in step (5). The line will cross the picking area identified in step (4) (because the line must pass through the picking point inside the picking area), and the points of the part of the line that passes through the picking area are extracted. The spatial three-dimensional coordinates of these points are obtained through the depth information of the depth camera, and finally these spatial points are fitted into a straight line in some way (such as the least squares method). The straight line is the growth direction of the tea buds. The growth direction of the tea buds is opposite to the direction in which the picking machine needs to move to the picking point.

[0054] (8) The above steps can determine the picking point and direction of the tea buds.

[0055] Among them, in the process of tea labeling, the tea buds and picking areas of the tea need to be labeled separately, so as to use the deep learning network to train the two.

[0056] The matching problem of picking points and the center points of the tea bud bounding box is also a key point of the present invention. The picking points of the tea buds on the picture obtained by the instance segmentation network and the center points of the tea bud bounding box are not one-to-one corresponding. That is, suppose that three tea buds to be picked and three picking points are identified on a picture. People can directly judge that a certain picking point belongs to a certain tea bud, but the computer cannot make a direct judgment. At this time, there must be a corresponding point matching method to match the picking points with the corresponding tea buds. The computer cannot directly match the tea buds with their corresponding picking points. The method proposed by the present invention divides the picking points and the center points of the tea bud bounding box into a first point set and a second point set. The points in the first point set search for the points closest to them in the second point set one by one. Matching them can make the picking points and their corresponding tea bud bounding box center points one by one. Because it is found through actual observation that the picking point of a tea bud is closest to its own tea bud bounding box center point compared to the bounding box center points of other tea buds.

[0057] This embodiment also provides a device for identifying tea buds and determining their positions and postures, which is specifically as follows:

[0058] like Figure 2As shown, the equipment for identifying tea buds and determining their posture mainly includes an acquisition module, a processing module, and an output module connected in sequence. The acquisition module is used to synchronously collect RGB images and depth information of tea leaves to be identified in the tea garden, and the processing module is used to process the collected images, including identifying the tea buds and the corresponding picking areas in the images, calculating the coordinates of the picking points and the center points of the tea bud boundary box, connecting the two points to obtain the crossing points of the picking area, and matching the crossing points with the depth information obtained by the depth camera to obtain the three-dimensional coordinates of these points, fitting these points into a straight line, and obtaining the spatial posture of the straight line. The output module is used to output the picking points and picking postures of the tea buds identified in the image to the picking operation equipment.

[0059] The problem solved by the present invention is not only to identify the tea buds and the picking position of the tea leaves, but also to obtain the growth direction of the tea leaves. The growth direction and the picking direction are opposite directions. After obtaining the picking point coordinates and the growth direction, the picking equipment can be better guided to carry out accurate picking of tea leaves.

[0060] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for identifying tea buds and determining their position and posture, characterized in that: The following steps are involved: Collecting tea leaves images and preprocessing the tea leaves images; A data set is constructed based on the preprocessed tea images, and an instance segmentation network is used for training to obtain a deep learning network model; the instance segmentation network is a Mask-RCNN network; For the tea leaves to be identified, an RGB image is collected, and a depth image is collected using a depth camera, and the RGB image is identified based on the deep learning network model to obtain an identification result; Based on the recognition results, an instance segmentation network is used to obtain the tea picking points, and the RGB image is matched with the depth image to obtain the three-dimensional coordinates of the tea picking points based on the camera optical center. Based on the recognition result, the center point of the tea bud bounding box is obtained, and the tea picking point is matched with the center point of the tea bud bounding box; Get the tea picking direction based on the matching results; Instructing the picking equipment to pick tea leaves according to the three-dimensional coordinates of each picking point and the corresponding tea picking direction; Among them, the method of using instance segmentation network to obtain tea picking points is as follows: Where p is the probability that a pixel in the depth image is predicted as a picking point by the deep learning network, x and y are the horizontal and vertical coordinates of the pixel in the depth image, and x c and c is the horizontal and vertical coordinate value of the tea picking point, i is each numbered pixel point in the calculation process, and c represents the centroid of the picking area; The process of obtaining the center point of the tea bud boundary box includes: Based on the recognition result, a tea bud boundary box is obtained, and the middle value of the upper left point and the lower right point of the tea bud boundary box is calculated to obtain the center point of the tea bud boundary box; The process of matching the tea picking point with the center point of the tea bud boundary box includes: Construct the first point set and the second point set for the tea picking point and the center point of the tea bud bounding box respectively, so that the points in the first point set find the closest points in the second point set one by one and match them, and make the tea picking point and the corresponding tea bud bounding box center point correspond one by one; The process of obtaining the tea picking direction based on the matching result includes: Connect the center point of the tea bud boundary box with the tea picking point, extract the points where the line passes through the picking area, obtain the spatial three-dimensional coordinates of the extracted points based on the depth information of the depth camera, fit the spatial three-dimensional coordinates into a straight line, obtain the growth direction of the tea bud, and regard the opposite direction of the tea bud growth direction as the tea picking direction.

2. The method for identifying tea buds and determining their position and posture according to claim 1, characterized in that: The process of preprocessing the tea leaf image includes: Eliminate invalid images and amplify valid images. The amplification methods include rotating images, changing grayscale, and mirroring. The whole tea buds and picking areas in the amplified tea image are labeled.

3. The method for identifying tea buds and determining their position and posture according to claim 1, characterized in that: The identification results include tea varieties, tea bud areas, and tea picking areas; after obtaining the tea varieties, the deep learning network model automatically identifies the tea picking areas based on the tea varieties.

4. The method for identifying tea buds and determining their position and posture according to claim 1, characterized in that: During the process of recognizing the depth image based on the deep learning network model, a picking area may be specified, and the deep learning network model performs recognition based on the specified picking area.

5. A device for identifying young tea buds and determining their position and posture, used to execute the method for identifying young tea buds and determining their position and posture as claimed in any one of claims 1 to 4, characterized in that: It includes a collection module, a processing module, and an output module connected in sequence; The acquisition module is used to synchronously acquire RGB images and depth information of tea leaves to be identified in the tea garden; The processing module is used to process the RGB image and depth information to obtain the picking point and picking posture of the tea buds; The output module is used to transmit the picking point and picking posture of the tea buds to the picking operation equipment; The processing module uses an instance segmentation network to obtain tea picking points and the center point of the tea bud boundary box, connects the center point of the tea bud boundary box with the tea picking points, extracts points where the connection line passes through the picking area, and obtains the spatial three-dimensional coordinates of the extracted points according to the depth information, fits the spatial three-dimensional coordinates into a straight line, obtains the tea bud growth direction, and regards the opposite direction of the tea bud growth direction as the tea picking direction; the instance segmentation network is a Mask-RCNN network; The method in which the processing module uses the instance segmentation network to obtain the tea picking point is as follows: Where p is the probability that a pixel in the depth image is predicted as a picking point by the deep learning network, x and y are the horizontal and vertical coordinates of the pixel in the depth image, and x c and c is the horizontal and vertical coordinate value of the tea picking point, i is each numbered pixel point in the calculation process, and c represents the centroid of the picking area; The process of obtaining the center point of the tea bud boundary box by the processing module includes: Based on the recognition result, a tea bud boundary box is obtained, and the middle value of the upper left point and the lower right point of the tea bud boundary box is calculated to obtain the center point of the tea bud boundary box; The processing module is also used to match the tea picking point with the center point of the tea bud boundary box, specifically including: The first point set and the second point set are constructed for the tea picking points and the center points of the tea bud bounding box respectively, so that the points in the first point set find the nearest points in the second point set one by one and match them, and the tea picking points and the corresponding tea bud bounding box center points are matched one by one.

Images