Fruit picking point positioning method

Abstract

The invention provides a fruit picking point positioning method. The method comprises the following steps: acquiring a to-be-detected picture; inputting the to-be-detected picture into a trained target detection network to obtain a fruit detection frame and a fruit stem detection frame of the to-be-detected picture; matching the fruit detection frame with the fruit stem detection frame to obtain matching data; and according to the matching data, obtaining a fruit picking point. According to the method, fruits and fruit stems in a to-be-detected picture are detected through a target detection network, a fruit detection frame and a fruit stem detection frame are obtained respectively, then the two detection frames are matched so that the corresponding relation between the fruits and the fruit stems can be determined, finally, according to matching data, the picking points of the fruits are calculated, and the picking accuracy of the fruits is improved. Therefore, the obtained picking points are more accurate.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a fruit picking point positioning method. Background Art

[0002] With the booming development of the strawberry planting industry, the realization of unmanned and automated strawberry picking has become a key area of ​​intelligent agricultural research. At present, intelligent mobile automatic picking robots mainly use three mainstream technical methods to locate strawberry picking points. The first method is to use visual systems and machine learning algorithms, such as support vector machines (SVM) and relevance vector machines (RVM), to identify strawberry fruits in images, and combine manual experience to determine the location of the fruit stems and their picking points. Although this method has a fast recognition speed, it does not accurately identify the fruit stems, resulting in insufficient positioning accuracy in complex scenes, which may cause fruit damage. The second method is to focus on identifying the fruit stems and calculating their center of gravity to determine the picking points. Although it improves the accuracy of fruit stem recognition, it may mistakenly pick immature fruits and cause resource waste. The third method combines detection and segmentation technology to accurately identify strawberry fruits and fruit stems, and then perform morphological processing to determine the picking points. Although this method performs well in recognition accuracy, it involves complex algorithms, has high requirements for computing resources, and may affect real-time performance. Strawberries belong to the berry fruit. Due to the soft nature of the fruit, they are usually picked by gripper or cutting robots. Strawberries have diverse shapes, and the recognition result of the classification algorithm is the smallest rectangle of the area where the fruit or stalk is located. Gripper robots cannot confirm the specific outlines of the fruit and stalk, which can easily cause damage to the fruit during the picking process. The method of locating the stalk and picking point based on manual experience is not universal. Cutting robots often make empty cuts or cut the wrong stalks of other fruits, which greatly wastes the ability of the picking robot and fruit resources, and cannot improve the efficiency of farm picking. Summary of the invention

[0003] In view of the above defects of the prior art, the present invention provides a fruit picking point positioning method to solve the technical problem of inaccurate fruit picking point positioning.

[0004] To achieve the above-mentioned purpose and other related purposes, the present invention provides a method for locating a fruit picking point, comprising: obtaining a picture to be detected; inputting the picture to be detected into a trained target detection network to obtain a fruit detection frame and a fruit stalk detection frame of the picture to be detected; matching the fruit detection frame and the fruit stalk detection frame to obtain matching data; and obtaining a fruit picking point based on the matching data.

[0005] In one embodiment of the present invention, the fruit detection frame and the fruit stalk detection frame are matched to obtain matching data, including: obtaining an association matrix based on the fruit detection frame and the fruit stalk detection frame, in which the elements in the i-th row and the j-th column are the displacements between the center point of the upper edge of the i-th fruit detection frame and the center point of the lower edge of the j-th fruit stalk detection frame; according to the association matrix, the fruit detection frame and the fruit stalk detection frame are matched to obtain matching data.

[0006] In one embodiment of the present invention, the fruit detection frame and the fruit stalk detection frame are matched according to the association matrix, including: finding an element in the association matrix whose absolute value of displacement is less than a preset minimum distance, matching the fruit detection frame and the fruit stalk detection frame corresponding to the element, and setting all elements in the row and column where the element is located to null values; retaining the elements in the association matrix whose displacement is within a preset range, and setting all other elements to null values; finding an element in a row or a column of the association matrix that has only one non-null value, matching the fruit detection frame and the fruit stalk detection frame corresponding to the element, and setting all elements in the row and column where the element is located to null values; finding all non-null value elements in each row of the association matrix, and matching the fruit detection frame and the fruit stalk detection frame corresponding to the element with the smallest absolute value of displacement.

[0007] In one embodiment of the present invention, in the step of retaining the elements in the association matrix whose displacement is within a preset range, the preset range is -20 to 60 mm.

[0008] In one embodiment of the present invention, the format of the matching data is [F dis ,S box ,F box ], where: if the fruit detection frame matches the stem detection frame, S box is the matching fruit detection frame, F box is the matching fruit stem detection frame, F dis For S box and F box The displacement value in the association matrix; if the fruit detection frame does not match the stalk detection frame, S box is the unmatched fruit detection frame, F box and F dis is the preset mark; if the fruit stem detection frame does not match the fruit detection frame, F box is the unmatched fruit stem detection frame, S box and F dis The preset identifier.

[0009] In one embodiment of the present invention, in the association matrix, when the center point of the lower edge of the fruit stalk detection frame is above the center point of the upper edge of the fruit detection frame, the displacement between the two is recorded as a positive value, otherwise, the displacement between the two is recorded as a negative value; based on the matching data, the fruit picking point is obtained, including: for the matched fruit detection frame and fruit stalk detection frame, judging whether the displacement between the two is a negative value: if yes, taking the intersection of the median line of the fruit stalk detection frame and the upper edge of the fruit detection frame as the picking point; if not, taking the center point of the lower edge of the fruit stalk detection frame as the picking point.

[0010] Beneficial effects of the present invention: The present invention proposes a method for locating a fruit picking point, which detects the fruit and the fruit stalk in the image to be detected through a target detection network, obtains a fruit detection frame and a fruit stalk detection frame respectively, and then matches the two detection frames to determine the correspondence between the fruit and the fruit stalk, and finally calculates the fruit picking point based on the matching data, so that the picking point obtained will be more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0012] Figure 1 A general flow chart provided for an embodiment of the present invention; Figure 2 A flowchart of obtaining matching data provided by an embodiment of the present invention; Figure 3 A schematic diagram of the displacement of the center point of a detection frame provided by an embodiment of the present invention; Figure 4 A detection frame matching flow chart provided by an embodiment of the present invention; Figure 5 A schematic diagram of multiple strawberry picking points provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0013] The following describes the embodiments of the present invention through specific specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. It should be noted that the following embodiments and the features in the embodiments can be combined with each other without conflict. In addition to the specific methods, equipment, and materials used in the embodiments, according to the technical personnel in this technical field's grasp of the prior art and the records of the present invention, any methods, equipment, and materials of the prior art that are similar or equivalent to the methods, equipment, and materials described in the embodiments of the present invention can also be used to implement the present invention.

[0014] It should be understood that the terms used in the embodiments of the present invention are for describing specific embodiments rather than for limiting the scope of protection of the present invention. Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those generally understood by those skilled in the art.

[0015] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In some of the embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0016] The flowchart and block diagram in the accompanying drawings illustrate the system architecture, functions and operations that may be implemented according to the methods and computer program products of various embodiments disclosed in the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, the program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0017] It should be noted that the fruits in the present invention include all fruits with stalks, such as strawberries, raspberries, apples, pears, oranges, etc. In the following specific embodiments, strawberries are used as an example for explanation, but the invention is also applicable to other fruits.

[0018] See also Figure 1 , Figure 1A fruit picking point positioning method provided in one embodiment of the present invention includes steps S100 to S400.

[0019] Step S100, obtaining a picture to be detected. The picture to be detected is a picture taken by the picking machine. After the picture to be detected is processed in subsequent steps to obtain a picking point, the picking machine picks the fruit according to the identified picking point.

[0020] Step S200: Input the image to be detected into the trained target detection network to obtain the fruit detection frame and the fruit stem detection frame of the image to be detected. The target detection network can be, for example, the YOLO series, R-CNN series, SSD, RetinaNet, etc. Different target detection networks have their own advantages and disadvantages. The selection of a suitable model needs to be comprehensively considered according to the actual application scenario and requirements.

[0021] The target detection network needs to be trained to obtain a trained target detection network. The training process is generally divided into constructing a sample set, using the sample set to train the target detection network, and evaluating and testing the trained target detection network. Once the requirements are met, a trained target detection network can be obtained.

[0022] Constructing sample sets: Collect strawberry images by field shooting to create a set of strawberry data sets; collect strawberry images at different stages of ripening; collect 2,000 strawberry images with an image resolution of 1,600 pixels × 1,600 pixels, expand the training set by enhancing and weakening brightness, chromaticity, contrast and sharpness, and geometric transformation, and then use roLabelImg to annotate strawberry images. The data to be annotated include ripe strawberries and ripe strawberry stalks. After the annotation is completed, the generated xml file is converted into a txt file required for rotation target detection through python, and then the txt file required for rotation target detection is converted into a txt file in the PASCAL VOC data set format required by YOLO through python for storage. Finally, the strawberry image data set is divided into training set, validation set and test set according to 8:1:1. Since the sample set is annotated with ripe strawberries, the output is also a ripe strawberry detection frame during subsequent detection.

[0023] Model training: Input the training set into the target detection network for training. During training, parameters can be set according to actual conditions.

[0024] Model evaluation: After the model training is completed, for example, it can be evaluated using precision, recall, average precision (AP) and mAP@0.5. Precision indicates the probability that the detection area contains true samples, Recall indicates how many targets are detected in the detection image, AP measures how well the model learns in each category, and mAP measures how well the trained model learns in all categories. It is the average of AP over all categories. The specific calculation formula is as follows.

[0025] , , , , , Among them, TP represents the number of correct targets predicted by the model, FP represents the number of samples that identify non-targets as targets, and FN represents the number of samples that identify targets as non-targets; p(r) is the precision of recall measurement, which is the PR curve formed by Precision and Recall; N is the number of target categories, AP i is the AP of the i-th category; mAP@0.5 represents the average accuracy of target detection with a confidence level of 0.5. In addition, Param and FLOPs can be used to evaluate the performance of target detection models. Param usually refers to the learnable weights and biases in the network, and its number can be used to evaluate the complexity and storage requirements of the model; FLOPs refers to the number of floating-point operations, which is used to evaluate the computational complexity and operating efficiency of the model.

[0026] In specific applications, two target detection networks can be trained, one for detecting fruits and the other for detecting fruit stems.

[0027] It is also possible to train a target detection network that can detect fruits and fruit stems. When this solution is adopted, since the target detection network outputs multiple target detection frames, generally no distinction is made between fruits and fruit stems. In the present invention, the aspect ratio of the detection frame can be used to distinguish whether it is a fruit detection frame or a fruit stem detection frame. Generally speaking, fruit stem detection frames are mostly slender, so a threshold can be set. When the aspect ratio of the detection frame is greater than the threshold, the detection frame is a fruit stem detection frame, and when the aspect ratio of the detection frame is less than the threshold, the detection frame is a fruit detection frame.

[0028] Step S300, matching the fruit detection frame and the fruit stem detection frame to obtain matching data. After the fruit detection frame and the fruit stem detection frame are identified, the two can be matched to ensure that the fruit detection frame and the fruit stem detection frame can accurately correspond to each other, which is conducive to the subsequent picking point positioning.

[0029] See also Figure 2 In a specific embodiment of the present invention, step S300 includes step S310 and step S320.

[0030] Step S310: Obtain an association matrix based on the fruit detection frame and the fruit stem detection frame, where i and j are not necessarily equal. Assuming there are M fruit detection frames and N fruit stem detection frames, the size of the association matrix is ​​M×N, where i is 1, 2, ..., M, and j is 1, 2, ..., N. In the association matrix, the element in the i-th row and j-th column is the displacement between the center point of the upper edge of the i-th fruit detection frame and the center point of the lower edge of the j-th fruit stem detection frame.

[0031] See also Figure 3 The fruit detection frame of the strawberry is expressed as B0(x00, y00), B1(x01, y01), B2(x02, y02), B3(x03, y03), and the stalk detection frame of the strawberry is expressed as F0(x10, y10), F1(x11, y11), F2(x12, y12), F3(x13, y13). The coordinates of the center point of the upper edge of the fruit detection frame are CB23(x0, y0), and the coordinates of the center point of the lower edge of the stalk detection block are CF01(x1, y1). The displacement between the two points can be obtained through the coordinates of point CB23 and point CF01.

[0032] In a specific embodiment of the present invention, in the association matrix, when the center point of the lower edge of the fruit stem detection frame is above the center point of the upper edge of the fruit detection frame, the displacement between the two is recorded as a positive value, otherwise, the displacement between the two is recorded as a negative value. In this case, the displacement between the two center points may be positive or negative. The distance mentioned below, i.e., the absolute value of the displacement, is a value greater than or equal to 0.

[0033] Step S320: Match the fruit detection frame and the fruit stem detection frame according to the association matrix to obtain matching data. This step is to match the fruit detection frame and the fruit stem detection frame so that the fruit and its corresponding fruit stem are matched to facilitate the subsequent picking point positioning.

[0034] See also Figure 4 In a specific embodiment of the present invention, step S320 includes steps S321 to S324.

[0035] Step S321, find the element whose absolute value of displacement in the association matrix is ​​less than the preset minimum distance, match the fruit detection frame and the fruit stem detection frame corresponding to the element, and set all elements of the row and column where the element is located to null values. In this step, the preset minimum distance can be set to 0, or a number slightly larger than 0. When the displacement corresponding to the two detection frames is very small, the two can be considered to be matched. After matching, all elements of the row and column where the element is located are set to null values ​​so that the matched fruit detection frame and fruit stem detection frame are no longer considered in subsequent matching, thereby reducing the workload of subsequent matching.

[0036] Step S322, retain the elements in the association matrix within the preset range, and set all other elements to null values. This preset range corresponds to the distance range between the detection frame of the same fruit and its corresponding fruit stem detection frame in the actual scene, for example, it can be -20~60mm, and, in order to facilitate subsequent matching, if there are many elements in a row or a column within the preset range, only the three elements closest to 0 can be retained. Set all other elements to null values, that is, set the elements with a distance less than -20mm or a distance greater than 60mm to null values, because the fruit stem detection frame and the fruit detection frame that meet these distances are definitely not the same fruit, so they can be directly set to null values ​​to reduce the amount of calculation during subsequent matching.

[0037] Step S323, find an element with only one non-null value in a row or a column in the association matrix, match the fruit detection frame and the fruit stem detection frame corresponding to the element, and set all elements in the row and column where the element is located to null values. For an element with only one non-null value, it means that a fruit detection frame can only match a fruit stem detection frame, or a fruit stem detection frame can only match a fruit detection frame, so these detection frames can be matched. The operation of setting to null values ​​after matching is also to remove the matched rows and columns.

[0038] Step S324, find all non-null elements in each row of the association matrix, and match the fruit detection frame and the fruit stem detection frame corresponding to the element with the smallest absolute value of displacement. After the above steps, most of the fruit detection frames and fruit stem detection frames have been matched, and there are still some relatively complicated situations. At this time, the principle of nearest matching can be adopted for matching, that is, find the element with the smallest absolute value of displacement for matching. When executing this step, it is not necessary to follow the order, but to globally search for the element with the smallest absolute value of displacement.

[0039] Through the above steps S321 to S324, the matching of the fruit detection frame and the fruit stem detection frame can be completed. It should be noted that not every detection frame has another matching detection frame, such as some occlusion phenomena.

[0040] In a specific embodiment of the present invention, the format of the matching data is [F dis,S box ,F box ], where: if the fruit detection frame matches the stem detection frame, S box is the matching fruit detection frame, F box is the matching fruit stem detection frame, F dis For S box and F box The displacement value in the association matrix, for example, the matching result is [displacement between detection frames, fruit detection frame, fruit stem detection frame]. For this matching result, further processing is required to obtain the picking point.

[0041] If the fruit detection frame does not match the stem detection frame, S box is the unmatched fruit detection frame, F box and F dis is a preset mark, which can be -1. For example, the matching result is [-1, fruit detection frame, -1]. In this case, it means that the fruit is detected, but the fruit stem is blocked. This kind of fruit can be marked as manually picked and picked manually later.

[0042] If the stem detection frame does not match the fruit detection frame, F box is the unmatched fruit stem detection frame, S box and F dis is a preset mark, for example, the matching result is [-1, -1, fruit stalk detection frame]. In this case, it generally corresponds to a false detection, that is, the fruit stalk of an immature fruit or other leaf fruit stalk is mistakenly detected, and the fruit stalk detection frame can be directly deleted.

[0043] Step S400, obtaining the fruit picking point according to the matching data. After obtaining the matching data (mainly the matched fruit detection frame and the fruit stem detection frame, and the unmatched ones can be ignored), further processing is required to obtain the fruit picking point.

[0044] See also Figure 5 In a specific embodiment of the present invention, step S400 specifically includes: for the matched fruit detection frame and fruit stem detection frame, determining whether the displacement between the two is a negative value: if yes, taking the intersection of the median line of the fruit stem detection frame and the upper edge of the fruit detection frame as the picking point; if no, taking the center point of the lower edge of the fruit stem detection frame as the picking point. Figure 5 It can be seen that the two strawberries on the left are picked at the intersection point, the third strawberry is picked at the center point of the lower edge of the stalk detection frame, and the stalk of the fourth strawberry is not detected, so there is no corresponding stalk detection frame and it can be marked as manually picked.

[0045] In general, the method of the present invention detects only ripe strawberries and the stalks of ripe strawberries by using a target detection network, and adopts a custom stalk picking point positioning method. First, the affiliation between the stalk and the fruit is determined (i.e., matching is performed). If the affiliation is satisfied, the optimal picking point position of the stalk corresponding to the ripe fruit is calculated, and then the corresponding coordinates are input into the picking robot for reading and picking. The present invention can achieve accurate picking point positioning between strawberries of different forms, which can not only ensure the accuracy and efficiency of detection, but also deal with the problems of missed detection and wrong detection in the picking process, improve the picking efficiency and the fruit intact rate, and mark the data set by rotating the frame, so as to effectively reduce the detection of blank areas, achieve more accurate positioning and separation of targets, and take into account robustness and real-time performance as a whole.

[0046] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for locating a fruit picking point, characterized in that: include: Get the image to be detected; Input the image to be detected into the trained object detection network to obtain a fruit detection frame and a fruit stem detection frame of the image to be detected; Matching the fruit detection frame and the fruit stem detection frame to obtain matching data; The picking point of the fruit is obtained according to the matching data.

2. The fruit picking point positioning method according to claim 1, characterized in that: Matching the fruit detection frame and the fruit stem detection frame to obtain matching data includes: According to the fruit detection frame and the fruit stalk detection frame, an association matrix is ​​obtained, in which the element of the i-th row and the j-th column is the displacement between the center point of the upper edge of the i-th fruit detection frame and the center point of the lower edge of the j-th fruit stalk detection frame; According to the association matrix, the fruit detection frame and the fruit stem detection frame are matched to obtain matching data.

3. The fruit picking point positioning method according to claim 1, characterized in that: According to the association matrix, matching the fruit detection frame and the fruit stem detection frame includes: Find the element whose absolute value of displacement in the association matrix is ​​less than the preset minimum distance, match the fruit detection frame and the fruit stem detection frame corresponding to the element, and set all elements in the row and column where the element is located to null values; retaining the elements in the association matrix whose displacements are within a preset range, and setting all other elements to null values; Find an element in a row or a column in the association matrix that has only one non-null value, match the fruit detection frame and the fruit stem detection frame corresponding to the element, and set all elements in the row and column where the element is located to null values; Find all non-null value elements in each row of the association matrix, and match the fruit detection frame and the fruit stem detection frame corresponding to the element with the smallest absolute value of displacement.

4. The fruit picking point positioning method according to claim 3, characterized in that: In the step of retaining the elements in the association matrix whose displacement is within a preset range, the preset range is -20 to 60 mm.

5. The fruit picking point positioning method according to claim 2, characterized in that: The format of the matching data is [F dis ,S box ,F box ],in: If the fruit detection frame matches the stem detection frame, S box is the matching fruit detection frame, F box is the matching fruit stem detection frame, F dis For S box and F box displacement values ​​in the incidence matrix; If the fruit detection frame does not match the stem detection frame, S box is the unmatched fruit detection frame, F box and F dis is a preset identifier; If the stem detection frame does not match the fruit detection frame, F box is the unmatched fruit stem detection frame, S box and F dis The preset identifier.

6. The method for locating a fruit picking point according to claim 5, characterized in that: In the association matrix, when the center point of the lower edge of the fruit stalk detection frame is above the center point of the upper edge of the fruit detection frame, the displacement between the two is recorded as a positive value, otherwise, the displacement between the two is recorded as a negative value; According to the matching data, the picking point of the fruit is obtained, including: For the matched fruit detection frame and stalk detection frame, determine whether the displacement between the two is a negative value: If yes, the intersection of the median line of the fruit stalk detection frame and the upper edge of the fruit detection frame is used as the picking point; If not, the center point of the lower edge of the fruit stalk detection frame is used as the picking point.

Images