A fruit picking point positioning method based on improved deep learning network

Abstract

The present invention provides a method for locating a fruit picking point based on an improved deep learning network, comprising: obtaining a picture to be detected; inputting the picture to be detected into a trained improved YOLOv8 detection network to obtain a fruit detection frame and a fruit stem detection frame of the picture to be detected; matching the fruit detection frame and the fruit stem detection frame to obtain matching data; and obtaining a fruit picking point based on the matching data. The method detects the fruit detection frame and the fruit stem detection frame more accurately through the improved YOLOv8 detection network, and then matches the two detection frames, thereby determining the corresponding relationship between the fruit and the fruit stem, and finally calculating the fruit picking point based on the matching data, so that the obtained picking point will be more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a fruit picking point positioning method based on an improved deep learning network. Background Art

[0002] With the booming development of the strawberry planting industry, the realization of unmanned and automated strawberry picking has become a key area of ​​intelligent agricultural research. At present, intelligent mobile automatic picking robots mainly use three mainstream technical methods to locate strawberry picking points. The first method is to use visual systems and machine learning algorithms, such as support vector machines (SVM) and relevance vector machines (RVM), to identify strawberry fruits in images, and combine manual experience to determine the location of the fruit stems and their picking points. Although this method has a fast recognition speed, it does not accurately identify the fruit stems, resulting in insufficient positioning accuracy in complex scenes, which may cause fruit damage. The second method is to focus on identifying the fruit stems and calculating their center of gravity to determine the picking points. Although it improves the accuracy of fruit stem recognition, it may mistakenly pick immature fruits and cause resource waste. The third method combines detection and segmentation technology to accurately identify strawberry fruits and fruit stems, and then perform morphological processing to determine the picking points. Although this method performs well in recognition accuracy, it involves complex algorithms, has high requirements for computing resources, and may affect real-time performance. Strawberries belong to the berry fruit. Due to the soft nature of the fruit, they are usually picked by gripper or cutting robots. Strawberries have diverse shapes, and the recognition result of the classification algorithm is the smallest rectangle of the area where the fruit or stalk is located. Gripper robots cannot confirm the specific outlines of the fruit and stalk, which can easily cause damage to the fruit during the picking process. The method of locating the stalk and picking point based on manual experience is not universal. Cutting robots often make empty cuts or cut the wrong stalks of other fruits, which greatly wastes the ability of the picking robot and fruit resources, and cannot improve the efficiency of farm picking. Summary of the invention

[0003] In view of the above defects of the prior art, the present invention provides a fruit picking point positioning method based on an improved deep learning network to solve the technical problem of inaccurate positioning of fruit picking points.

[0004] To achieve the above-mentioned purpose and other related purposes, the present invention provides a fruit picking point positioning method based on an improved deep learning network, comprising: obtaining a picture to be detected; inputting the picture to be detected into a trained improved YOLOv8 detection network to obtain a fruit detection frame and a fruit stalk detection frame of the picture to be detected; matching the fruit detection frame and the fruit stalk detection frame to obtain matching data; obtaining the fruit picking point according to the matching data; the improved YOLOv8 detection network is constructed by the following steps: constructing a YOLOv8 detection network, in which an OBB module is provided for obtaining a rotated detection frame; replacing at least one C2f module in the YOLOv8 detection network with a FasterECNet module, and adding an RFBANet module before the OBB module to obtain the improved YOLOv8 detection network.

[0005] In one embodiment of the present invention, replacing at least one C2f module in the YOLOv8 detection network with a FasterECNet module includes: replacing the last three C2f modules in the Backbone of the YOLOv8 detection network with a FasterECNet module.

[0006] In one embodiment of the present invention, the FasterECNet module processes the input feature x1 according to the following formula to obtain the output feature y1: y1=f(x1)+f(Conv 1×1 (Relu(BN(Conv 1×1 (PConv 3×3 (x1)))))), where f() represents the processing process of the EMA unit or the EMLA unit.

[0007] In one embodiment of the present invention, the EMLA unit is obtained by the following steps: a 3×1 convolution and a 1×3 convolution are used to replace the 3×3 convolution in the EMA unit to obtain the EMLA unit.

[0008] In one embodiment of the present invention, the RFBANet module is improved by the RFB module according to the following steps: the 5×5 convolution in the RFB module is replaced by cascaded 1×3 convolution, 3×1 convolution and 3×3 convolution.

[0009] In one embodiment of the present invention, the fruit detection frame and the fruit stalk detection frame are matched to obtain matching data, including: obtaining an association matrix based on the fruit detection frame and the fruit stalk detection frame, in which the elements in the i-th row and the j-th column are the displacements between the center point of the upper edge of the i-th fruit detection frame and the center point of the lower edge of the j-th fruit stalk detection frame; according to the association matrix, the fruit detection frame and the fruit stalk detection frame are matched to obtain matching data.

[0010] In one embodiment of the present invention, the fruit detection frame and the fruit stalk detection frame are matched according to the association matrix, including: finding an element in the association matrix whose absolute value of displacement is less than a preset minimum distance, matching the fruit detection frame and the fruit stalk detection frame corresponding to the element, and setting all elements in the row and column where the element is located to null values; retaining the elements in the association matrix whose displacement is within a preset range, and setting all other elements to null values; finding an element in a row or a column of the association matrix that has only one non-null value, matching the fruit detection frame and the fruit stalk detection frame corresponding to the element, and setting all elements in the row and column where the element is located to null values; finding all non-null value elements in each row of the association matrix, and matching the fruit detection frame and the fruit stalk detection frame corresponding to the element with the smallest absolute value of displacement.

[0011] In one embodiment of the present invention, the format of the matching data is [F dis ,S box ,F box ], where: if the fruit detection frame matches the stem detection frame, S box is the matching fruit detection frame, F box is the matching fruit stem detection frame, F dis For S box and F box The displacement value in the association matrix; if the fruit detection frame does not match the stalk detection frame, S box is the unmatched fruit detection frame, F box and F dis is the preset mark; if the fruit stem detection frame does not match the fruit detection frame, F box is the unmatched fruit stem detection frame, S box and F dis The preset identifier.

[0012] In one embodiment of the present invention, in the association matrix, when the center point of the lower edge of the fruit stalk detection frame is above the center point of the upper edge of the fruit detection frame, the displacement between the two is recorded as a positive value, otherwise, the displacement between the two is recorded as a negative value; based on the matching data, the fruit picking point is obtained, including: for the matched fruit detection frame and fruit stalk detection frame, judging whether the displacement between the two is a negative value: if yes, taking the intersection of the median line of the fruit stalk detection frame and the upper edge of the fruit detection frame as the picking point; if not, taking the center point of the lower edge of the fruit stalk detection frame as the picking point.

[0013] Beneficial effects of the present invention: The present invention proposes a fruit picking point positioning method based on an improved deep learning network. The method detects the fruit detection frame and the fruit stalk detection frame more accurately through the improved YOLOv8 detection network, and then matches the two detection frames to determine the correspondence between the fruit and the fruit stalk. Finally, the fruit picking point is calculated based on the matching data, so that the picking point obtained will be more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 A general flow chart provided for an embodiment of the present invention;

[0016] Figure 2 A structural diagram of a YOLOv8 detection network provided by an embodiment of the present invention;

[0017] Figure 3 A structural diagram of an improved YOLOv8 detection network provided by an embodiment of the present invention;

[0018] Figure 4 A structural diagram of a FasterECNet module provided in one embodiment of the present invention;

[0019] Figure 5 A structural diagram of an EMLA unit provided in one embodiment of the present invention;

[0020] Figure 6 A structural diagram of an RFBANet module provided in one embodiment of the present invention;

[0021] Figure 7 A flowchart of obtaining matching data provided by an embodiment of the present invention;

[0022] Figure 8 A schematic diagram of the displacement of the center point of a detection frame provided by an embodiment of the present invention;

[0023] Fig. 9 A detection frame matching flow chart provided by an embodiment of the present invention;

[0024] Fig.10 A schematic diagram of multiple strawberry picking points provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following describes the embodiments of the present invention through specific specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. It should be noted that the following embodiments and the features in the embodiments can be combined with each other without conflict. In addition to the specific methods, equipment, and materials used in the embodiments, according to the technical personnel in this technical field's grasp of the prior art and the records of the present invention, any methods, equipment, and materials of the prior art that are similar or equivalent to the methods, equipment, and materials described in the embodiments of the present invention can also be used to implement the present invention.

[0026] It should be understood that the terms used in the embodiments of the present invention are for describing specific embodiments rather than for limiting the scope of protection of the present invention. Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those generally understood by those skilled in the art.

[0027] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In some of the embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0028] The flowchart and block diagram in the accompanying drawings illustrate the system architecture, functions and operations that may be implemented according to the methods and computer program products of various embodiments disclosed in the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, the program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0029] It should be noted that the fruits in the present invention include all fruits with stalks, such as strawberries, raspberries, apples, pears, oranges, etc. In the following specific embodiments, strawberries are used as an example for explanation, but the invention is also applicable to other fruits.

[0030] See also Figure 1 , Figure 1 A fruit picking point positioning method based on an improved deep learning network is provided in one embodiment of the present invention, comprising steps S100 to S400.

[0031] Step S100, obtaining a picture to be detected. The picture to be detected is a picture taken by the picking machine. After the picture to be detected is processed in subsequent steps to obtain a picking point, the picking machine picks the fruit according to the identified picking point.

[0032] Step S200: input the image to be detected into the trained improved YOLOv8 detection network to obtain the fruit detection frame and the fruit stalk detection frame of the image to be detected.

[0033] See also Figure 2 , which shows the structure of the existing YOLOv8 detection network. The YOLOv8 detection network is a SOTA model that builds on the success of previous YOLO versions and introduces new features and improvements to further improve performance and flexibility. Specific innovations include a new backbone network, a new Ancher-Free detection head, and a new loss function that can run on a variety of hardware platforms from CPU to GPU. As a cutting-edge, state-of-the-art (SOTA) model, YOLOv8 builds on the success of previous versions and introduces new features and improvements to enhance performance, flexibility, and efficiency.

[0034] In the present invention, the YOLOv8 detection network is improved to obtain an improved YOLOv8 detection network, so as to facilitate the simultaneous detection of fruits and fruit stems. A specific embodiment will be provided below to illustrate the improved YOLOv8 detection network. After the improved YOLOv8 detection network is obtained, a trained improved YOLOv8 detection network can be obtained by training it. The training process is generally divided into constructing a sample set, training the improved YOLOv8 detection network using the sample set, and evaluating and testing the trained improved YOLOv8 detection network. After meeting the requirements, a trained improved YOLOv8 detection network can be obtained.

[0035] Constructing sample sets: Collect strawberry images by field shooting to create a set of strawberry data sets; collect strawberry images at different stages of ripening; collect 2,000 strawberry images with an image resolution of 1,600 pixels × 1,600 pixels, expand the training set by enhancing and weakening brightness, chromaticity, contrast and sharpness, and geometric transformation, and then use roLabelImg to annotate strawberry images. The data to be annotated include ripe strawberries and ripe strawberry stalks. After the annotation is completed, the generated xml file is converted into a txt file required for rotation target detection through python, and then the txt file required for rotation target detection is converted into a txt file in the PASCAL VOC data set format required by YOLO through python for storage. Finally, the strawberry image data set is divided into training set, validation set and test set according to 8:1:1. Since the sample set is annotated with ripe strawberries, the output is also a ripe strawberry detection frame during subsequent detection.

[0036] Model training: The training set is input into the improved YOLOv8 detection network for training. During training, parameters can be set according to actual conditions. For example, the model training parameters are set as follows: learning rate = 0.001, momentum = 0.9, decay = 0.0005, batch size = 4, rounds = 200.

[0037] Model evaluation: After the model training is completed, for example, it can be evaluated using precision, recall, average precision (AP) and mAP@0.5. Precision indicates the probability that the detection area contains true samples, Recall indicates how many targets are detected in the detection image, AP measures how well the model learns in each category, and mAP measures how well the trained model learns in all categories. It is the average of AP over all categories. The specific calculation formula is as follows.

[0038] ,

[0039] ,

[0040] ,

[0041] ,

[0042] ,

[0043] Among them, TP represents the number of correct targets predicted by the model, FP represents the number of samples that identify non-targets as targets, and FN represents the number of samples that identify targets as non-targets; p(r) is the precision of recall measurement, which is the PR curve formed by Precision and Recall; N is the number of target categories, AP i is the AP of the i-th category; mAP@0.5 represents the average accuracy of target detection with a confidence level of 0.5. In addition, Param and FLOPs can be used to evaluate the performance of target detection models. Param usually refers to the learnable weights and biases in the network, and its number can be used to evaluate the complexity and storage requirements of the model; FLOPs refers to the number of floating-point operations, which is used to evaluate the computational complexity and operating efficiency of the model.

[0044] Regarding improving the YOLOv8 detection network, the goal of the present invention is to provide a detection network that can accurately identify fruits and stalks. Some specific implementation methods are provided below for reference.

[0045] See also Figure 3 In a specific embodiment of the present invention, an improved YOLOv8 detection network is constructed by the following steps: constructing a YOLOv8 detection network, in which an OBB module is provided for obtaining a rotated detection frame; replacing at least one C2f module in the YOLOv8 detection network with a FasterECNet module, and adding an RFBANet module before the OBB module to obtain an improved YOLOv8 detection network.

[0046] In the YOLOv8 detection network, detect refers to the task module for target detection, and OBB (Oriented Bounding Box) refers to rotation target detection. As one of the core modules of the YOLOv8 detection network, Detect is mainly responsible for target detection tasks. Specifically, it processes the input image layer by layer and finally obtains a feature map for predicting the target position and category. The network structure of this part includes the backbone network, the Neck layer and the Head layer. OBB (Oriented Bounding Box) is a special method for rotating object detection. It can better handle directional objects. In YOLOv8, OBB detection is achieved by adding a feature map for predicting the rotation angle, which greatly improves the detection effect of such targets. In the present invention, the fruits and fruit stalks to be detected are often tilted and have a certain rotation angle. In order to better achieve positioning, the OBBOBB module is used to obtain the rotating fruit detection frame and fruit stalk detection frame.

[0047] The original C2f module in the detection network is replaced with the FasterECNet module. This structure is a lightweight, general, low-latency end-side network architecture that can effectively reduce computational redundancy and the number of memory accesses. It can also refine the model features and improve the ability to extract features, so that the detection focuses more on ripe strawberries and their corresponding stalks, and suppresses the interference of background factors such as ripe strawberries and strawberry leaves. RFBANet is added after the original Neck network C2f, and deformable convolution and lightweight backbone networks are used to better extract image features and highly discriminative features, making the detector faster and more accurate.

[0048] In a specific embodiment of the present invention, at least one C2f module in the YOLOv8 detection network is replaced with a FasterECNet module, including: replacing the last three C2f modules in the Backbone of the YOLOv8 detection network with the FasterECNet module. Theoretically, any number of C2f modules in the YOLOv8 detection network can be replaced by the FasterECNet module, but in this embodiment, only the last three C2f modules in the Backbone are replaced, and the detection effect of the detection network obtained thereby is the best. Figure 3 This replacement method is also shown in FIG.

[0049] See also Figure 4 In a specific embodiment of the present invention, the FasterECNet module processes the input feature x1 according to the following formula to obtain the output feature y1: y1=f(x1)+f(Conv 1×1 (Relu(BN(Conv 1×1 (PConv 3×3(x1)))))), where f() represents the processing process of the EMA unit or the EMLA unit.

[0050] In this embodiment, Conv 1×1 PConv represents a normal convolution with a convolution kernel size of 1×1; 3×3 It represents partial convolution with a convolution kernel size of 3×3. The core idea of ​​partial convolution is that it does not perform a uniform convolution operation on all input data, but dynamically adjusts the area of ​​action of the convolution kernel according to the validity of the data (i.e. whether the data is missing). BN is a batch normalization layer, and Relu is a nonlinear activation function. Convolution layer + BN + Relu is a very common standard block structure. The convolution layer is responsible for extracting features, and the BN layer stabilizes training and accelerates convergence by normalizing these features. The Relu layer introduces nonlinearity, enabling the network to learn more complex data representations.

[0051] In this embodiment, the EMA unit is a key module in the YOLOv8 network for improving target detection performance. Its full name is the Efficient Multi-Scale Attention Module, which contains two parallel branches, one branch uses a 1×1 convolution kernel to extract local features, and the other branch uses a 3×3 convolution kernel to capture a wider range of spatial information. This multi-scale structure can not only effectively establish short-range and long-range dependencies, but also fuse these two features through cross-space learning methods to improve the detection ability of targets of different sizes. The EMLA unit is obtained by improving the EMA unit. In this embodiment, the EMA unit can be directly used to form the FasterECNet module; the EMA unit can also be improved to obtain the EMLA unit (which will be introduced in detail in the next paragraph), and then the EMLA unit is used to form the FasterECNet module.

[0052] See also Figure 5 In a specific embodiment of the present invention, the EMLA unit is obtained by the following steps: a 3×1 convolution and a 1×3 convolution are used to replace the 3×3 convolution in the EMA unit to obtain the EMLA unit. The specific structure of the EMLA unit is as follows: Figure 5 As shown, the only difference between it and the EMA unit is the rightmost branch.

[0053] See also Figure 6In a specific embodiment of the present invention, the RFBANet module is obtained by improving the RFB module according to the following steps: the 5×5 convolution in the RFB module is replaced by cascaded 1×3 convolution, 3×1 convolution and 3×3 convolution. In the YOLOv8 network, the RFB module (Receptive Field Block) is a multi-branch convolution block that aims to enhance the feature representation ability of the lightweight CNN model by simulating the size and eccentricity of the receptive field in the human visual system. It is used to improve the features extracted from the lightweight backbone network to improve the accuracy and speed of the detector. The module includes multiple branches, each branch contains a different convolution kernel size and expansion rate, and finally the feature maps of all branches are connected in series to form a spatial pooling or convolution array to generate the final feature representation. In this embodiment, on the basis of the original RFB module, some of the layers therein are replaced to obtain the RFBANet module. The structure diagram of the RFBANet module is shown as follows. Figure 6 shown.

[0054] By replacing the above modules, an improved YOLOv8 detection network will be obtained. Combined with the above training method, the improved YOLOv8 detection network can be trained. After training, when subsequently recognizing the image to be detected, you only need to input the image to be detected into the trained improved YOLOv8 detection network to obtain the corresponding fruit detection frame and fruit stem detection frame.

[0055] By improving the YOLOv8 detection network, it is possible to detect ripe fruits and their stalks at the same time, ensuring the accuracy and efficiency of detection, while traditional methods usually only identify fruits or stalks, resulting in low positioning accuracy. The existing technology has difficulties in accurately confirming the specific contours of fruits and stalks, so the present invention uses a rotating frame to mark the data set, reducing the detection of blank areas, achieving more accurate target positioning and separation, and the improved model can also adapt to the morphological changes of fruits at different angles and postures, enhancing the accuracy of recognition, while traditional methods perform poorly when dealing with diverse fruit morphologies.

[0056] Understandably, the improved YOLOv8 detection network outputs a target detection frame, and generally does not distinguish between fruits and fruit stems. In the present invention, the aspect ratio of the detection frame can be used to distinguish whether it is a fruit detection frame or a fruit stem detection frame. Generally speaking, fruit stem detection frames are mostly slender, so a threshold can be set. When the aspect ratio of the detection frame is greater than the threshold, the detection frame is a fruit stem detection frame, and when the aspect ratio of the detection frame is less than the threshold, the detection frame is a fruit detection frame.

[0057] Step S300, matching the fruit detection frame and the fruit stem detection frame to obtain matching data. After the fruit detection frame and the fruit stem detection frame are identified, the two can be matched to ensure that the fruit detection frame and the fruit stem detection frame can accurately correspond to each other, which is conducive to the subsequent picking point positioning.

[0058] See also Figure 7 In a specific embodiment of the present invention, step S300 includes step S310 and step S320.

[0059] Step S310: Obtain an association matrix based on the fruit detection frame and the fruit stem detection frame, where i and j are not necessarily equal. Assuming there are M fruit detection frames and N fruit stem detection frames, the size of the association matrix is ​​M×N, where i is 1, 2, ..., M, and j is 1, 2, ..., N. In the association matrix, the element in the i-th row and j-th column is the displacement between the center point of the upper edge of the i-th fruit detection frame and the center point of the lower edge of the j-th fruit stem detection frame.

[0060] See also Figure 8 The fruit detection frame of the strawberry is expressed as B0(x00, y00), B1(x01, y01), B2(x02, y02), B3(x03, y03), and the stalk detection frame of the strawberry is expressed as F0(x10, y10), F1(x11, y11), F2(x12, y12), F3(x13, y13). The coordinates of the center point of the upper edge of the fruit detection frame are CB23(x0, y0), and the coordinates of the center point of the lower edge of the stalk detection block are CF01(x1, y1). The displacement between the two points can be obtained through the coordinates of point CB23 and point CF01.

[0061] In a specific embodiment of the present invention, in the association matrix, when the center point of the lower edge of the fruit stem detection frame is above the center point of the upper edge of the fruit detection frame, the displacement between the two is recorded as a positive value, otherwise, the displacement between the two is recorded as a negative value. In this case, the displacement between the two center points may be positive or negative. The distance mentioned below, i.e., the absolute value of the displacement, is a value greater than or equal to 0.

[0062] Step S320: Match the fruit detection frame and the fruit stem detection frame according to the association matrix to obtain matching data. This step is to match the fruit detection frame and the fruit stem detection frame so that the fruit and its corresponding fruit stem are matched to facilitate the subsequent picking point positioning.

[0063] See also Fig. 9 In a specific embodiment of the present invention, step S320 includes steps S321 to S324.

[0064] Step S321, find the element whose absolute value of displacement in the association matrix is ​​less than the preset minimum distance, match the fruit detection frame and the fruit stem detection frame corresponding to the element, and set all elements of the row and column where the element is located to null values. In this step, the preset minimum distance can be set to 0, or a number slightly larger than 0. When the displacement corresponding to the two detection frames is very small, the two can be considered to be matched. After matching, all elements of the row and column where the element is located are set to null values ​​so that the matched fruit detection frame and fruit stem detection frame are no longer considered in subsequent matching, thereby reducing the workload of subsequent matching.

[0065] Step S322, retain the elements in the association matrix within the preset range, and set all other elements to null values. This preset range corresponds to the distance range between the detection frame of the same fruit and its corresponding fruit stem detection frame in the actual scene, for example, it can be -20~60mm, and, in order to facilitate subsequent matching, if there are many elements in a row or a column within the preset range, only the three elements closest to 0 can be retained. Set all other elements to null values, that is, set the elements with a distance less than -20mm or a distance greater than 60mm to null values, because the fruit stem detection frame and the fruit detection frame that meet these distances are definitely not the same fruit, so they can be directly set to null values ​​to reduce the amount of calculation during subsequent matching.

[0066] Step S323, find an element with only one non-null value in a row or a column in the association matrix, match the fruit detection frame and the fruit stem detection frame corresponding to the element, and set all elements in the row and column where the element is located to null values. For an element with only one non-null value, it means that a fruit detection frame can only match a fruit stem detection frame, or a fruit stem detection frame can only match a fruit detection frame, so these detection frames can be matched. The operation of setting to null values ​​after matching is also to remove the matched rows and columns.

[0067] Step S324, find all non-null elements in each row of the association matrix, and match the fruit detection frame and the fruit stem detection frame corresponding to the element with the smallest absolute value of displacement. After the above steps, most of the fruit detection frames and fruit stem detection frames have been matched, and there are still some relatively complicated situations. At this time, the principle of nearest matching can be adopted for matching, that is, find the element with the smallest absolute value of displacement for matching. When executing this step, it is not necessary to follow the order, but to globally search for the element with the smallest absolute value of displacement.

[0068] Through the above steps S321 to S324, the matching of the fruit detection frame and the fruit stem detection frame can be completed. It should be noted that not every detection frame has another matching detection frame, such as some occlusion phenomena.

[0069] In a specific embodiment of the present invention, the format of the matching data is [F dis,S box ,F box ], where: if the fruit detection frame matches the stem detection frame, S box is the matching fruit detection frame, F box is the matching fruit stem detection frame, F dis For S box and F box The displacement value in the association matrix, for example, the matching result is [displacement between detection frames, fruit detection frame, fruit stem detection frame]. For this matching result, further processing is required to obtain the picking point.

[0070] If the fruit detection frame does not match the stem detection frame, S box is the unmatched fruit detection frame, F box and F dis is a preset mark, which can be -1. For example, the matching result is [-1, fruit detection frame, -1]. In this case, it means that the fruit is detected, but the fruit stem is blocked. This kind of fruit can be marked as manually picked and picked manually later.

[0071] If the stem detection frame does not match the fruit detection frame, F box is the unmatched fruit stem detection frame, S box and F dis is a preset mark, for example, the matching result is [-1, -1, fruit stalk detection frame]. In this case, it generally corresponds to a false detection, that is, the fruit stalk of an immature fruit or other leaf fruit stalk is mistakenly detected, and the fruit stalk detection frame can be directly deleted.

[0072] Step S400, obtaining the fruit picking point according to the matching data. After obtaining the matching data (mainly the matched fruit detection frame and the fruit stem detection frame, and the unmatched ones can be ignored), further processing is required to obtain the fruit picking point.

[0073] See also Fig.10 In a specific embodiment of the present invention, step S400 specifically includes: for the matched fruit detection frame and fruit stem detection frame, determining whether the displacement between the two is a negative value: if yes, taking the intersection of the median line of the fruit stem detection frame and the upper edge of the fruit detection frame as the picking point; if no, taking the center point of the lower edge of the fruit stem detection frame as the picking point. Fig.10 It can be seen that the two strawberries on the left are picked at the intersection point, the third strawberry is picked at the center point of the lower edge of the stalk detection frame, and the stalk of the fourth strawberry is not detected, so there is no corresponding stalk detection frame and it can be marked as manually picked.

[0074] In general, the method of the present invention detects only ripe strawberries and the stalks of ripe strawberries by using an improved YOLOv8 detection network, and adopts a custom stalk picking point positioning method. First, the affiliation between the stalk and the fruit is determined (i.e., matching is performed). If the affiliation is satisfied, the optimal picking point position of the stalk corresponding to the ripe fruit is calculated, and then the corresponding coordinates are input into the picking robot for reading and picking. The present invention can achieve accurate picking point positioning between strawberries of different forms, which can not only ensure the accuracy and efficiency of detection, but also deal with the problems of missed detection and wrong detection in the picking process, improve the picking efficiency and the intact rate of the fruit, and mark the data set by rotating the frame, so as to effectively reduce the detection of blank areas, achieve more accurate positioning and separation targets, and take into account robustness and real-time performance as a whole.

[0075] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A fruit picking point positioning method based on an improved deep learning network, It is characterized in that include: Get the image to be detected; Input the image to be detected into the trained improved YOLOv8 detection network to obtain the fruit detection frame and the fruit stem detection frame of the image to be detected; Matching the fruit detection frame and the fruit stem detection frame to obtain matching data; According to the matching data, obtaining the picking point of the fruit; The improved YOLOv8 detection network is constructed by the following steps: Constructing a YOLOv8 detection network, wherein the YOLOv8 detection network is provided with an OBB module for obtaining a rotated detection frame; At least one C2f module in the YOLOv8 detection network is replaced with a FasterECNet module, and an RFBANet module is added before the OBB module to obtain the improved YOLOv8 detection network.

2. According to claim 1, the fruit picking point positioning method based on the improved deep learning network, It is characterized in that Replacing at least one C2f module in the YOLOv8 detection network with a FasterECNet module includes: The FasterECNet module is used to replace the last three C2f modules in the Backbone of the YOLOv8 detection network.

3. According to the fruit picking point positioning method based on the improved deep learning network according to claim 1, It is characterized in that The FasterECNet module takes the input feature x 1 The output feature y is obtained by processing according to the following formula 1 :y 1 =f(x 1 )+f(Conv 1×1 (Relu(BN(Conv 1×1 (PConv 3×3 (x 1 ))))))), where f() represents the processing process of the EMA unit or the EMLA unit.

4. The fruit picking point positioning method based on the improved deep learning network according to claim 3, It is characterized in that The EMLA unit is obtained by the following steps: replacing the 3×3 convolution in the EMA unit with a 3×1 convolution and a 1×3 convolution to obtain the EMLA unit.

5. The fruit picking point positioning method based on the improved deep learning network according to claim 1, It is characterized in that The RFBANet module is improved by the RFB module according to the following steps: the 5×5 convolution in the RFB module is replaced by cascaded 1×3 convolution, 3×1 convolution and 3×3 convolution.

6. The fruit picking point positioning method based on the improved deep learning network according to claim 1, It is characterized in that Matching the fruit detection frame and the fruit stem detection frame to obtain matching data includes: According to the fruit detection frame and the fruit stalk detection frame, an association matrix is ​​obtained, in which the element of the i-th row and the j-th column is the displacement between the center point of the upper edge of the i-th fruit detection frame and the center point of the lower edge of the j-th fruit stalk detection frame; According to the association matrix, the fruit detection frame and the fruit stem detection frame are matched to obtain matching data.

7. The fruit picking point positioning method based on the improved deep learning network according to claim 6, It is characterized in that According to the association matrix, matching the fruit detection frame and the fruit stem detection frame includes: Find the element whose absolute value of displacement in the association matrix is ​​less than the preset minimum distance, match the fruit detection frame and the fruit stem detection frame corresponding to the element, and set all elements in the row and column where the element is located to null values; retaining the elements in the association matrix whose displacements are within a preset range, and setting all other elements to null values; Find an element in a row or a column in the association matrix that has only one non-null value, match the fruit detection frame and the fruit stem detection frame corresponding to the element, and set all elements in the row and column where the element is located to null values; Find all non-null value elements in each row of the association matrix, and match the fruit detection frame and the fruit stem detection frame corresponding to the element with the smallest absolute value of displacement.

8. The fruit picking point positioning method based on the improved deep learning network according to claim 7, It is characterized in that The format of the matching data is [F dis ,S box ,F box ],in: If the fruit detection frame matches the stem detection frame, S box is the matching fruit detection frame, F box is the matching fruit stem detection frame, F dis For S box and F box displacement values ​​in the incidence matrix; If the fruit detection frame does not match the stem detection frame, S box is the unmatched fruit detection frame, F box and F dis is a preset identifier; If the stem detection frame does not match the fruit detection frame, F box is the unmatched fruit stem detection frame, S box and F dis The preset identifier.

9. The fruit picking point positioning method based on the improved deep learning network according to claim 8, It is characterized in that In the association matrix, when the center point of the lower edge of the fruit stalk detection frame is above the center point of the upper edge of the fruit detection frame, the displacement between the two is recorded as a positive value, otherwise, the displacement between the two is recorded as a negative value; According to the matching data, the picking point of the fruit is obtained, including: For the matched fruit detection frame and stalk detection frame, determine whether the displacement between the two is a negative value: If yes, the intersection of the median line of the fruit stalk detection frame and the upper edge of the fruit detection frame is used as the picking point; If not, the center point of the lower edge of the fruit stalk detection frame is used as the picking point.

Images