Fruit target and picking point synchronous detection method and system

Abstract

The invention relates to a fruit target and picking point synchronous detection method and system. The method comprises the following steps: S1, constructing an image data set; s2, a fruit posture and picking point synchronous detection model based on a Beer-YOLO model is constructed, the Beer-YOLO model takes YOLOv8 as a basic model, DBeerBottleNet is adopted as a backbone network, feature information of different scales is fully extracted through feature fusion of large targets and small targets of different layers, meanwhile, a C2f module in a neck network is replaced with a PDConv module, and a fruit posture and picking point synchronous detection model is constructed; the method comprises the following steps: firstly, extracting local space features and channel features to realize efficient extraction of local space features and channel features, and in addition, replacing a C2f module connected with a neck network in a head network with a PS2Conv module to capture more fruit detail features; training the model through the image data set; and S3, inputting a to-be-detected image into the trained model to obtain a fruit target, a posture of the fruit target and a picking point. The method and the system can accurately and efficiently detect the strip-shaped fruit target and the posture and the picking point of the strip-shaped fruit target at the same time, so that the fruit picking efficiency and accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of agricultural automation, and in particular to a method and system for synchronously detecting fruit targets and picking points. Background Art

[0002] With the increasing shortage of agricultural labor resources, traditional manual picking methods can no longer meet the needs of efficient production in modern agriculture. As a key equipment for the development of smart agriculture, intelligent picking robots can realize automatic fruit picking tasks by integrating machine vision, artificial intelligence and automation technology. However, the agricultural planting environment is complex, and there are many occlusion problems, including occlusion of branches, occlusion of leaves, and occlusion between fruits, which seriously affect target detection and picking point positioning. Especially for some long fruits, such as green peppers, the fruits are longer and have more complex shapes, which can be straight, curved or even curly. Therefore, the occlusion problem is more serious and complicated. In addition, for green peppers, different degrees of bending have different values. If fruits with different postures can be preliminarily sorted out during picking, the automatic picking effect will be further improved. Therefore, it is necessary to develop a synchronous detection technology for fruit posture and picking point to solve the problems existing in the prior art. Summary of the invention

[0003] The object of the present invention is to provide a method and system for synchronously detecting fruit targets and picking points, which can accurately and efficiently synchronously detect fruit targets, their postures and picking points, thereby improving the efficiency and accuracy of fruit picking.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is: a method for synchronously detecting fruit targets and picking points, comprising the following steps: S1, obtain images of fruits to be picked, and mark the fruit targets and key points of the images, thereby constructing an image dataset; S2. Construct a fruit posture and picking point synchronization detection model based on the Beer-YOLO model. The Beer-YOLO model uses YOLOv8 as the basic model and DBeerBottleNet as the backbone network to fully extract feature information of different scales through the feature fusion of large targets and small targets at different feature extraction layers. At the same time, the C2f module in the neck network is replaced by the PDConv module to achieve efficient extraction of local spatial features and channel features. In addition, the C2f module connected to the neck network in the head network is replaced by the PS2Conv module to enhance the local feature extraction capability, thereby capturing more local detail features of the fruit; the fruit posture and picking point synchronization detection model based on the Beer-YOLO model is trained through the image data set; S3. Input the image to be detected into the trained fruit posture and picking point synchronous detection model to obtain the fruit target and its posture and picking point.

[0005] Furthermore, in step S1, the fruits to be picked on multiple different fruit trees in the real agricultural environment are photographed from multiple angles by a visual sensor to obtain multiple images of the fruits to be picked; For each fruit in each image, three key points are marked: the PICK key point is marked on the fruit stalk, i.e. the picking point, the TOP key point is marked at the connection between the fruit stalk and the fruit, and the BOTTOM key point is marked at the tip of the bottom of the fruit; at the same time, the fruit target box is marked, and the fruit target box just frames the fruit and includes the picking point; All images that have been annotated with fruit targets and key points are integrated to construct an image dataset, which is then divided into training set, validation set, and test set according to a set ratio for subsequent model training, parameter adjustment, and evaluation.

[0006] Furthermore, DBeerBottleNet, which serves as the backbone network of the Beer-YOLO model, consists of two beer bottle networks placed in opposite directions. The first beer bottle network placed in the forward direction includes multiple feature extraction layers with feature sizes varying from large to small, and the second beer bottle network placed in the reverse direction includes multiple feature extraction layers with feature sizes varying from small to large. The different feature extraction layers of the first beer bottle network and the second beer bottle network are feature fused to fully extract feature information of different scales.

[0007] Further, the first beer bottle network includes 6 feature extraction layers: L1~L6, the output feature sizes of L1, L2, and L3 layers are the same, the output feature sizes of L5 and L6 layers are the same, and convolution and downsampling are used in L3 and L4 layers to change the output feature sizes; the second beer bottle network includes 6 feature extraction layers: L1'~L6', the output feature sizes of L1' and L2' layers are the same, the output feature sizes of L4', L5', and L6' layers are the same, and convolution and upsampling are used in L2' and L3' layers to change the output feature sizes; Input the image of the Beer-YOLO model, and input it into the L1 layer of the first beer bottle network and the L1' layer of the second beer bottle network at the same time. The image input into the L1 layer passes through the L1~L6 layers in sequence for feature extraction. The output features of the L1 layer, the output features of the L5 layer, and the output features of the L1' layer are combined and input into the L2' layer for feature extraction. The output features of the L1 layer, the output features of the L2 layer, and the output features of the L5 layer are combined with the output features of the L2' layer and input into the L3' layer for feature extraction. The output features of the L3 layer are combined with the output features of the L3' layer and input into the L4' layer for feature extraction. The output features of the L4 layer are combined with the output features of the L4' layer and input into the L5' layer for feature extraction. The output features of the L5 layer are combined with the output features of the L5' layer and input into the L6' layer for feature extraction. The output features of the L1' layer are used as the first output features of the backbone network, the output features of the L3' layer are used as the second output features of the backbone network, and the output features of the L6 layer are combined with the output features of the L6' layer as the third output features of the backbone network and input into the neck network.

[0008] Furthermore, the output feature size of the L1, L2, and L3 layers is 160*160, the output feature size of the L4 layer is 80*80, and the output feature size of the L5 and L6 layers is 40*40; the output feature size of the L1' and L2' layers is 40*40, the output feature size of the L3' layer is 80*80, and the output feature size of the L4', L5', and L6' layers is 160*160.

[0009] Furthermore, each feature extraction layer is composed of a Conv module and a PS2Conv module; the Conv module is composed of a Conv2d layer, a BatchNorm2d layer and a SiLU activation function.

[0010] Furthermore, the implementation method of the PS2Conv module is: The PS2Conv module includes a first Conv module, a first PConv module, a first ScConv module, a second PConv module, a second ScConv module, a first Concat module and a second Conv module; the input features are sequentially extracted through the first Conv module, the first PConv module, the first ScConv module, the second PConv module and the second ScConv module; the output features of the first Conv module are also combined with the output features of the first ScConv module through a ShortCut connection and input into the second PConv module; the output features of the first Conv module, the first PConv module, the first ScConv module and the second PConv module are respectively derived and spliced ​​with the output features of the second ScConv module in channel dimension through the first Concat module to capture features of different scales through feature splicing; the output of the first Concat module is then convolved through the second Conv module to integrate the spliced ​​features to obtain the output features of the PS2Conv module.

[0011] Furthermore, the implementation method of the PDConv module is: The PDConv module includes a third Conv module, a fourth Conv module, a fifth Conv module, a third PConv module, a sixth Conv module, a fourth PConv module, a seventh Conv module, an eighth Conv module, a first DConv module, a ninth Conv module, a second DConv module, a second Concat module and a tenth Conv module; the features input into the PDConv module are divided into three paths, the first path is input into the third Conv module for feature extraction, and the second path passes through the fourth Conv module, the fifth Conv module, the third PConv module, the sixth Conv module, the fourth PConv module, the seventh Conv module, the eighth Conv module, the first DConv module, the ninth Conv module, the second DConv module, the second Concat module and the tenth Conv module in sequence. v module for feature extraction, and the third path passes through the seventh Conv module, the eighth Conv module, the first DConv module, the ninth Conv module, and the second DConv module in sequence for feature extraction; the output features of the third Conv module, the third PConv module, the fourth PConv module, the first DConv module, and the second DConv module are spliced ​​in the channel dimension through the second Concat module to capture features of different scales through feature splicing; the output of the second Concat module is then convolved through the tenth Conv module to integrate the spliced ​​features to obtain the output features of the PDConv module.

[0012] Furthermore, the head network of the Beer-YOLO model is provided with large, medium and small scale target detection heads and key point detection heads to synchronously detect the fruit target frame and three key points, and then determine the fruit posture and picking point through the three key points; In the key point detection head, vector calculation is performed on the three key points: starting from the TOP key point, two vectors are generated, one pointing to the PICK key point and the other pointing to the BOTTOM key point; n angle thresholds K1, K2, K3, ..., Kn are set, and the posture of the fruit is judged by calculating the relationship between the angle θ of the two vectors and each angle threshold: when the angle θ>K1, the fruit is judged to be straight, and when the angle θ≤K1, the fruit is judged to be curved, and the degree of curvature of the fruit is determined according to the relationship between the angle θ and each angle threshold, thereby realizing posture detection; through posture detection, preliminary fruit sorting can be achieved during the picking process.

[0013] The present invention also provides a fruit target and picking point synchronous detection system, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the above method can be implemented.

[0014] Compared with the prior art, the present invention has the following beneficial effects: the present invention provides a method and system for synchronous detection of fruit targets and picking points, which improves the YOLOv8 model, adopts the double beer bottle network DBeerBottleNet as the backbone network, and improves the relevant modules in the model to fully extract and fuse the feature information of different scales in the image of the fruit to be picked, so as to accurately and efficiently detect the fruit targets, especially the strip-shaped fruit targets and their postures and picking points, and thus improve the efficiency and accuracy of the intelligent picking robot for automatic fruit picking. Therefore, the present invention has strong practicality and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is an architecture diagram of the Beer-YOLO model in an embodiment of the present invention; Figure 2 is an image annotated with fruit objects and key points in an embodiment of the present invention; Figure 3 is a structural diagram of a PS2Conv module in an embodiment of the present invention; Figure 4 is a structural diagram of a PDConv module in an embodiment of the present invention; Figure 5 Schematic diagram of performing vector calculation on three key points in an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0017] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0018] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0019] like Figure 1 As shown, this embodiment provides a method for synchronously detecting fruit targets and picking points, comprising the following steps: S1, obtain images of fruits to be picked, and mark the fruit targets and key points of the images, thereby constructing an image dataset; S2. Construct a fruit posture and picking point synchronous detection model based on the Beer-YOLO model. The Beer-YOLO model uses YOLOv8 as the basic model and DBeerBottleNet as the backbone network (Backbone). The feature fusion of large and small targets at different feature extraction layers is used to fully extract feature information of different scales. At the same time, the C2f module in the neck network (Neck) is replaced with the PDConv module to achieve efficient extraction of local spatial features and channel features. In addition, the C2f module connected to the neck network in the head network (Head) is replaced with the PS2Conv module to enhance the local feature extraction capability, thereby capturing more local detail features of the fruit; the fruit posture and picking point synchronous detection model based on the Beer-YOLO model is trained through the image data set; S3. Input the image to be detected into the trained fruit posture and picking point synchronous detection model to obtain the fruit target and its posture and picking point.

[0020] In step S1, a visual sensor is used to photograph fruits to be picked on multiple different fruit trees in a real agricultural environment from multiple angles to obtain multiple images of the fruits to be picked.

[0021] For each fruit in each image, three key points are marked: the PICK key point is marked on the fruit stem, i.e. the picking point, the TOP key point is marked at the connection between the fruit stem and the fruit, and the BOTTOM key point is marked at the tip of the bottom of the fruit; at the same time, the fruit target frame is marked, and the fruit target frame just frames the fruit and includes the picking point. The image after the fruit target and key points are marked is as follows Figure 2shown.

[0022] All images that have been annotated with fruit targets and key points are integrated to construct an image dataset, which is then divided into training set, validation set, and test set according to a set ratio for subsequent model training, parameter adjustment, and evaluation.

[0023] like Figure 1 As shown in the figure, DBeerBottleNet, which is the backbone network of the Beer-YOLO model, consists of two beer bottle networks placed in opposite directions. The first beer bottle network placed in the forward direction includes multiple feature extraction layers with feature sizes varying from large to small, and the second beer bottle network placed in the reverse direction includes multiple feature extraction layers with feature sizes varying from small to large. The different feature extraction layers of the first beer bottle network and the second beer bottle network are feature fused to fully extract feature information of different scales.

[0024] The first beer bottle network includes 6 feature extraction layers: L1~L6, the output feature sizes of L1, L2, and L3 layers are the same, the output feature sizes of L5 and L6 layers are the same, and convolution and downsampling are used to change the output feature sizes in L3 and L4 layers; the second beer bottle network includes 6 feature extraction layers: L1'~L6', the output feature sizes of L1' and L2' layers are the same, the output feature sizes of L4', L5', and L6' layers are the same, and convolution and upsampling are used to change the output feature sizes in L2' and L3' layers.

[0025] Input the image of the Beer-YOLO model, and input it into the L1 layer of the first beer bottle network and the L1' layer of the second beer bottle network at the same time. The image input into the L1 layer passes through the L1~L6 layers in sequence for feature extraction. The output features of the L1 layer, the output features of the L5 layer, and the output features of the L1' layer are combined and input into the L2' layer for feature extraction. The output features of the L1 layer, the output features of the L2 layer, and the output features of the L5 layer are combined with the output features of the L2' layer and input into the L3' layer for feature extraction. The output features of the L3 layer are combined with the output features of the L3' layer and input into the L4' layer for feature extraction. The output features of the L4 layer are combined with the output features of the L4' layer and input into the L5' layer for feature extraction. The output features of the L5 layer are combined with the output features of the L5' layer and input into the L6' layer for feature extraction.

[0026] The output features of the L1' layer are used as the first output features of the backbone network, the output features of the L3' layer are used as the second output features of the backbone network, and the output features of the L6 layer are combined with the output features of the L6' layer as the third output features of the backbone network and input into the neck network.

[0027] In this embodiment, the output feature size of the L1, L2, and L3 layers is 160*160, the output feature size of the L4 layer is 80*80, and the output feature size of the L5 and L6 layers is 40*40; the output feature size of the L1' and L2' layers is 40*40, the output feature size of the L3' layer is 80*80, and the output feature size of the L4', L5', and L6' layers is 160*160.

[0028] DBeerBottleNet is composed of two beer bottle networks placed in opposite directions, and realizes the connection between the high-level and bottom-level layers of the two beer bottle networks. The connection between the high-level and bottom-level layers is not only conducive to feature fusion to fully extract feature information of different scales, but also can effectively realize lossless network propagation through these connections during the network back propagation process, thereby reducing information loss in the feature propagation process.

[0029] In DBeerBottleNet, each feature extraction layer consists of a Conv module and a PS2Conv module. The Conv module is the most basic module in the YOLOv8 architecture, which consists of a Conv2d layer, a BatchNorm2d layer, and a SiLU activation function. It should be noted that the Conv modules involved in the PS2Conv module and the PDConv module also refer to this basic module, which consists of a Conv2d layer, a BatchNorm2d layer, and a SiLU activation function.

[0030] like Figure 3 As shown, the implementation method of the PS2Conv module is: The PS2Conv module includes a first Conv module, a first PConv module, a first ScConv module, a second PConv module, a second ScConv module, a first Concat module and a second Conv module; the input features are sequentially extracted through the first Conv module, the first PConv module, the first ScConv module, the second PConv module and the second ScConv module; the output features of the first Conv module are also combined with the output features of the first ScConv module through a ShortCut connection and input into the second PConv module; the output features of the first Conv module, the first PConv module, the first ScConv module and the second PConv module are respectively derived and spliced ​​with the output features of the second ScConv module in channel dimension through the first Concat module to capture features of different scales through feature splicing; the output of the first Concat module is then convolved through the second Conv module to integrate the spliced ​​features to obtain the output features of the PS2Conv module.

[0031] The PS2Conv module combines Pointwise Convolution (PConv) and SpatialConvolution (ScConv), and enhances feature extraction capabilities through multi-branching and feature splicing. The PS2Conv has the following features: Multi-branch design: The feature map undergoes different convolution operations (PConv and ScConv) through multiple branches and is finally concatenated together through Concat to fuse different features.

[0032] The skip connection retains the original features, ensures the integrity of the information, and avoids excessive feature loss.

[0033] Combination of PConv and ScConv: PConv (1x1 convolution) focuses on the fusion and dimension compression of channel features to improve network efficiency.

[0034] ScConv (spatial convolution) extracts local spatial features through larger convolution kernels (such as 3x3 or 5x5), enhancing the ability to capture texture, edge and other information.

[0035] like Figure 4 As shown, the implementation method of the PDConv module is: The PDConv module includes a third Conv module, a fourth Conv module, a fifth Conv module, a third PConv module, a sixth Conv module, a fourth PConv module, a seventh Conv module, an eighth Conv module, a first DConv module, a ninth Conv module, a second DConv module, a second Concat module and a tenth Conv module; the features input into the PDConv module are divided into three paths, the first path is input into the third Conv module for feature extraction, and the second path passes through the fourth Conv module, the fifth Conv module, the third PConv module, the sixth Conv module, the fourth PConv module, the seventh Conv module, the eighth Conv module, the first DConv module, the ninth Conv module, the second DConv module, the second Concat module and the tenth Conv module in sequence. v module for feature extraction, and the third path passes through the seventh Conv module, the eighth Conv module, the first DConv module, the ninth Conv module, and the second DConv module in sequence for feature extraction; the output features of the third Conv module, the third PConv module, the fourth PConv module, the first DConv module, and the second DConv module are spliced ​​in the channel dimension through the second Concat module to capture features of different scales through feature splicing; the output of the second Concat module is then convolved through the tenth Conv module to integrate the spliced ​​features to obtain the output features of the PDConv module.

[0036] The PDConv module achieves efficient extraction of local spatial features and channel features through multi-branch design and feature concatenation (Concat). The branch structure of this module allows different paths to focus on different types of feature extraction, such as capturing detailed spatial information through DConv and fusing inter-channel features through PConv. At the same time, the jump connection retains the original feature information, ensures the integrity of the information flow, and finally achieves rich feature expression through feature fusion. The overall design takes into account the diversity of feature extraction and computational efficiency, and is suitable for application in high-resolution tasks that require detail capture.

[0037] The head network of the Beer-YOLO model is provided with large, medium and small scale target detection heads and key point detection heads to synchronously detect the fruit target frame and three key points, and then determine the fruit posture and picking point through the three key points.

[0038] In the key point detection head, vector calculations are performed on the three key points: Figure 5 As shown in the figure, with the TOP key point as the starting point, two vectors are generated, one pointing to the PICK key point and the other pointing to the BOTTOM key point; n angle thresholds K1, K2, K3, ..., Kn are set, and the posture of the fruit is judged by calculating the relationship between the angle θ of the two vectors and each angle threshold: when the angle θ>K1, the fruit is judged to be straight, and when the angle θ≤K1, the fruit is judged to be curved, and the curvature of the fruit is determined according to the relationship between the angle θ and each angle threshold, thereby realizing posture detection. Through posture detection, preliminary fruit sorting can be achieved during the picking process.

[0039] This embodiment also provides a fruit target and picking point synchronization detection system, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the above-mentioned method steps can be implemented.

[0040] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0041] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0042] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0043] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0044] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A method for synchronously detecting fruit targets and picking points, characterized in that: The following steps are involved: S1, obtain images of fruits to be picked, and mark the fruit targets and key points of the images, thereby constructing an image dataset; S2. Construct a fruit posture and picking point synchronization detection model based on the Beer-YOLO model. The Beer-YOLO model uses YOLOv8 as the basic model and DBeerBottleNet as the backbone network to fully extract feature information of different scales through the feature fusion of large targets and small targets at different feature extraction layers. At the same time, the C2f module in the neck network is replaced by the PDConv module to achieve efficient extraction of local spatial features and channel features. In addition, the C2f module connected to the neck network in the head network is replaced by the PS2Conv module to enhance the local feature extraction capability, thereby capturing more local detail features of the fruit; the fruit posture and picking point synchronization detection model based on the Beer-YOLO model is trained through the image data set; S3. Input the image to be detected into the trained fruit posture and picking point synchronous detection model to obtain the fruit target and its posture and picking point.

2. A method for synchronously detecting fruit targets and picking points according to claim 1, characterized in that: In step S1, the fruits to be picked on multiple different fruit trees in a real agricultural environment are photographed from multiple angles by a visual sensor to obtain multiple images of the fruits to be picked; For each fruit in each image, three key points are marked: the PICK key point is marked on the fruit stalk, i.e. the picking point, the TOP key point is marked at the connection between the fruit stalk and the fruit, and the BOTTOM key point is marked at the tip of the bottom of the fruit; at the same time, the fruit target box is marked, and the fruit target box just frames the fruit and includes the picking point; All images that have been annotated with fruit targets and key points are integrated to construct an image dataset, which is then divided into training set, validation set, and test set according to a set ratio for subsequent model training, parameter adjustment, and evaluation.

3. A method for synchronously detecting fruit targets and picking points according to claim 1, characterized in that: DBeerBottleNet, the backbone network of the Beer-YOLO model, consists of two beer bottle networks placed in opposite directions. The first beer bottle network placed in the forward direction includes multiple feature extraction layers with feature sizes varying from large to small, and the second beer bottle network placed in the reverse direction includes multiple feature extraction layers with feature sizes varying from small to large. The different feature extraction layers of the first beer bottle network and the second beer bottle network are feature fused to fully extract feature information of different scales.

4. A method for synchronously detecting fruit targets and picking points according to claim 3, characterized in that: The first beer bottle network includes 6 feature extraction layers: L1~L6, the output feature sizes of L1, L2, and L3 layers are the same, the output feature sizes of L5 and L6 layers are the same, and convolution and downsampling are used in L3 and L4 layers to change the output feature sizes; the second beer bottle network includes 6 feature extraction layers: L1'~L6', the output feature sizes of L1' and L2' layers are the same, the output feature sizes of L4', L5', and L6' layers are the same, and convolution and upsampling are used in L2' and L3' layers to change the output feature sizes; Input the image of the Beer-YOLO model, and input it into the L1 layer of the first beer bottle network and the L1' layer of the second beer bottle network at the same time. The image input into the L1 layer passes through the L1~L6 layers in sequence for feature extraction. The output features of the L1 layer, the output features of the L5 layer, and the output features of the L1' layer are combined and input into the L2' layer for feature extraction. The output features of the L1 layer, the output features of the L2 layer, and the output features of the L5 layer are combined with the output features of the L2' layer and input into the L3' layer for feature extraction. The output features of the L3 layer are combined with the output features of the L3' layer and input into the L4' layer for feature extraction. The output features of the L4 layer are combined with the output features of the L4' layer and input into the L5' layer for feature extraction. The output features of the L5 layer are combined with the output features of the L5' layer and input into the L6' layer for feature extraction. The output features of the L1' layer are used as the first output features of the backbone network, the output features of the L3' layer are used as the second output features of the backbone network, and the output features of the L6 layer are combined with the output features of the L6' layer as the third output features of the backbone network and input into the neck network.

5. A method for synchronously detecting fruit targets and picking points according to claim 4, characterized in that: The output feature size of L1, L2, and L3 layers is 160*160, the output feature size of L4 layer is 80*80, and the output feature size of L5 and L6 layers is 40*40; the output feature size of L1' and L2' layers is 40*40, the output feature size of L3' layer is 80*80, and the output feature size of L4', L5', and L6' layers is 160*160.

6. A method for synchronously detecting fruit targets and picking points according to claim 4, characterized in that: Each feature extraction layer consists of a Conv module and a PS2Conv module; the Conv module consists of a Conv2d layer, a BatchNorm2d layer and a SiLU activation function.

7. A method for synchronously detecting fruit targets and picking points according to claim 1, characterized in that: The implementation method of the PS2Conv module is: The PS2Conv module includes a first Conv module, a first PConv module, a first ScConv module, a second PConv module, a second ScConv module, a first Concat module and a second Conv module; the input features are sequentially extracted through the first Conv module, the first PConv module, the first ScConv module, the second PConv module and the second ScConv module; the output features of the first Conv module are also combined with the output features of the first ScConv module through a ShortCut connection and input into the second PConv module; the output features of the first Conv module, the first PConv module, the first ScConv module and the second PConv module are respectively derived and spliced ​​with the output features of the second ScConv module in channel dimension through the first Concat module to capture features of different scales through feature splicing; the output of the first Concat module is then convolved through the second Conv module to integrate the spliced ​​features to obtain the output features of the PS2Conv module.

8. The method for synchronously detecting fruit targets and picking points according to claim 1, characterized in that: The implementation method of the PDConv module is: The PDConv module includes a third Conv module, a fourth Conv module, a fifth Conv module, a third PConv module, a sixth Conv module, a fourth PConv module, a seventh Conv module, an eighth Conv module, a first DConv module, a ninth Conv module, a second DConv module, a second Concat module and a tenth Conv module; the features input into the PDConv module are divided into three paths, the first path is input into the third Conv module for feature extraction, the second path is sequentially passed through the fourth Conv module, the fifth Conv module, the third PConv module, the sixth Conv module, and the fourth PConv module for feature extraction, and the third path is sequentially passed through the seventh Conv module, the eighth Conv module, the first DConv module, the ninth Conv module, and the second DConv module for feature extraction; the output features of the third Conv module, the third PConv module, the fourth PConv module, the first DConv module and the second DConv module are spliced ​​in channel dimension through the second Concat module to capture features of different scales through feature splicing; The output of the second Concat module is then convolved through the tenth Conv module to integrate the concatenated features to obtain the output features of the PDConv module.

9. A method for synchronously detecting fruit targets and picking points according to claim 1, characterized in that: The head network of the Beer-YOLO model is equipped with large, medium and small scale target detection heads and key point detection heads to synchronously detect the fruit target frame and three key points, and then determine the fruit posture and picking point through the three key points; In the key point detection head, vector calculation is performed on the three key points: starting from the TOP key point, two vectors are generated, one pointing to the PICK key point and the other pointing to the BOTTOM key point; n angle thresholds K1, K2, K3, ..., Kn are set, and the posture of the fruit is judged by calculating the relationship between the angle θ of the two vectors and each angle threshold: when the angle θ>K1, the fruit is judged to be straight, and when the angle θ≤K1, the fruit is judged to be curved, and the degree of curvature of the fruit is determined according to the relationship between the angle θ and each angle threshold, thereby realizing posture detection; through posture detection, preliminary fruit sorting can be achieved during the picking process.

10. A fruit target and picking point synchronous detection system, characterized in that: The method comprises a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method according to any one of claims 1 to 9 can be implemented.

Images