A method and system for synchronous detection of fruit targets and picking points

Abstract

The present invention relates to a method and system for synchronous detection of fruit targets and picking points. The method comprises: S1, constructing an image dataset; S2, constructing a synchronous detection model for fruit posture and picking points based on a Beer-YOLO model. The Beer-YOLO model uses YOLOv8 as a base model and DBeerBottleNet as a backbone network to fully extract feature information of different scales by fusing features of large and small targets at different layers. The C2f module in the neck network is replaced with a PDConv module to achieve efficient extraction of local spatial features and channel features. In addition, the C2f module connected to the neck network in the head network is replaced with a PS2Conv module to capture more detailed features of the fruit part; the model is trained using the image dataset; S3, inputting the trained model into the image to be detected to obtain the fruit target, its posture, and picking point. The method and system can simultaneously and accurately detect bar-shaped fruit targets, their posture, and picking points, thereby improving the efficiency and accuracy of fruit picking.

Description

Technical Field

[0001] The present invention relates to the technical field of agricultural automation, and in particular to a method and system for synchronously detecting fruit targets and picking points. Background Art

[0002] With the increasing shortage of agricultural labor resources, traditional manual picking methods can no longer meet the needs of efficient modern agricultural production. Intelligent picking robots, as key equipment for the development of smart agriculture, integrate machine vision, artificial intelligence, and automation technologies to automatically harvest fruit. However, the complex agricultural growing environment presents various occlusion issues, including occlusion by branches, leaves, and fruit itself. This severely impacts target detection and picking point location. This is particularly true for elongated fruits, such as green peppers, which are longer and have more complex shapes, ranging from straight to curved and even curled. Furthermore, for green peppers, different degrees of curvature have different values. Being able to initially sort fruits with different postures during picking would further enhance automated harvesting. Therefore, it is necessary to develop a technology for simultaneous detection of fruit posture and picking point to address the challenges of existing technologies. Summary of the Invention

[0003] The purpose of the present invention is to provide a method and system for synchronous detection of fruit targets and picking points, which can accurately and efficiently synchronously detect fruit targets, their postures and picking points, thereby improving the efficiency and accuracy of fruit picking.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is: a method for synchronous detection of fruit targets and picking points, comprising the following steps:

[0005] S1. Obtain images of fruits to be picked, annotate the images with fruit targets and key points, and then construct an image dataset.

[0006] S2. Construct a fruit posture and picking point synchronous detection model based on the Beer-YOLO model. The Beer-YOLO model uses YOLOv8 as the basic model and DBeerBottleNet as the backbone network to fully extract feature information of different scales through feature fusion of large and small targets at different feature extraction layers. At the same time, the C2f module in the neck network is replaced with the PDConv module to achieve efficient extraction of local spatial features and channel features. In addition, the C2f module connected to the neck network in the head network is replaced with the PS2Conv module to enhance the local feature extraction capability and thereby capture more local detail features of the fruit. The fruit posture and picking point synchronous detection model based on the Beer-YOLO model is trained using an image dataset.

[0007] S3. Input the image to be detected into the trained fruit posture and picking point synchronous detection model to obtain the fruit target, its posture and picking point.

[0008] Furthermore, in step S1, a plurality of images of the fruits to be picked on a plurality of different fruit trees in a real agricultural environment are photographed from multiple angles by a visual sensor to obtain a plurality of images of the fruits to be picked;

[0009] For each fruit in each image, three key points are marked: the PICK key point is marked on the fruit stem, i.e. the picking point; the TOP key point is marked at the connection between the fruit stem and the fruit; and the BOTTOM key point is marked at the tip of the bottom of the fruit. At the same time, a fruit target box is marked, which just frames the fruit and includes the picking point.

[0010] All images that have been annotated with fruit targets and key points are integrated to construct an image dataset, which is then divided into training set, validation set, and test set according to a set ratio for subsequent model training, parameter adjustment, and evaluation.

[0011] Furthermore, DBeerBottleNet, the backbone network of the Beer-YOLO model, consists of two beer bottle networks placed in opposite directions. The first beer bottle network placed in the forward direction includes multiple feature extraction layers with feature sizes varying from large to small, and the second beer bottle network placed in the reverse direction includes multiple feature extraction layers with feature sizes varying from small to large. The different feature extraction layers of the first beer bottle network and the second beer bottle network are feature fused to fully extract feature information of different scales.

[0012] Furthermore, the first beer bottle network includes six feature extraction layers: L1 to L6, where the output feature sizes of the L1, L2, and L3 layers are the same, and the output feature sizes of the L5 and L6 layers are the same. Convolution and downsampling are respectively used after the L3 and L4 layers to change the output feature sizes. The second beer bottle network includes six feature extraction layers: L1' to L6', where the output feature sizes of the L1' and L2' layers are the same, and the output feature sizes of the L4', L5', and L6' layers are the same. Convolution and upsampling are respectively used after the L2' and L3' layers to change the output feature sizes.

[0013] Input the image of the Beer-YOLO model and input it into the L1 layer of the first beer bottle network and the L1' layer of the second beer bottle network at the same time. The image input into the L1 layer passes through the L1 to L6 layers in sequence for feature extraction. The output features of the L1 layer, the output features of the L5 layer, and the output features of the L1' layer are combined and input into the L2' layer for feature extraction. The output features of the L1 layer, the output features of the L2 layer, the output features of the L5 layer, and the output features of the L2' layer are combined and input into the L3' layer for feature extraction. The output features of the L3 layer and the output features of the L3' layer are combined and input into the L4' layer for feature extraction. The output features of the L4 layer and the output features of the L4' layer are combined and input into the L5' layer for feature extraction. The output features of the L5 layer and the output features of the L5' layer are combined and input into the L6' layer for feature extraction.

[0014] The output features of the L1' layer are used as the first output features of the backbone network, the output features of the L3' layer are used as the second output features of the backbone network, and the output features of the L6 layer and the L6' layer are combined as the third output features of the backbone network and input into the neck network.

[0015] Furthermore, the output feature size of the L1, L2, and L3 layers is 160*160, the output feature size of the L4 layer is 80*80, and the output feature size of the L5 and L6 layers is 40*40; the output feature size of the L1' and L2' layers is 40*40, the output feature size of the L3' layer is 80*80, and the output feature size of the L4', L5', and L6' layers is 160*160.

[0016] Furthermore, each feature extraction layer is composed of a Conv module and a PS2Conv module; the Conv module is composed of a Conv2d layer, a BatchNorm2d layer and a SiLU activation function.

[0017] Furthermore, the PS2Conv module is implemented as follows:

[0018] The PS2Conv module includes a first Conv module, a first PConv module, a first ScConv module, a second PConv module, a second ScConv module, a first Concat module and a second Conv module; the input features are sequentially extracted through the first Conv module, the first PConv module, the first ScConv module, the second PConv module and the second ScConv module; the output features of the first Conv module are also combined with the output features of the first ScConv module through a ShortCut connection and input into the second PConv module; the output features of the first Conv module, the first PConv module, the first ScConv module and the second PConv module are respectively derived and spliced ​​with the output features of the second ScConv module in the channel dimension through the first Concat module to capture features of different scales through feature splicing; the output of the first Concat module is then convolved through the second Conv module to integrate the spliced ​​features to obtain the output features of the PS2Conv module.

[0019] Furthermore, the implementation method of the PDConv module is:

[0020] The PDConv module includes a third Conv module, a fourth Conv module, a fifth Conv module, a third PConv module, a sixth Conv module, a fourth PConv module, a seventh Conv module, an eighth Conv module, a first DConv module, a ninth Conv module, a second DConv module, a second Concat module and a tenth Conv module; the features input to the PDConv module are divided into three paths, the first path is input to the third Conv module for feature extraction, the second path passes through the fourth Conv module, the fifth Conv module, the third PConv module, the sixth Conv module, the fourth PConv module, the v module for feature extraction, and the third path passes through the seventh Conv module, the eighth Conv module, the first DConv module, the ninth Conv module, and the second DConv module for feature extraction in sequence; the output features of the third Conv module, the third PConv module, the fourth PConv module, the first DConv module, and the second DConv module are spliced ​​in the channel dimension through the second Concat module to capture features of different scales through feature splicing; the output of the second Concat module is then convolved through the tenth Conv module to integrate the spliced ​​features to obtain the output features of the PDConv module.

[0021] Furthermore, the head network of the Beer-YOLO model is equipped with large, medium and small scale object detection heads and key point detection heads to simultaneously detect the fruit target frame and three key points, and then determine the fruit posture and picking point through the three key points;

[0022] In the key point detection head, vector calculation is performed on the three key points: starting from the TOP key point, two vectors are generated, one pointing to the PICK key point and the other pointing to the BOTTOM key point; n angle thresholds K1, K2, K3, ..., Kn are set, and the posture of the fruit is judged by calculating the relationship between the angle θ between the two vectors and each angle threshold: when the angle θ>K1, the fruit is judged to be straight; when the angle θ≤K1, the fruit is judged to be curved, and the degree of curvature of the fruit is determined based on the relationship between the angle θ and each angle threshold, thereby realizing posture detection; through posture detection, preliminary fruit sorting can be achieved during the picking process.

[0023] The present invention also provides a fruit target and picking point synchronous detection system, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the above method can be implemented.

[0024] Compared with the existing technology, the present invention has the following advantages: It provides a method and system for synchronously detecting fruit targets and picking points. This method improves the YOLOv8 model, adopts the dual beer bottle network DBeerBottleNet as the backbone network, and improves relevant modules in the model to fully extract and fuse feature information of different scales in the image of the fruit to be picked. This allows for accurate and efficient detection of fruit targets, particularly strip-shaped fruit targets, their posture, and picking points, thereby improving the efficiency and accuracy of automatic fruit picking by intelligent picking robots. Therefore, the present invention has strong practicality and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is an architecture diagram of the Beer-YOLO model in an embodiment of the present invention;

[0026] Figure 2 The image is annotated with fruit objects and key points in the embodiment of the present invention;

[0027] Figure 3 is a structural diagram of the PS2Conv module in an embodiment of the present invention;

[0028] Figure 4 is a structural diagram of the PDConv module in an embodiment of the present invention;

[0029] Figure 5 Schematic diagram of performing vector calculation on three key points in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0031] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0032] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0033] like Figure 1 As shown, this embodiment provides a method for synchronously detecting fruit targets and picking points, including the following steps:

[0034] S1. Obtain images of fruits to be picked, annotate the images with fruit targets and key points, and then construct an image dataset.

[0035] S2. Construct a fruit posture and picking point synchronous detection model based on the Beer-YOLO model. The Beer-YOLO model uses YOLOv8 as the basic model and DBeerBottleNet as the backbone network (Backbone). It fully extracts feature information of different scales by fusing the features of large and small targets at different feature extraction layers. At the same time, the C2f module in the neck network (Neck) is replaced with the PDConv module to achieve efficient extraction of local spatial features and channel features. In addition, the C2f module connected to the neck network in the head network (Head) is replaced with the PS2Conv module to enhance the local feature extraction capability and thereby capture more local detail features of the fruit. The fruit posture and picking point synchronous detection model based on the Beer-YOLO model is trained using an image dataset.

[0036] S3. Input the image to be detected into the trained fruit posture and picking point synchronous detection model to obtain the fruit target, its posture and picking point.

[0037] In step S1, a visual sensor is used to shoot fruits to be picked on multiple different fruit trees in a real agricultural environment from multiple angles to obtain multiple images of the fruits to be picked.

[0038] For each fruit in each image, three key points are marked: the PICK key point is marked on the fruit stem, i.e. the picking point; the TOP key point is marked at the connection between the fruit stem and the fruit; and the BOTTOM key point is marked at the tip of the bottom of the fruit; at the same time, a fruit target frame is marked, which just frames the fruit and includes the picking point. The image after the fruit target and key points are marked is as follows Figure 2 shown.

[0039] All images that have been annotated with fruit targets and key points are integrated to construct an image dataset, which is then divided into training set, validation set, and test set according to a set ratio for subsequent model training, parameter adjustment, and evaluation.

[0040] like Figure 1 As shown in the figure, DBeerBottleNet, the backbone network of the Beer-YOLO model, consists of two beer bottle networks placed in opposite directions. The first beer bottle network placed in the forward direction includes multiple feature extraction layers with feature sizes varying from large to small, and the second beer bottle network placed in the reverse direction includes multiple feature extraction layers with feature sizes varying from small to large. The different feature extraction layers of the first beer bottle network and the second beer bottle network are feature fused to fully extract feature information of different scales.

[0041] The first beer bottle network includes 6 feature extraction layers: L1~L6, the output feature sizes of L1, L2, and L3 layers are the same, the output feature sizes of L5 and L6 layers are the same, and convolution and downsampling are used after L3 and L4 layers to change the output feature sizes; the second beer bottle network includes 6 feature extraction layers: L1'~L6', the output feature sizes of L1' and L2' layers are the same, the output feature sizes of L4', L5', and L6' layers are the same, and convolution and upsampling are used after L2' and L3' layers to change the output feature sizes.

[0042] Input the image of the Beer-YOLO model and input it into the L1 layer of the first beer bottle network and the L1' layer of the second beer bottle network at the same time. The image input into the L1 layer passes through the L1~L6 layers in sequence for feature extraction. The output features of the L1 layer, the output features of the L5 layer, and the output features of the L1' layer are combined and input into the L2' layer for feature extraction. The output features of the L1 layer, the output features of the L2 layer, the output features of the L5 layer, and the output features of the L2' layer are combined and input into the L3' layer for feature extraction. The output features of the L3 layer and the output features of the L3' layer are combined and input into the L4' layer for feature extraction. The output features of the L4 layer and the output features of the L4' layer are combined and input into the L5' layer for feature extraction. The output features of the L5 layer and the output features of the L5' layer are combined and input into the L6' layer for feature extraction.

[0043] The output features of the L1' layer are used as the first output features of the backbone network, the output features of the L3' layer are used as the second output features of the backbone network, and the output features of the L6 layer and the L6' layer are combined as the third output features of the backbone network and input into the neck network.

[0044] In this embodiment, the output feature size of the L1, L2, and L3 layers is 160*160, the output feature size of the L4 layer is 80*80, and the output feature size of the L5 and L6 layers is 40*40; the output feature size of the L1' and L2' layers is 40*40, the output feature size of the L3' layer is 80*80, and the output feature size of the L4', L5', and L6' layers is 160*160.

[0045] DBeerBottleNet is composed of two beer bottle networks placed in opposite directions, and realizes the connection between the high-level and low-level layers. The connection between the high-level and low-level layers not only facilitates feature fusion to fully extract feature information at different scales, but also effectively realizes lossless network propagation through these connections during the network backpropagation process, reducing information loss in the feature propagation process.

[0046] In DBeerBottleNet, each feature extraction layer consists of a Conv module and a PS2Conv module. The Conv module is the most basic module in the YOLOv8 architecture, consisting of a Conv2d layer, a BatchNorm2d layer, and a SiLU activation function. It should be noted that the Conv module involved in the PS2Conv module and the PDConv module also refers to this basic module, which consists of a Conv2d layer, a BatchNorm2d layer, and a SiLU activation function.

[0047] like Figure 3 As shown, the implementation method of the PS2Conv module is:

[0048] The PS2Conv module includes a first Conv module, a first PConv module, a first ScConv module, a second PConv module, a second ScConv module, a first Concat module and a second Conv module; the input features are sequentially extracted through the first Conv module, the first PConv module, the first ScConv module, the second PConv module and the second ScConv module; the output features of the first Conv module are also combined with the output features of the first ScConv module through a ShortCut connection and input into the second PConv module; the output features of the first Conv module, the first PConv module, the first ScConv module and the second PConv module are respectively derived and spliced ​​with the output features of the second ScConv module in the channel dimension through the first Concat module to capture features of different scales through feature splicing; the output of the first Concat module is then convolved through the second Conv module to integrate the spliced ​​features to obtain the output features of the PS2Conv module.

[0049] The PS2Conv module combines Pointwise Convolution (PConv) and SpatialConvolution (ScConv) and enhances feature extraction capabilities through multi-branching and feature splicing. The PS2Conv has the following features:

[0050] Multi-branch design: The feature map undergoes different convolution operations (PConv and ScConv) through multiple branches and is finally concatenated together through Concat to fuse different features.

[0051] The skip connection preserves the original features, ensures the integrity of the information and avoids excessive feature loss.

[0052] Combination of PConv and ScConv: PConv (1x1 convolution) focuses on the fusion of channel features and dimension compression to improve network efficiency.

[0053] ScConv (spatial convolution) extracts local spatial features through larger convolution kernels (such as 3x3 or 5x5), enhancing the ability to capture texture, edge and other information.

[0054] like Figure 4 As shown, the implementation method of the PDConv module is:

[0055] The PDConv module includes a third Conv module, a fourth Conv module, a fifth Conv module, a third PConv module, a sixth Conv module, a fourth PConv module, a seventh Conv module, an eighth Conv module, a first DConv module, a ninth Conv module, a second DConv module, a second Concat module and a tenth Conv module; the features input to the PDConv module are divided into three paths, the first path is input to the third Conv module for feature extraction, the second path passes through the fourth Conv module, the fifth Conv module, the third PConv module, the sixth Conv module, the fourth PConv module, the v module for feature extraction, and the third path passes through the seventh Conv module, the eighth Conv module, the first DConv module, the ninth Conv module, and the second DConv module for feature extraction in sequence; the output features of the third Conv module, the third PConv module, the fourth PConv module, the first DConv module, and the second DConv module are spliced ​​in the channel dimension through the second Concat module to capture features of different scales through feature splicing; the output of the second Concat module is then convolved through the tenth Conv module to integrate the spliced ​​features to obtain the output features of the PDConv module.

[0056] The PDConv module achieves efficient extraction of local spatial and channel features through a multi-branch design and feature concatenation (Concat). This branching structure allows different paths to focus on different types of feature extraction, such as capturing detailed spatial information through DConv and fusing inter-channel features through PConv. Furthermore, skip connections preserve the original feature information, ensuring the integrity of the information flow. Ultimately, feature fusion enables rich feature expression. This overall design balances feature extraction diversity with computational efficiency, making it suitable for high-resolution tasks requiring detailed capture.

[0057] The head network of the Beer-YOLO model is equipped with large, medium, and small scale target detection heads and key point detection heads to simultaneously detect the fruit target frame and three key points, and then determine the fruit posture and picking point through the three key points.

[0058] In the key point detection head, vector calculation is performed on the three key points: Figure 5As shown in the figure, starting from the TOP keypoint, two vectors are generated: one pointing to the PICK keypoint and the other to the BOTTOM keypoint. N angle thresholds K1, K2, K3, ..., Kn are set. The fruit's posture is determined by calculating the relationship between the angle θ between the two vectors and each angle threshold. When the angle θ > K1, the fruit is considered straight; when the angle θ ≤ K1, the fruit is considered curved. The degree of curvature of the fruit is determined based on the relationship between the angle θ and each angle threshold, thus achieving posture detection. This posture detection enables preliminary fruit sorting during the picking process.

[0059] This embodiment also provides a fruit target and picking point synchronous detection system, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the above-mentioned method steps can be implemented.

[0060] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0061] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0062] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0063] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0064] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

Claims

1. A method for synchronous detection of fruit targets and picking points, characterized in that: The following steps are involved: S1. Obtain images of fruits to be picked, annotate the images with fruit targets and key points, and then construct an image dataset. S2. Build a fruit posture and picking point synchronous detection model based on the Beer-YOLO model: The Beer-YOLO model uses YOLOv8 as the base model and DBeerBottleNet as the backbone network. DBeerBottleNet consists of two beer bottle networks placed in opposite directions. The forward-facing beer bottle network includes multiple feature extraction layers with feature sizes varying from large to small, and the reverse-facing beer bottle network includes multiple feature extraction layers with feature sizes varying from small to large. The different feature extraction layers of the forward-facing beer bottle network and the reverse-facing beer bottle network are fused to extract feature information of different scales. At the same time, the C2f module in the neck network is replaced with a PDConv module to extract local spatial features and channel features. Based on a parallel branch structure, PConv and DConv form branches and perform feature fusion to form the PDConv module. PConv is a point-by-point convolution. In addition, the C2f module connecting the head network to the neck network is replaced with a PS2Conv module to enhance the local feature extraction capability. The PS2Conv module is composed of a combination of PConv and ScConv based on a serial structure and cross-layer feature fusion. The ScConv is a spatial convolution. The image dataset is simultaneously input into the forward-facing beer bottle network and the reverse-facing beer bottle network, and the fruit posture and picking point synchronous detection model based on the Beer-YOLO model is trained using the image dataset; S3. Input the image to be detected into the trained fruit posture and picking point synchronous detection model based on the Beer-YOLO model to obtain the fruit target, its posture and picking point; wherein, the image to be detected is simultaneously input into the forward-placed beer bottle network and the reverse-placed beer bottle network.

2. A method for synchronously detecting fruit targets and picking points according to claim 1, characterized in that: The step S1 includes: The visual sensor is used to shoot multiple angles of fruits to be picked on different fruit trees in a real agricultural environment to obtain multiple images of the fruits to be picked; For each fruit in each image, three key points are marked: the PICK key point is marked on the fruit stem, which is the picking point; the TOP key point is marked at the connection between the fruit stem and the fruit; and the BOTTOM key point is marked at the tip of the bottom of the fruit. At the same time, the fruit target frame is marked, and the fruit target frame just frames the fruit and includes the picking point; all images after fruit target and key point annotation are integrated to construct an image dataset, and divided into training set, validation set and test set according to the set ratio, so as to train the model, adjust parameters and evaluate it in the future.

3. The method for synchronously detecting fruit targets and picking points according to claim 1, characterized in that: DBeerBottleNet, the backbone network of the Beer-YOLO model, consists of two beer bottle networks placed in opposite directions. The first beer bottle network placed in the forward direction includes multiple feature extraction layers with feature sizes varying from large to small, while the second beer bottle network placed in the reverse direction includes multiple feature extraction layers with feature sizes varying from small to large. The different feature extraction layers of the first and second beer bottle networks are fused to extract feature information of different scales.

4. A method for synchronously detecting fruit targets and picking points according to claim 3, characterized in that: DBeerBottleNet specifically includes: The first beer bottle network includes 6 feature extraction layers: L1 to L6; the output feature sizes of L1, L2, and L3 layers are the same; the output feature sizes of L5 and L6 layers are the same; convolution and downsampling are used after L3 and L4 layers respectively to change the output feature size; The second beer bottle network includes 6 feature extraction layers: L1'~L6'; the output feature sizes of L1' and L2' layers are the same; the output feature sizes of L4', L5', and L6' layers are the same; convolution and upsampling are used after L2' and L3' layers respectively to change the output feature size; The image input to the Beer-YOLO model is fed into both the L1 layer of the first beer bottle network and the L1' layer of the second beer bottle network. The image input to the L1 layer passes through the L1 to L6 layers in sequence for feature extraction; The output feature of the L1' layer is used as the first output feature of the backbone network and input into the neck network; The output features of the L1' layer are combined with the output features of the L1 layer and the output features of the L5 layer and input into the L2' layer for feature extraction; The output features of the L2' layer are combined with the output features of the L2 layer, the output features of the L1 layer, and the output features of the L5 layer and input into the L3' layer for feature extraction; The output features of the L3' layer are used as the second output features of the backbone network and input into the neck network; The output features of the L3' layer are combined with the output features of the L3 layer and input into the L4' layer for feature extraction. The output features of the L4' layer are combined with the output features of the L4 layer and input into the L5' layer for feature extraction. The output features of the L5' layer are combined with the output features of the L5 layer and input into the L6' layer for feature extraction; The output features of the L6' layer and the output features of the L6 layer are combined as the third output features of the backbone network and input into the neck network.

5. A method for synchronously detecting fruit targets and picking points according to claim 4, characterized in that: The output feature size of the L1, L2, and L3 layers is 160*160, the output feature size of the L4 layer is 80*80, and the output feature size of the L5 and L6 layers is 40*40; the output feature size of the L1' and L2' layers is 40*40, the output feature size of the L3' layer is 80*80, and the output feature size of the L4', L5', and L6' layers is 160*160.

6. A method for synchronously detecting fruit targets and picking points according to claim 4, characterized in that: Each feature extraction layer in the DBeerBottleNet consists of a Conv module and a PS2Conv module; The Conv module in the DBeerBottleNet consists of a Conv2d layer, a BatchNorm2d layer, and a SiLU activation function.

7. The method for synchronously detecting fruit targets and picking points according to claim 1, characterized in that: The implementation method of the PS2Conv module is: The PS2Conv module includes a first Conv module, a first PConv module, a first ScConv module, a second PConv module, a second ScConv module, a first Concat module and a second Conv module; The input features are sequentially extracted through the first Conv module, the first PConv module, the first ScConv module, the second PConv module, and the second ScConv module; And wherein, the output features of the first Conv module are further combined with the output features of the first ScConv module through a ShortCut connection and input into the second PConv module; The output features of the first Conv module, the first PConv module, the first ScConv module, and the second PConv module are respectively extracted and concatenated with the output features of the second ScConv module in the channel dimension through the first Concat module to capture features of different scales through feature concatenation; Afterwards, the output features of the first Concat module are convolved again through the second Conv module to integrate the concatenated features to obtain the output features of the PS2Conv module.

8. The method for synchronously detecting fruit targets and picking points according to claim 1, characterized in that: The implementation method of the PDConv module is: The PDConv module includes a third Conv module, a fourth Conv module, a fifth Conv module, a third PConv module, a sixth Conv module, a fourth PConv module, a seventh Conv module, an eighth Conv module, a first DConv module, a ninth Conv module, a second DConv module, a second Concat module and a tenth Conv module; The features input to the PDConv module are divided into three paths; The first path passes through the third Conv module for feature extraction; The second path passes through the fourth Conv module, the fifth Conv module, the third PConv module, the sixth Conv module, and the fourth PConv module in sequence for feature extraction; The third path passes through the seventh Conv module, the eighth Conv module, the first DConv module, the ninth Conv module, and the second DConv module in sequence for feature extraction; And wherein, the output features of the third Conv module, the third PConv module, the fourth PConv module, the first DConv module and the second DConv module are spliced ​​in the channel dimension through the second Concat module to capture features of different scales through feature splicing; Afterwards, the output features of the second Concat module are convolved again through the tenth Conv module to integrate the concatenated features to obtain the output features of the PDConv module.

9. The method for synchronously detecting fruit targets and picking points according to claim 2, characterized in that: The Beer-YOLO model's head network is equipped with large, medium, and small scale object detection heads and key point detection heads to simultaneously detect the fruit target frame and three key points, and then determine the fruit posture and picking point through the three key points; In the key point detection head, vector calculation is performed on the three key points, including: starting from the TOP key point, generating two vectors, one pointing to the PICK key point and the other pointing to the BOTTOM key point; setting n angle thresholds K1, K2, K3, ..., Kn, and determining the posture of the fruit by calculating the relationship between the angle θ between the two vectors and each angle threshold; when the angle θ>K1, the fruit is judged to be straight; when the angle θ≤K1, the fruit is judged to be curved; and based on the relationship between the angle θ and each angle threshold, the degree of curvature of the fruit is determined, thereby achieving posture detection; Through posture detection, preliminary fruit sorting is achieved during the picking process.

10. A fruit target and picking point synchronous detection system, characterized in that: The method comprises a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method according to any one of claims 1 to 9 can be implemented.

Images