Method and device for detecting picking key points of tomato fruits in red ripe stage based on multi-scale attention mechanism

Abstract

The invention discloses a red ripe tomato fruit picking key point detection method and device based on a multi-scale attention mechanism. The method comprises the following steps: preprocessing a shot image to obtain a data set, constructing a YOLOv8-pose model backbone network, and adding an LSKA attention mechanism module in a model to make the model pay more attention to a target; modifying a check part in a network structure, introducing a BiFPN module, and completing bidirectional information flow among different scales and layers; and training the network model, and detecting the test data set by using the trained model. According to the method, the accuracy of detecting the fruit stems and fruits of the tomatoes in the red ripe period is improved, and the probability of missing detection and false detection of the model is reduced; the problems that the convergence speed of training is slow and the performance is reduced along with the deepening of the network depth of the model are solved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, deep learning and artificial intelligence technology, and more specifically to a method and device for detecting key points for picking red ripe tomato fruits based on a multi-scale attention mechanism. Background Art

[0002] Tomatoes, picking by grasping can easily damage the fruit, so tomato picking robots need to cut the fruit stalks to pick them. Therefore, identifying tomato fruits and stalks is a key step for picking robots.

[0003] There are significant differences between tomato fruit and shear point detection and general target detection. Many target detection methods are not fully applicable to tomato fruit and stalk detection. The main reasons are as follows:

[0004] (1) The visual similarity between fruits (such as adjacent tomatoes) is high, and the positioning of key points for picking tomatoes is prone to errors and uncertainties.

[0005] (2) The changes in lighting conditions in the natural environment, the occlusion between tomato fruits during robot movement, and other complex factors increase the complexity of the tomato recognition task.

[0006] my country is a large agricultural country. Artificial intelligence-based picking robots can replace manual labor to do heavy picking work. Traditional tomato picking requires a large amount of labor, is highly complex to operate, and is highly seasonal. As China's aging society becomes more serious, labor costs are rising while fruit farmers' income is falling. In view of the above problems, the application and promotion of picking robots are of great significance.

[0007] Early detection of crop diseases can effectively increase crop yields. Gao et al. proposed a lightweight improved model ActNN-YOLOv5s-RepFPN to study tomato diseases in complex scenarios. They used Mosaic data enhancement to expand data, used ActNN to replace the network components in YOLOv5s, and built a RepFPN feature pyramid in the Neck of YOLOv5s to enhance the flow of feature information and achieve a hardware neural network design with balanced computing and memory for tomato disease detection. In order to better embed the detection algorithm into the tomato picking robot, Zhang et al. proposed a tomato picking robot target detection algorithm based on Shufflenetv2-YOLOv5. Based on the YOLOv5 algorithm, the lightweight network Shuffenetv2 is used as the backbone network to reduce the number of network parameters and the amount of calculation. The ECA attention mechanism is introduced, and the original activation function is replaced by the FReLU activation function to further improve the detection accuracy and effectively increase the robustness of the model.

[0008] Since the birth of robots, they have developed rapidly. In the fruit harvesting process, before the picking robot can realize the function of picking fruits, it needs to identify the fruits from the complex natural environment through the target detection system. The existing picking robots usually only have the target detection and recognition of fruits, and lack the key point detection of the fruit stalks, so they cannot be directly applied to tomato picking operations; when the robot takes images during the movement, the tomato fruits will overlap with each other as the robot's position changes, which seriously affects its model detection performance. Summary of the invention

[0009] The present invention aims to overcome the shortcomings of the prior art and proposes a method and device for detecting key points of tomato fruits in the red ripe period based on a multi-scale attention mechanism. The present invention improves the detection algorithm model while improving the algorithm performance of detecting red ripe tomato fruits and fruit stalks.

[0010] The technical solution of the present invention is as follows:

[0011] The first aspect of the present invention relates to a method for detecting key points for picking red ripe tomato fruits based on a multi-scale attention mechanism, comprising the following steps:

[0012] Step 1, construct an improved YOLOv8 model based on LSKA attention mechanism and BiFPN feature fusion module, the improved YOLOv8 model includes Backbone, Neck and Head, the Backbone includes Conv module, C2f module and SPPF module; the Neck includes upsample layer, Concat layer, C2f module and Conv module, and the BiFPN feature fusion module is used to replace the Concat layer in the Neck;

[0013] Step 2: Obtain an initial dataset of red ripe tomato images, perform data enhancement preprocessing on the images in the initial dataset, and obtain an enhanced dataset;

[0014] Step 3: Divide the enhanced dataset into training, validation, and test sets, use the training set to train the improved YOLOv8 model, and use the validation and test sets to evaluate the trained improved YOLOv8 model, including:

[0015] Step 3.1: Use the Backbone structure in the improved YOLOv8 model constructed in step 1 to extract the feature vector of the images in the training set, and output them in the 4th, 6th and 9th layers. The features output by the 9th layer are input into the LSKA module for processing; then output the processed features into the Neck structure; use CIoU as the bounding box loss function, and adopt non-maximum suppression; use the officially defined KeypointLoss loss function and BCEWithLogitsLoss loss function to calculate the key point position loss and key point confidence loss.

[0016] Step 3.2: Use the validation and test sets to evaluate the trained model. The evaluation indicators include accuracy P, average precision mAP, recall R, and average key point precision mAP-kp.

[0017] Step 4: Use the trained improved YOLOv8 model to perform target recognition and key point detection on tomatoes in a real planting environment.

[0018] Among them, BiFPN is used in step 1 to perform feature fusion operation. Different feature input resolutions are different, and their contributions to the output of the final feature network will also be different, so the network needs to learn the weights. BiFPN can realize bidirectional cross-scale information flow and fast normalized feature fusion between high-level and low-level feature information, including: inputting feature maps of different levels into the BiFPN_Concat layer (taking BiFPN_Concat2 as an example), given two feature maps Where c1 and C2 are the number of input channels, H and W represent the height and width of the feature map respectively. The BiFPN_Concat2 module weight is initialized to w = [1, 1], giving each input feature map the same initial importance, and normalizing it each time it is forward propagated. Set ∈ = 0.0001 to prevent the denominator from returning to zero. The normalized weight calculation formula is:

[0019]

[0020] In BiFPN_Concat2, n = 2. After normalization, weight = [weight0, weight1], and weight0 + weight1 = 1. The two input feature maps are weighted with the normalized weights: In the above operation, × is element-wise multiplication. During the training process, the model optimizes w based on the loss feedback, thereby changing the weight. Then BiFPN_Concat2 concatenates the two weighted feature maps in the channel dimension to obtain the output feature map:

[0021]

[0022] at this time The feature information from F0 and F1 is successfully integrated.

[0023] Among them, in the initial data set of red ripe tomatoes obtained in step 2, Labelme is used to manually annotate the data, marked in JSON format, and converted into TXT format through a script program; the label category is tomato, and the key point categories are P1, P1_hide, P2, P2_hide, P3 and P3_hide, where the hide suffix indicates that the key point exists in the image but is blocked.

[0024] Among them, the input resolution of the image is 4000×3000.

[0025] The data enhancement preprocessing in step 2 refers to one or more of random rotation, flipping and adding Gaussian noise.

[0026] Among them, the random rotation is specifically: select the image to randomly select an angle of 15° or -15° clockwise for rotation; the flipping is specifically: select the image to flip 180°; the adding Gaussian noise is specifically: select the image to randomly perform Gaussian blur processing with a kernel size of 11 or 7.

[0027] In step 3.1, the features output by the 9th layer are input into the LSKA module for processing. Large weight kernels can obtain large effective receptive fields (ERFs). LSKA decomposes the DW-Conv with a kernel size of k*k into two serial 1D DW-Convs with kernel sizes of 1×(2d-1) and (2d-1)×1 and two serial 1D DW-Convs with kernel sizes of and DW-D-Conv, after 1×1Conv output, the deep dilation convolution is responsible for capturing the global spatial information of the deep convolution output, which can achieve an effective receptive field equivalent to that of a large weight kernel while alleviating the quadratic increase in computational cost caused by using a large kernel alone for deep convolution. Specifically, given an input feature map F∈R C×H×W , where C is the number of input channels, H and W represent the height and width of the feature map respectively, and the output of LSKA is as follows:

[0028]

[0029] A C =W 1×1 *Z C (5)

[0030]

[0031] Where d is the expansion rate, k represents the maximum receptive field, and * and represent convolution and Hadamard product respectively, represents the input of two cascaded 1D kernels of size 1×(2d-1)×1 for DW-Conv F C The result of Hadamard operation.

[0032] The training parameters in step 3.1 are set as follows: the input image size is 640×640 pixels, the batch size is 32, the learning rate is initialized to 0.01 and finally to 0.032, the batch size is 100, and other parameters remain default.

[0033] Among them, in step 3.2, the accuracy P, average precision mAP and recall R are used to compare and analyze the detection performance of tomato fruit; the specific formulas are shown in formulas (7) to (9):

[0034]

[0035] Where: TP is the number of correctly detected frames, FP is the number of incorrectly detected frames, and FN is the number of missed targets. AP is the average accuracy of a certain category, and mAP is the average AP of all detection categories. The specific calculation formula is as follows:

[0036]

[0037] Where: N is the number of target categories that need to be detected. In this experiment, only one type of red-ripe tomatoes needs to be detected, so N=1.

[0038] The target key point similarity OKS is obtained through detection, and the average accuracy AP can be obtained from OKS. mAP-kp is calculated from AP as the evaluation index of the key point.

[0039] OKS p The calculation formula is:

[0040]

[0041] Where, d pi ——The Euclidean distance between the predicted value and the true value of the i-th key point of the p-th target

[0042] v pi ——The visibility index of the i-th key point of the p-th target

[0043] s p ——The area of ​​the pth target bounding box

[0044] δ i ——The standard deviation between the labeled value and the actual value of the i-th key point.

[0045] The second aspect of the present invention relates to a device for detecting key points for picking red-ripe tomato fruits based on a multi-scale attention mechanism, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a method for detecting key points for picking red-ripe tomato fruits based on a multi-scale attention mechanism of the present invention.

[0046] The beneficial effects of the present invention are as follows:

[0047] 1. Add the LSKA attention mechanism module to the model. LSKA uses depthwise separable convolution and depthwise dilated convolution to effectively capture long-distance dependencies in feature maps, expand the effective receptive field range, and make the model pay more attention to the target itself and better focus on the key points of tomato picking. Use the OKS key point loss function as its loss function to reduce the impact of the number of key points on the prediction results. This solves the problem that the visual similarity between fruits is high, and the positioning of the key points of tomato picking is prone to errors and uncertainties.

[0048] 2. The BiFPN feature pyramid module is introduced into the Neck part to complete the two-way information flow between different scales and levels, enhance the model feature fusion capability, strengthen the features of the occluded part of the image while suppressing the occluded area and background noise, and improve the model's ability to detect occluded tomatoes. It solves the problem of model detection performance degradation caused by multiple complex factors such as changes in lighting conditions under natural conditions and occlusion problems in fruit images taken during robot movement. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is the YOLOv8-pose network structure diagram used in the prior art;

[0050] Figure 2 This is the improved YOLOv8-pose network structure diagram of the present invention;

[0051] Figure 3 It is the structural diagram of the LSKA attention mechanism of the present invention;

[0052] Figure 4 It is a BiFPN module structure diagram of the present invention;

[0053] Figure 5 It is an implementation flow chart of the present invention;

[0054] Figure 6 It is a schematic diagram of the device of the present invention. DETAILED DESCRIPTION

[0055] The following is a more detailed description of the embodiments of the present invention. Figure 5 The embodiments described below with reference to the accompanying drawings are only used to explain the present invention and should not be interpreted as limiting the present invention.

[0056] Example 1

[0057] Reference Figure 1-5 This embodiment provides a method for detecting key points for picking red ripe tomato fruits based on a multi-scale attention mechanism, comprising the following steps:

[0058] Step 1, construct a red ripe tomato fruit and picking key point detection model based on LSKA attention mechanism and BiFPN feature fusion module to improve YOLOv8, the improved YOLOv8 model includes Backbone, Neck and Head, the Backbone includes Conv module, C2f module and SPPF module; the Neck includes upsample layer, Concat layer, C2f module and Conv module, and the BiFPN feature fusion module is used to replace the Concat layer in the Neck;

[0059] Step 2: Obtain an initial dataset of red ripe tomato images, perform data enhancement preprocessing on the images in the initial dataset, and obtain an enhanced dataset;

[0060] Step 3: Divide the enhanced dataset into training, validation, and test sets, use the training set to train the improved YOLOv8 model, and use the validation and test sets to evaluate the trained improved YOLOv8 model, including:

[0061] Step 3.1: Use the Backbone structure in the improved YOLOv8 model constructed in step 1 to extract the feature vector of the images in the training set, and output them in the 4th, 6th and 9th layers. The features output by the 9th layer are input into the LSKA module for processing; then output the processed features into the Neck structure; use CIoU as the bounding box loss function, and use non-maximum suppression NMS; use the official definition class KeypointLoss loss function and BCEWithLogitsLoss loss function to calculate the key point position loss and key point confidence loss.

[0062] Step 3.2: After training, the validation and test sets are used to evaluate the trained model. The evaluation indicators used include accuracy P, mean average precision mAP, recall R, and mean average precision of key points mAP-kp.

[0063] Step 4: Use the trained improved YOLOv8 model to perform target recognition and key point detection on tomatoes in a real planting environment.

[0064] The Backbone structure is a convolutional neural network CNN, which can effectively extract the feature information of the input image. Among them, the Conv module is the basic convolution unit of Backbone, and its mechanism of action is to perform two-dimensional convolution, two-dimensional regularization, and SiLU activation operations on the input in sequence. C2f is the main structure of Backbone for extracting features. It consists of a 1×1 convolution layer, n Bottleneck layers containing shortcut operations, and a connected Conv layer, which divides the input data into two branches for processing. One branch directly transmits the output, and the other branch is processed by multiple Bottleneck modules. Such a branch design helps to increase the nonlinear ability and representation ability of the network, thereby improving the network's modeling ability for complex data, effectively reducing feature redundancy problems, efficiently implementing feature extraction operations, and ensuring the accuracy and speed of reasoning. Spatial pyramid pooling SPP performs pooling operations of different sizes on the feature map and integrates the results to obtain a fixed-size output. It can effectively handle targets with varying sizes, thereby improving the generalization ability and robustness of the neural network. SPPF is not for obtaining a fixed-size representation layer, but for Pooling at different sizes.

[0065] The Neck structure increases the flow of feature information between different levels through pipelines. Feature information between different levels can be transmitted to each other through FPN to achieve feature fusion. Since it can generate multi-scale feature representations, it works well when processing objects of different sizes.

[0066] The head structure includes three detection layers. The input of each detection layer is a pixel feature map of size 80*80, 40*40 and 20*20, which are used for detection images of different sizes. On the basis of this target detection frame, the unique branch of the Pose detection head is extended to predict key points. This experiment includes a category of red ripe tomatoes and three key points, so each detection layer outputs a 14-channel vector, including 1 confidence, 4 bounding box coordinates and 3 key points (x coordinate, y coordinate and visibility), so as to realize the detection of image targets and key points.

[0067] The deep network has a large receptive field and strong semantic information representation ability, but the resolution of the feature map is low and the geometric information representation ability is weak (spatial geometric feature details are lacking). The underlying network has a small receptive field table and strong geometric detail information representation ability. Although the resolution is high, the semantic information representation ability is weak. Therefore, BiFPN is introduced to perform feature fusion operations. The BiFPN structure is as follows: Figure 3As shown in the figure, different feature input resolutions are different, and their contributions to the final feature network output will also be different, so the network needs to learn the weights. BiFPN can achieve bidirectional cross-scale information flow and fast normalized feature fusion between high-level and low-level feature information, including: inputting feature maps of different levels into the BiFPN_Concat layer (taking BiFPN_Concat2 as an example), given two feature maps Where C1 and C2 are the number of input channels, H and W represent the height and width of the feature map respectively. The BiFPN_Concat2 module weight is initialized to w = [1, 1], giving each input feature map the same initial importance, and normalizing it each time it is forward propagated. Set ∈ = 0.0001 to prevent the denominator from returning to zero. The normalized weight calculation formula is:

[0068]

[0069] In BiFPN_Concat2, n = 2. After normalization, weight = [weight0, weight1], and weight0 + weight1 = 1. The two input feature maps are weighted with the normalized weights: In the above operation, × is element-wise multiplication. During the training process, the model will optimize w based on the loss feedback, and then change the weight. Then BiFPN_Concat2 will concatenate the two weighted feature maps in the channel dimension to obtain the output feature map:

[0070]

[0071] at this time The feature information from F0 and F1 is successfully integrated.

[0072] In this case study, the method for detecting key points for picking red-ripe tomato fruits based on a multi-scale attention mechanism is used. The features output by the 9th layer are input into the LSKA module for processing. The structure of LSKA is shown in the figure below. Figure 2 As shown in Figure 2, large weight kernels can obtain large effective receptive fields (ERFs). LSKA decomposes the DW-Conv with a kernel size of k*k into two serial 1D DW-Convs with kernel sizes of 1×(2d-1) and (2d-1)×1 and two serial 1D DW-Convs with kernel sizes of and DW-D-Conv, after 1×1Conv output, the deep dilation convolution is responsible for capturing the global spatial information of the deep convolution output, which can achieve an effective receptive field equivalent to that of a large weight kernel while alleviating the quadratic increase in computational cost caused by using a large kernel alone for deep convolution. Specifically, given an input feature map F∈RC×H×W , where C is the number of input channels, H and W represent the height and width of the feature map respectively, and the output of LSKA is as follows:

[0073]

[0074] A C =W 1×1 *Z C (5)

[0075]

[0076] Where d is the expansion rate, k represents the maximum receptive field, and * and represent convolution and Hadamard product respectively, represents the output of two concatenated 1D DW-Conv kernels of size 1×(2d-1) and (2d-1)×1, respectively. C represents 2 1D kernels in series with size and The output of the DW-D-COnv, Indicates rounding down, A C Represents Z C The output of the 1×1 convolution is: Represents feature map A C And the input feature map F C The result of Hadamard operation.

[0077] The experimental environment in this implementation case: The operating platform of this experiment is Window11 operating system, the CPU is a 13th Gen Intel(R) Core(TM) i9-13900HX, the GPU is NVIDIA GeForce 4060, 16G memory, and 1TB solid-state storage. The training environment is created by Anaconda3, and the environment configuration is Python 3.10.9, Pytorch 2.1.0 and CUDA 12.1 GPU computing platform, cuDNN6.5 deep neural network acceleration library.

[0078] The training parameters of the method for detecting key points for picking red-ripe tomato fruits based on a multi-scale attention mechanism described in this implementation case are set as follows: the input image size is 640×640 pixels, the batch size is 32, the learning rate is initialized to 0.01 and finally to 0.032, the batch size is 100, and other parameters remain default.

[0079] When obtaining the initial data set, the first data set used was an open data set for industrial tomato detection in complex natural environments (https: / / www.scidb.cn / ). This data set was taken manually and the data came from the tomato industry planting base in Bayingolin Mongol Autonomous Prefecture and Jing County, Xinjiang Uygur Autonomous Region. It takes into account the challenges of changes in light, distance, occlusion, etc. in various natural environments. All pictures are in jpg format. A total of 240 images containing red ripe tomatoes and each tomato target contains the key points of picking, the key points of the fruit stalk, and the key points of the bottom of the tomato are selected in the data set. Labelme is used to manually annotate the data. The labeling process is as follows: Figure 6 , marked in JSON format, and converted to TXT format through a script program; the label category is tomato, and the key point categories are P1, P1_hide, P2, P2_hide, P3, and P3_hide, where the hide suffix indicates that the key point exists in the image but is blocked. The input resolution of the image is 4000×3000.

[0080] The implementation case describes a method for detecting key points for picking red-ripe tomato fruits based on a multi-scale attention mechanism. Key point annotations include picking key points (P1, P1_hide), fruit stalk key points (P2, P2_hide) and tomato bottom key points (P3, P3_hide). The picking key points are selected at a position of 1cm-2cm along the growth direction of the fruit stem at the fruit stalk key point, the fruit stalk key point is the position of the tomato stalk, and the bottom key point is the center position of the bottom of the tomato.

[0081] In the method for detecting key points for picking red-ripe tomato fruits based on a multi-scale attention mechanism described in this implementation case, data enhancement preprocessing refers to the use of one or more of random rotation, flipping, and adding Gaussian noise; the number of data sets is expanded to 960 through data enhancement preprocessing, and the training set, test set, and validation set are divided into 8:1:1 ratios.

[0082] In the method for detecting key points for picking red-ripe tomato fruits based on a multi-scale attention mechanism described in this implementation case, the random rotation is specifically: selecting an image and rotating it at a random angle of 15° or -15° clockwise; the flipping is specifically: selecting an image and flipping it 180°; the adding of Gaussian noise is specifically: selecting an image and randomly performing Gaussian blur processing with a kernel size of 11 or 7.

[0083] In order to verify the effectiveness of the proposed model in detecting tomatoes, the commonly used quantitative indicators in the field of target detection are used to compare and analyze its detection performance of tomato fruits using accuracy P, average precision mAP and recall R; the specific formulas are shown in formulas (7) to (9):

[0084]

[0085] Where: tP is the number of correctly detected frames, FP is the number of incorrectly detected frames, and FN is the number of missed targets. AP is the average accuracy of a certain category, and mAP is the average AP of all detection categories. The specific calculation formula is as follows:

[0086]

[0087] Where: N is the number of target categories that need to be detected. In this experiment, only one type of red-ripe tomatoes needs to be detected, so N=1.

[0089] The target key point similarity OKS is obtained through detection, and the average accuracy AP can be obtained from OKS. mAP-kp is calculated from AP as the evaluation index of the key point.

[0090] OKS p The calculation formula is:

[0091]

[0092] Where, d pi ——The Euclidean distance between the predicted value and the true value of the i-th key point of the p-th target

[0093] v pi ——The visibility index of the i-th key point of the p-th target

[0094] s p ——The area of ​​the pth target bounding box

[0095] δ i ——The standard deviation between the labeled value and the actual value of the i-th key point.

[0096] Example 2

[0097] See attached Figure 6 The present embodiment relates to a device for detecting key points for picking red-ripe tomato fruits based on a multi-scale attention mechanism, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a method for detecting key points for picking red-ripe tomato fruits based on a multi-scale attention mechanism in Example 1.

[0098] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A method for detecting key points for picking red ripe tomato fruits based on a multi-scale attention mechanism, comprising the following steps: Step 1, construct an improved YOLOv8 model based on LSKA attention mechanism and BiFPN feature fusion module, the improved YOLOv8 model includes Backbone, Neck and Head, the Backbone includes Conv module, C2f module and SPPF module; the Neck includes upsample layer, Concat layer, C2f module and Conv module, and the BiFPN feature fusion module is used to replace the Concat layer in the Neck; Step 2: Obtain an initial dataset of red ripe tomato images, perform data enhancement preprocessing on the images in the initial dataset, and obtain an enhanced dataset; Step 3: Divide the enhanced dataset into training, validation, and test sets, use the training set to train the improved YOLOv8 model, and use the validation and test sets to evaluate the trained improved YOLOv8 model, including: Step 3.1, use the Backbone structure in the improved YOLOv8 model constructed in step 1 to extract the feature vector of the picture in the training set, and output it in the 4th, 6th and 9th layers. The features output by the 9th layer are input into the LSKA module for processing; then output the processed features into the Neck structure; use CIoU as the bounding box loss function, and adopt non-maximum suppression; use the official definition class KeypointLoss loss function and BCEWithLogitsLoss loss function to calculate the key point position loss and key point confidence loss; Step 3.2: Use the validation and test sets to evaluate the trained model. The evaluation indicators include accuracy P, mean average precision mAP, recall R, and mean average precision mAP-kp of key points. Step 4: Use the trained improved YOLOv8 model to perform target recognition and key point detection on tomatoes in a real planting environment.

2. According to claim 1, a method for detecting key points for picking red ripe tomato fruits based on a multi-scale attention mechanism is characterized in that: In step 1, BiFPN is used to perform feature fusion operations. Different feature inputs have different resolutions, and their contributions to the final feature network output will also be different, so the network needs to learn the weights. BiFPN realizes bidirectional cross-scale information flow and fast normalized feature fusion between high-level and low-level feature information, including: inputting feature maps of different levels into the BiFPN_Concat layer (taking BiFPN_Concat2 as an example), given two feature maps Where C1 and C2 are the number of input channels, H and W represent the height and width of the feature map respectively. The BiFPN_Concat2 module weight is initialized to w = [1, 1], giving each input feature map the same initial importance, and normalizing it each time it is forward propagated. Set ∈ = 0.0001 to prevent the denominator from returning to zero. The normalized weight calculation formula is: In BiFPN_Concat2, n = 2; after normalization, weight = [weight0, weight1], and weight0 + weight1 = 1, and the two input feature maps are weighted with the normalized weights: In the above operation, × is element-wise multiplication. During training, the model optimizes w based on loss feedback, thereby changing weight. Then BiFPN_Concat2 concatenates the two weighted feature maps in the channel dimension to obtain the output feature map: at this time The feature information from F0 and F1 is successfully integrated.

3. The method for detecting key points for picking red ripe tomato fruits based on a multi-scale attention mechanism according to claim 1, characterized in that: In step 2, the initial dataset of red-ripe tomatoes was obtained, and Labelme was used to manually annotate the data, marked in JSON format, and converted into TXT format through a script program; the label category was tomato, and the key point categories were P1, P1_hide, P2, P2_hide, P3, and P3_hide, where the hide suffix indicated that the key point existed in the image but was obscured.

4. The method for detecting key points for picking red ripe tomato fruits based on a multi-scale attention mechanism according to claim 3, characterized in that: The input resolution of the image is 4000×3000.

5. The method for detecting key points of red ripe tomato fruit picking based on a multi-scale attention mechanism according to claim 1, characterized in that: The data enhancement preprocessing in step 2 refers to one or more of random rotation, flipping and adding Gaussian noise.

6. The method for detecting key points of red ripe tomato fruit picking based on a multi-scale attention mechanism according to claim 5, characterized in that: The random rotation is specifically: selecting a picture to randomly rotate at an angle of 15° or -15° clockwise; the flipping is specifically: selecting a picture to flip 180°; and the adding of Gaussian noise is specifically: selecting a picture to randomly perform Gaussian blur processing with a kernel size of 11 or 7.

7. The method for detecting key points of red ripe tomato fruit picking based on multi-scale attention mechanism according to claim 1, characterized in that: In step 3.1, the features output by the 9th layer are input into the LSKA module for processing; the large weight kernel obtains a large effective receptive field (ERFs), and LSKA decomposes the DW-Conv with a kernel size of k*k into two serial 1D DW-Conv with kernel sizes of 1×(2d-1) and (2d-1)×1 and two serial 1D DW-Conv with kernel sizes of and DW-D-Conv, after 1×1Conv output, the deep dilated convolution is responsible for capturing the global spatial information of the deep convolution output, achieving an effective receptive field equivalent to that of a large weight kernel while alleviating the quadratic increase in computational cost caused by using a large kernel alone for deep convolution; specifically, given an input feature map F∈R C×H×W , where C is the number of input channels, H and W represent the height and width of the feature map respectively, and the output of LSKA is as follows: AND C =In 1×1 *WITH C (5) Where d is the expansion rate, k represents the maximum receptive field, and * and represent convolution and Hadamard product respectively, represents the output of two concatenated 1D DW-Conv kernels of size 1×(2d-1) and (2d-1)×1, respectively. C represents 2 1D kernels in series with size and The output of the DW-D-COnv, Indicates rounding down, A C Represents Z C The output of the 1×1 convolution is: Represents feature map A C And the input feature map F C The result of Hadamard operation.

8. The method for detecting key points of red ripe tomato fruit picking based on multi-scale attention mechanism according to claim 1, characterized in that: The training parameters in step 3.1 are set as follows: the input image size is 640×640 pixels, the batch size is 32, the learning rate is initialized to 0.01 and finally to 0.032, the batch size is 100, and other parameters remain default.

9. The method for detecting key points of red ripe tomato fruit picking based on multi-scale attention mechanism according to claim 1, characterized in that: In step 3.2, the accuracy P, average precision mAP and recall R are used to compare and analyze the detection performance of tomato fruit; the specific formulas are shown in formulas (7) to (9): Where: TP is the number of correctly detected frames, FP is the number of incorrectly detected frames, FN is the number of missed targets; AP is the average accuracy of a certain category, mAP is the average AP of all detection categories, and the specific calculation formula is as follows: Where: N is the number of target categories that need to be detected. In this experiment, only one type of red ripe tomatoes needs to be detected, so N = 1; The target key point similarity OKS is obtained through detection, and the average accuracy AP is obtained from OKS. mAP-kp is calculated from AP as the evaluation index of the key point; OKS p The calculation formula is: Where, d pi ——The Euclidean distance between the predicted value and the true value of the i-th key point of the p-th target v pi ——The visibility index of the i-th key point of the p-th target s p ——The area of ​​the pth target bounding box δ i ——The standard deviation between the labeled value and the actual value of the i-th key point.

10. A device for detecting key points of red ripe tomato fruit picking based on a multi-scale attention mechanism, characterized in that: It includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement a method for detecting key points for picking red-ripe tomato fruits based on a multi-scale attention mechanism as described in any one of claims 1-9.

Images