Real-time tomato image detection method and system, tomato picking method and system

Abstract

A real-time tomato image detection method and system, as well as a tomato picking method and system, relate to the fields of intelligent crop harvesting and deep learning technology. These methods address the challenges of existing tomato detection technologies in adaptability, real-time performance, and accuracy in complex environments. A tomato image dataset is obtained and preprocessed to generate a training set of tomato images. A SWMD-YOLO model is constructed by replacing the standard convolution in the YOLO11 backbone network with SAConv and introducing MA-WTConv. Downsampling in the YOLO11 neck network is replaced with DySample and MSCA is introduced. The tomato image to be detected is fed into the SWMD-YOLO model to complete real-time tomato detection.

Description

Technical Field

[0001] The present invention relates to the fields of intelligent crop picking and deep learning technology, and particularly to a real-time tomato image detection method and system, and a tomato picking method and system. Background Art

[0002] The harvesting robot first uses a computer vision system to detect the fruit, then guides the robotic arm to harvest based on the detection results. Therefore, fruit detection is a critical step in the entire harvesting process, and its accuracy and speed directly impact the efficiency of the harvesting robot. However, tomato fruits exhibit diverse growth postures, overlap, and are heavily obscured by leaves, branches, and stems, posing certain challenges to robot recognition. Rapid and accurate identification of tomato fruits in complex greenhouse environments is a pressing issue in the development of tomato harvesting robots.

[0003] With the development of information technology, the widespread use of digital information transmission and the Industrial Internet of Things (IIoT) has made agricultural production management and monitoring more efficient and intelligent. In the field of crop harvesting, traditional manual picking methods are no longer able to meet the efficiency and precision requirements of modern agriculture. To address this, automated tomato inspection systems based on computer vision and deep learning technologies have emerged. Unlike traditional methods, these systems can analyze video streams or image data in real time to accurately identify ripe tomatoes and guide harvesting operations, significantly improving harvesting efficiency and reducing labor costs.

[0004] With advances in artificial intelligence and machine learning, deep learning-based tomato inspection technology has emerged. This technology offers a more efficient and accurate means of inspecting tomatoes. It not only analyzes image data in real time with minimal impact on crop growth, but also withstands certain environmental interference. This is crucial for improving harvesting efficiency and ensuring crop quality. Intelligent inspection systems enable farmers to better grasp harvest timing, maximizing yields and economic benefits, and driving agricultural production towards more refined and intelligent processes. However, existing technologies still have shortcomings in adaptability, real-time performance, and accuracy in complex environments. Summary of the Invention

[0005] The present invention solves the problems of adaptability, real-timeness and accuracy of existing tomato detection technology in complex environments.

[0006] The real-time tomato image detection method of the present invention comprises the following steps:

[0007] Step S1, obtaining a tomato image dataset, preprocessing the tomato image dataset, and obtaining a tomato image training set;

[0008] Step S2: Build the SWMD-YOLO model, specifically:

[0009] The standard convolution in the YOLO11 backbone network is replaced with SAConv, and MA-WTConv is introduced. The downsampling in the YOLO11 neck network is replaced with DySample, and MSCA is introduced.

[0010] In step S3, the tomato image to be detected is input into the SWMD-YOLO model to complete the real-time detection of the tomato.

[0011] Furthermore, in one embodiment of the present invention, in step S2, the MA-WTConv is specifically:

[0012] The tomato image is divided into four wavelet subbands. The low-frequency component, horizontal high-frequency component, vertical high-frequency component and diagonal high-frequency component are input into the four wavelet subbands for convolution operation respectively. The four wavelet subbands after the convolution operation are integrated through inverse wavelet transform and the integration result is output.

[0013] The convolution kernels of low-frequency components, horizontal high-frequency components, vertical high-frequency components and diagonal high-frequency components are designed respectively, and the convolution paths of high-frequency components, vertical high-frequency components and diagonal high-frequency components are independent.

[0014] Furthermore, in one embodiment of the present invention, the convolution kernel of the low-frequency component is specifically:

[0015]

[0016] in, To use the void rate d i 3×3 multi-scale dilated convolution operation, d i For different expansion coefficients, X LL is the low-frequency component of X, X is the input feature map, is the low-frequency feature response image extracted using different void rates, which represents the response to the input X at the current scale. LL The structural information modeling results, LL is the low-frequency component;

[0017]

[0018] Among them, F′ LL It is the intermediate feature map after the multi-scale dilated convolution output features are spliced ​​in the channel dimension, which is used to fuse the low-frequency feature responses under different receptive fields. Concat is the feature splicing module. are the results of three groups of multi-scale dilated convolutions, Conv 1×1 It is a 1×1 convolution;

[0019]

[0020] in, Enhance the feature map for the final low-frequency components.

[0021] Furthermore, in one embodiment of the present invention, the convolution kernel of the horizontal high-frequency component is specifically:

[0022]

[0023] in, is the horizontal high-frequency component enhanced feature map extracted after the 1×5 asymmetric convolution operation, Conv 1×5 is a 1×5 asymmetric convolution, X LH is the horizontal high-frequency component of X, X is the input feature map, and LH is the horizontal high-frequency component.

[0024] Furthermore, in one embodiment of the present invention, the convolution kernel of the vertical high-frequency component is specifically:

[0025]

[0026] in, Conv is the vertical high-frequency component enhanced feature map extracted after the 5×1 asymmetric convolution operation. 5×1 is a 5×1 asymmetric convolution, X HL is the vertical high-frequency component of X, X is the input feature map, and HL is the vertical high-frequency component.

[0027] Furthermore, in one embodiment of the present invention, the convolution kernel of the diagonal high-frequency component is specifically:

[0028]

[0029] in, For X HH The enhanced feature map obtained by fusing 3×3 lightweight group convolution with 1×1 convolution, GroupConv 3×3 To process X using 3×3 lightweight group convolution HH , X HH is the diagonal high-frequency component of X, X is the input feature map, Conv 1×1 is a 1×1 convolution, and HH is the diagonal high-frequency component.

[0030] Furthermore, in one embodiment of the present invention, the four wavelet sub-bands after the convolution operation are integrated by inverse wavelet transform, specifically:

[0031]

[0032] in, is the low-frequency component feature map obtained by wavelet decomposition of the i-th layer, is the horizontal high-frequency component feature map obtained by wavelet decomposition of the i-th layer, is the vertical high-frequency component feature map obtained by wavelet decomposition of the i-th layer, is the diagonal high-frequency component feature map obtained by the wavelet decomposition of the i-th layer, i is the level index of the current wavelet decomposition, WT is the wavelet transform operation, is the low-frequency component feature map obtained by the wavelet decomposition of the i-1 layer, is the horizontal high-frequency component feature map obtained by the wavelet decomposition of the i-1 layer, is the vertical high-frequency component feature map obtained by the wavelet decomposition of the i-1 layer, is the diagonal high-frequency component feature map obtained by the wavelet decomposition of the i-1 layer, X LL is the low-frequency component of X, X LH is the horizontal high-frequency component of X, X HL is the vertical high-frequency component of X, X HH is the diagonal high-frequency component of X, X is the input feature map, LL is the low-frequency component, LH is the horizontal high-frequency component, HL is the vertical high-frequency component, and HH is the diagonal high-frequency component.

[0033] The tomato picking method of the present invention is implemented based on any of the above methods for real-time detection of tomato images, and includes the following steps:

[0034] Step S4: Input the tomato image to be detected into the SWMD-YOLO model to complete real-time detection of the tomato, record the bounding box coordinates of the tomato output by the SWMD-YOLO model, and obtain the center coordinates of the tomato;

[0035] In step S5, the obtained tomato center coordinates are used as the target position for the robot arm path planning. The RRT algorithm is used to randomly sample and expand the tree structure from the current position of the robot arm to explore the feasible path from the current position to the target position, and complete the tomato picking.

[0036] The real-time tomato image detection system of the present invention includes the following modules:

[0037] The acquisition module obtains the tomato image dataset, preprocesses the tomato image dataset, and obtains the tomato image training set;

[0038] Construct the module and build the SWMD-YOLO model, specifically:

[0039] The standard convolution in the YOLO11 backbone network is replaced with SAConv, and MA-WTConv is introduced. The downsampling in the YOLO11 neck network is replaced with DySample, and MSCA is introduced.

[0040] The detection module inputs the tomato image to be detected into the SWMD-YOLO model to complete real-time detection of tomatoes.

[0041] The tomato picking system of the present invention is implemented based on the real-time tomato image detection system and includes the following modules:

[0042] The recording module inputs the tomato image to be detected into the SWMD-YOLO model to complete real-time detection of the tomato, and records the bounding box coordinates of the tomato output by the SWMD-YOLO model to obtain the center coordinates of the tomato;

[0043] The picking module uses the obtained tomato center coordinates as the target position for the robot arm's path planning. It uses the RRT algorithm to start from the current position of the robot arm, randomly sample and expand the tree structure, explore feasible paths from the current position to the target position, and complete the tomato picking.

[0044] The present invention solves the problems of adaptability, real-time performance, and accuracy in existing tomato detection technologies in complex environments. Specific beneficial effects include:

[0045] 1. The real-time detection method for tomato images described in the present invention, the existing tomato detection technology has problems in adaptability, real-timeness and accuracy in complex environments. In order to solve the above technical problems, SAConv and MA-WTConv are introduced into the backbone network of YOLO11, and the original standard convolution module is replaced by the enhanced SAConv. By dynamically adjusting the receptive field size and decomposing local and global features, the detection ability of small targets is enhanced, especially when the background is complex or the target size is small, it can effectively improve the model's capture and classification accuracy of small tomatoes. And in the neck network of YOLO11, the downsampling operation is replaced by DySampl e. DySample can adaptively adjust the upsampling strategy according to the input features, while reducing feature loss and more efficiently restoring the feature map detail information. Through multi-scale feature fusion, the model's recognition ability of occluded tomatoes is significantly enhanced. This addresses the occlusion problem and the challenge of feature fusion in complex scenes;

[0046] 2. The real-time tomato image detection method described in this paper introduces MSCA into the YOLO11 neck network. This module dynamically adjusts attention weights by combining multi-scale contextual information, effectively enhancing the feature expression of key areas while suppressing background noise interference, further improving the object detection accuracy in occluded and complex scenes.

[0047] 3. The real-time tomato image detection method described in this invention, combined with DySample and MSCA, enables the model to more accurately restore the characteristic edges of small objects, thereby improving the robustness and accuracy of detection;

[0048] 4. The tomato picking method described in the present invention, although the existing models can solve the efficiency problem of manual tomato picking, these models are either not real-time strong, and the complex processing leads to slow detection speed, or the recognition accuracy is not high, and it is impossible to accurately identify the tomatoes that can be picked, and these models are large in size and difficult to deploy. In order to solve the above technical problems, the present invention uses the SWMD-YOLO model to perform real-time detection of targets in tomato images. This model solves the problems of existing models, such as either not real-time strong, and the complex processing leads to slow detection speed, or the recognition accuracy is not high, and it is impossible to accurately identify the tomatoes that can be picked, and these models are large in size and difficult to deploy. RRT is introduced for robot arm path planning, which can adjust the path in real time to ensure that the robot arm can still accurately reach the target tomato when the environment changes. This dynamic path adjustment capability greatly improves the robustness and stability of the system in practical applications;

[0049] The real-time tomato image detection method described in this paper is of great significance to the field of intelligent tomato harvesting. Based on the SWMD-YOLO model, this real-time tomato detection method effectively solves the problem of low recognition accuracy for occluded cherry tomatoes. It can accurately and quickly identify tomato information, and offers advantages in accuracy, robustness, and ease of implementation. This provides farmers with an efficient tomato harvesting solution and is expected to be widely adopted in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0051] Figure 1 1 is a structural diagram of the SWMD-YOLO model described in Implementation 1;

[0052] Figure 2 1 is a diagram of the SAConv structure described in Embodiment 1;

[0053] Figure 3 is a schematic diagram of the DySample described in the first embodiment;

[0054] Figure 4 is a structural diagram of the MSCA described in the first embodiment;

[0055] Figure 5 is an example diagram comparing the original image and the image after data enhancement in the data set described in Implementation Method 1;

[0056] Figure 6 This is an example diagram of the actual scene detection effect described in the third embodiment;

[0057] Figure 7 This is the ablation data graph described in the first embodiment. DETAILED DESCRIPTION

[0058] The following will clearly and completely describe various embodiments of the present invention in conjunction with the accompanying drawings. The embodiments described with reference to the accompanying drawings are exemplary and intended to be used to explain the present invention, but should not be understood as limiting the present invention.

[0059] Implementation 1: The real-time tomato image detection method described in this implementation includes the following steps:

[0060] Step S1, obtaining a tomato image dataset, preprocessing the tomato image dataset, and obtaining a tomato image training set;

[0061] Step S2: Build the SWMD-YOLO model, specifically:

[0062] The standard convolution in the YOLO11 backbone network is replaced with SAConv, and MA-WTConv is introduced. The downsampling in the YOLO11 neck network is replaced with DySample, and MSCA is introduced.

[0063] In step S3, the tomato image to be detected is input into the SWMD-YOLO model to complete the real-time detection of the tomato.

[0064] Existing tomato detection technology has problems in adaptability, real-timeness and accuracy in complex environments.

[0065] To solve the above technical problems, this embodiment proposes a real-time tomato image detection method based on an improvement to the YOLO11 model (the 11th version of the YOLO (You Only Look Once) model), which includes the following steps:

[0066] Step S1: Obtain an open-source tomato dataset. The dataset is labeled with a category label and an occlusion degree label, specifically:

[0067] The tomato dataset, which is publicly available on Kaggle, contains 895 images. The class label is tomato, and the occlusion level labels include no occlusion, partial occlusion, and full occlusion.

[0068] Step S2: Configure the model training environment, specifically:

[0069] The acceleration computing structure is cuda12.4, the deep learning architecture is pytorch2.4.1, the CPU is Intel core i7-12700kf, the memory is 128G, and the GPU (graphics card) is NVIDIA GeForce RTX 4070Ti SUPER with 16G video memory.

[0070] Step S3, preprocessing the tomato image dataset to obtain a tomato image training set;

[0071] like Figure 5 As shown, the preprocessing is as follows: considering using brightness changes, saturation changes, contrast changes, Gaussian noise and salt and pepper noise to enhance the tomato image dataset, a dataset of 4475 tomato images is finally obtained. The training set, validation set and test set are divided into a ratio of 7:2:1. 3200 tomato images are used for model training, 895 tomato images are used for model validation, and 380 tomato images are used for model testing.

[0072] In data augmentation, noise transformation involves randomly adding Gaussian and salt-and-pepper noise of varying intensities to the image to simulate image quality fluctuations that can occur during high temperatures in a greenhouse. Gaussian noise transformation involves randomly adding noise to different regions of the image with a mean of 0 and a standard deviation of 25 to simulate lighting variations and equipment noise in a greenhouse environment. Salt-and-pepper noise transformation involves randomly adding white and black pixels to random locations in the image with a probability of 0.1 to simulate image sensor failure or other abnormal interference.

[0073] Step S4: construct the SWMD-YOLO model (image target detection model), specifically:

[0074] like Figure 1 As shown, the SWMD-YOLO model uses the YOLO11 model as the basic model. The YOLO11 model includes a backbone network, a neck network, and a head network. The standard convolution in the YOLO11 backbone network is replaced with SAConv (switchable hole convolution layer), and MA-WTConv (multi-scale adaptive wavelet convolution layer) is introduced. The downsampling in the YOLO11 neck network is replaced with DySample (dynamic upsampling module), and MSCA (multi-scale attention module) is introduced.

[0075] For SAConv, such as Figure 2As shown, the network consists of three parts. The first part is the Pre-Global Context. The feature map passes through a 1×1 point-by-point convolution layer and a global average pooling layer to compress and enhance information between channels and extract global context. The second part is the Switchable Atrous Convolution, which has two parallel branches. The dilated convolution branch uses 3×3 dilated convolutions with a dilation rate of 1 and different dilation rates. One branch has a dilation rate of 1, and the other has a dilation rate of 3. This helps expand the receptive field of the convolution without increasing computational cost, capturing richer multi-scale information. The average pooling branch performs a 5×5 average pooling layer to further capture local information. Finally, the results of the two branches are integrated through a 1×1 point-by-point convolution layer. The third part is the Post-Global Context. After processing through a 1×1 point-by-point convolution layer and a global average pooling layer, global information is further integrated. Switchable mechanism: Through a dynamic weighting parameter S, it is possible to switch between the results of the 3×3 dilated convolution and the 5×5 average pooling layer. Depending on the different requirements of the task, the module can adaptively adjust its focus on local details and global features. The specific 3×3 dilated convolution calculation formula is:

[0076]

[0077] Where y is the output of the 3×3 dilated convolution operation, X is the input feature map, and w is the weight matrix (convolution kernel) used for the 3×3 dilated convolution operation. Δw is the offset of the 3×3 dilated convolution kernel, r is the dilation ratio of the 3×3 dilated convolution, S(X) is a function that calculates a value based on the input feature map X to control whether to use the standard convolution kernel w or the modified convolution kernel Δw. Convert is the convolution path control function used to dynamically switch between the standard convolution and offset dilated convolution paths based on the value of S(X). Conv is a 3×3 dilated convolution.

[0078] like Figure 3 As shown in the figure, DySample uses point sampling to achieve dynamic upsampling. First, the input feature map of a given size of C×H×W is converted into a sampling set S of size 2×sH×sW through the sampling point generator, where the 2 in the first dimension of the sampling set S represents the x and y coordinates. Then, the gridsample module (network sampling module) is used to resample the input features to obtain an upsampled feature map of size C×sH×sW. In the design of DySample, the following formula is used to dynamically adjust the weights:

[0079] w i,j =S(X)1·w base +(1-S(x)1)·wsmooth ;

[0080] Among them, w i,j is the DySample convolution kernel weight applied at position i, j, w base is the basic convolution kernel weight, w smooth is the smoothed convolution kernel weight, and S(X)1 is the local response of the image.

[0081] like Figure 4 As shown, the specific process expression of MSCA is as follows:

[0082]

[0083] Among them, F is the input feature, Att and Out are the input and output of MSCA respectively, Scale is an element-by-element matrix multiplication operation. i , i∈{0,1,2,3} is the i-th branch, and DW-Conv is depthwise convolution.

[0084] Scale 0 is the identity connection. Two depthwise strip convolutions are used to approximate the standard depthwise convolution with large kernels. The kernel size of each branch is set to 7, 11, and 21 respectively.

[0085] In order to verify whether each module plays a role in the SWMD-YOLO model, such as Figure 7 As shown, the ablation experiment results are analyzed below:

[0086] Ablation experiments show that the SWMD-YOLO model improved mAP50 by 0.69% after introducing SAConv, from 91.36% to 92.04%, while also increasing FPS from 65.48 to 76.52. Furthermore, the introduction of MA-WTConv further improved mAP50 by 0.55%, from 91.67% to 92.22%, while decreasing FPS from 68.95 to 65.48. Subsequently, the introduction of MSCA further improved mAP50 by 0.61%, from 92.24% to 92.85%, while increasing mAP50-95 from 61.46% to 65.74%, and decreasing FPS from 70.21 to 63.91. To further improve performance while limiting parameter usage and computational resource consumption, MA-WTConv and MSCA were combined. The mAP50 score reached 93.47%, a 1.25% improvement over MA-WTConv alone and a 0.63% improvement over MSCA alone. mAP50-95 improved from 61.46% to 65.74%, and FPS increased from 70.21 to 72.03. This combination allows the SWMD-YOLO model to maintain high performance while achieving superior computational efficiency and detection speed.

[0087] Step S5: Load the constructed SWMD-YOLO model into the configured model training environment, modify the model parameter file, train the SWMD-YOLO model using the training set, and evaluate the trained SWMD-YOLO model using the validation set and test set. Specifically:

[0088] Step S501: Use the backbone network in the YOLO11 model constructed in step S3 to extract feature vectors of the tomato images in the training set and output them at the 5th, 7th, and 11th layers. The features output from the 11th layer are input to the C2PSA (spatial pyramid pooling layer) for processing. The processed features are then output to the neck network. DySample is combined with Concat (feature concatenation module) and C3k2 (three-layer 2×2 convolution module) to fuse multi-scale features. MSCA is introduced into the head network to optimize the attention of the P3, P4, and P5 feature layers. Finally, Detect (output detection module) is used for object classification and bounding box regression.

[0089] The training process was repeated 300 times, with 16 tomato images input per round. During training, the training log was observed in real time using Tensorboard (a data visualization tool), and the weight files were saved after training was complete.

[0090] Step S502: After training is completed, the trained SWMD-YOLO model is evaluated using the validation set and the test set. The evaluation indicators used include accuracy, recall, PR curve (precision-recall curve), AP (average precision), mAP50-95, and FPS (frames per second), specifically:

[0091]

[0092] FPS = 1 / T;

[0093] Among them, TP is the number of images in which the SWMD-YOLO model correctly detects tomato targets, FP is the number of images in which the SWMD-YOLO model incorrectly detects non-tomato targets, FN is the number of images in which the SWMD-YOLO model does not detect tomato targets, Precision is the accuracy, Recall is the recall rate, and the PR curve is constructed using the accuracy and recall rates. The area under the curve is recorded as AP. mAP50-95 is the AP with an IoU (intersection over union) threshold from 0.5 to 0.95, and T is the detection time for a single image.

[0094] In step S6, the tomato image to be detected is used as input, and the trained SWMD-YOLO model is applied to perform real-time detection of tomatoes.

[0095] Therefore, this embodiment develops a SWMD-YOLO model for tomato image detection that can identify occluded conditions. This model not only improves the accuracy and efficiency of tomato image detection, but also promotes the development of smart agriculture, optimizes agricultural production processes, improves the market competitiveness of crops, and helps achieve the goal of smart agriculture.

[0096] Implementation 2: This implementation further limits the real-time tomato image detection method described in Implementation 1. In step S2, the MA-WTConv is specifically:

[0097] The tomato image is divided into four wavelet subbands. The low-frequency component, horizontal high-frequency component, vertical high-frequency component and diagonal high-frequency component are input into the four wavelet subbands for convolution operation respectively. The four wavelet subbands after the convolution operation are integrated through inverse wavelet transform and the integration result is output.

[0098] The convolution kernels of low-frequency components, horizontal high-frequency components, vertical high-frequency components and diagonal high-frequency components are designed respectively, and the convolution paths of high-frequency components, vertical high-frequency components and diagonal high-frequency components are independent.

[0099] In this implementation, the convolution kernel of the low-frequency component is specifically:

[0100]

[0101] in, To use the void rate d i 3×3 multi-scale dilated convolution operation, d i For different expansion coefficients, X LL is the low-frequency component of X, X is the input feature map, is the low-frequency feature response image extracted using different void rates, which represents the response to the input X at the current scale. LL The structural information modeling results, LL is the low-frequency component;

[0102]

[0103] Among them, F′ LL It is the intermediate feature map after the multi-scale dilated convolution output features are spliced ​​in the channel dimension, which is used to fuse the low-frequency feature responses under different receptive fields. Concat is the feature splicing module. are the results of three groups of multi-scale dilated convolutions, Conv 1×1 It is a 1×1 convolution;

[0104]

[0105] in, Enhance the feature map for the final low-frequency components.

[0106] In this embodiment, the convolution kernel of the high-frequency component is specifically:

[0107]

[0108] in, is the horizontal high-frequency component enhanced feature map extracted after the 1×5 asymmetric convolution operation, Conv 1×5 is a 1×5 asymmetric convolution, X LH is the horizontal high-frequency component of X, X is the input feature map, and LH is the horizontal high-frequency component.

[0109] In this embodiment, the convolution kernel of the vertical high-frequency component is specifically:

[0110]

[0111] in, Conv is the vertical high-frequency component enhanced feature map extracted after the 5×1 asymmetric convolution operation. 5×1 is a 5×1 asymmetric convolution, X HL is the vertical high-frequency component of X, X is the input feature map, and HL is the vertical high-frequency component.

[0112] In this implementation, the convolution kernel of the diagonal high-frequency component is specifically:

[0113]

[0114] in, For X HH The enhanced feature map obtained by fusing 3×3 lightweight group convolution with 1×1 convolution, GroupConv 3×3 To process X using 3×3 lightweight group convolution HH , X HH is the diagonal high-frequency component of X, X is the input feature map, Conv 1×1 is a 1×1 convolution, and HH is the diagonal high-frequency component.

[0115] In this embodiment, the existing WTConv (wavelet convolution) uses a two-dimensional Haar wavelet transform to perform multi-level decomposition on the input image. The two-dimensional Haar wavelet transform uses four filters to decompose the image into four sub-bands: Low-frequency component: Captures low-frequency information of the image, such as the overall shape or outline. Horizontal high-frequency component: Captures horizontal edge information in the image. Vertical high-frequency component: Captures vertical edge information in the image. Diagonal high-frequency component: Captures diagonal details of the image. The convolution method is shown below:

[0116]

[0117] Among them, f LL is the low-frequency filter in the two-dimensional Haar wavelet decomposition, f LH is a horizontal high-frequency filter, f HL is a vertical high frequency filter, f HH is a diagonal high-frequency filter, LL is the low-frequency component, LH is the horizontal high-frequency component, HL is the vertical high-frequency component, and HH is the diagonal high-frequency component.

[0118] The convolution operation is shown below:

[0119] [X LL ,X LH ,X HL ,X HH ]=Conv([f LL ,f LH ,f HL ,f HH ],X);

[0120] Among them, X LL is the low-frequency component of X, X LH is the horizontal high-frequency component of X, X HL is the vertical high-frequency component of X, X HH is the diagonal high-frequency component of X, X is the input feature map, and Conv is the convolution operation.

[0121] However, the existing WTConv mainly uses two-dimensional Haar wavelet transform to perform multi-level decomposition of images. Although it can extract low-frequency and high-frequency information of images to a certain extent, its filter kernel parameters are fixed and it is difficult to adapt to the complex changes of different image contents. In particular, in tomato images taken in greenhouse environments, due to factors such as strong background interference, blurred target edges, and variable fruit shapes, the frequency features extracted by the two-dimensional Haar wavelet transform have problems such as limited expression ability, insufficient directional information, and unstable feature response. As a result, the model encounters technical bottlenecks such as insufficient feature extraction and reduced detection accuracy in subsequent detection tasks.

[0122] For example, Chinese patent CN104951783A discloses a "Method and System for Underwater Sea Cucumber Image Processing Based on Wavelet Transform Algorithm." This method uses a wavelet transform algorithm to denoise underwater sea cucumber images, producing high-definition images that facilitate identification and capture of underwater sea cucumbers. However, this wavelet transform algorithm has the following drawbacks: 1) The wavelet kernel is fixed and cannot be learned, lacking adaptability, and its feature extraction capabilities are limited in complex environments (such as occlusion, light changes, and local blur); 2) The processing dimension is single, performing only one layer of wavelet decomposition, which fails to fully exploit the multi-scale features in the image and is not conducive to the simultaneous perception of detailed textures and global structure.

[0123] In order to solve the above technical problems, this embodiment improves the convolution kernel of WTConv using conventional convolution. However, after using conventional convolution to improve the convolution kernel of WTConv, the target detection of tomato images has the following defects: 1) WTConv divides the input features into multiple frequency components, especially the high-frequency component features are relatively sparse and the edge response is easily interfered by noise. If conventional convolution is directly applied, it may lead to weakening of feature information or over-smoothing, making it difficult to effectively retain key texture structures. 2) The low-frequency component is the structural backbone of the image and contains the overall contour information of most targets. Fixed kernels or standard convolutions are usually used in this channel, which makes it difficult to take into account the modeling requirements of targets of different scales.

[0124] To further enhance the model's ability to model different component features and its directional response capabilities, differentiated convolution kernel structures are designed for the four components to adapt to their respective frequency structure characteristics. This implementation further improves the convolution kernel of WTConv and designs MA-WTConv (Multi-scale Adaptive Wavelet Convolution), specifically:

[0125] 1) Low-frequency component convolution kernel design:

[0126] This component contains the main structural contour information. In order to enhance the modeling ability of the overall target morphology (such as the tomato edge and occlusion relationship), multi-scale dilated convolution is introduced. The convolution method is shown in the following formula:

[0127]

[0128] in, To use the void rate d i 3×3 multi-scale dilated convolution operation, d i For different expansion coefficients (e.g., 1, 2, 3), the void ratio is not set arbitrarily, but is designed based on the common size scales in tomato images to effectively cover small fruits, large fruits, and fuzzy structures. is the low-frequency feature response image extracted using different void rates, which represents the response to the input X at the current scale. LL This operation can expand the receptive field without increasing the number of parameters and enhance the perception of target structures at different scales.

[0129] Next, the results of the multi-scale dilated convolution output are subjected to feature fusion, as shown in the following formula:

[0130]

[0131] Among them, F′ LL It is the intermediate feature map after the multi-scale dilated convolution output features are spliced ​​in the channel dimension, which is used to fuse the low-frequency feature responses under different receptive fields. Concat is the feature splicing module. are the results of three groups of multi-scale dilated convolutions, Conv 1×1 It is a 1×1 convolution;

[0132] Finally, 1×1 convolution is used for channel compression and fusion, as shown in the following formula:

[0133]

[0134] in, Enhance the feature map for the final low-frequency components.

[0135] The final low-frequency feature enhancement result is obtained through a 1×1 convolution operation.

[0136] 2) Convolution kernel design for horizontal high-frequency components:

[0137] The horizontal high-frequency component mainly expresses horizontal edge textures, such as the boundaries of horizontally arranged fruits and the horizontal extension of vines. In view of its directionality, an asymmetric convolution kernel of 1×k is used for directional enhancement to enhance the ability to extract horizontal structures. The convolution method is shown in the following formula:

[0138]

[0139] in, is the horizontal high-frequency component enhanced feature map extracted after the 1×5 asymmetric convolution operation, Conv 1×5 It is a 1×5 asymmetric convolution, which can enhance the feature extraction capability of long horizontal edges.

[0140] 3) Convolution kernel design for vertical high-frequency components:

[0141] The vertical high-frequency component responds to vertical edge information, such as vertically growing fruit stalks and branches. To enhance the vertical structure modeling, a 5×1 asymmetric convolution is used. The convolution method is shown in the following formula:

[0142]

[0143] in, Conv is the vertical high-frequency component enhanced feature map extracted after the 5×1 asymmetric convolution operation. 5×1 It is a 5×1 asymmetric convolution. The 5×1 asymmetric convolution has a larger receptive field in the vertical direction and can enhance the extraction of vertical details.

[0144] 4) Convolution kernel design for diagonal high-frequency components:

[0145] Diagonal high-frequency components often contain fine-grained texture and edge details. By using a lightweight convolutional structure to enhance its expressiveness, 1×1 convolution is subsequently used for channel compression and fusion. The calculation is shown in the following formula:

[0146]

[0147] in, For X HH The enhanced feature map obtained by fusing 3×3 lightweight group convolution with 1×1 convolution, GroupConv 3×3 To process X using 3×3 lightweight group convolution HH , which can reduce redundant feature interference.

[0148] Therefore, this implementation method improves the existing WTConv and includes the following aspects: 1) replacing the original fixed wavelet convolution kernel with a convolution kernel with trainable parameters, so that the model can adaptively learn the frequency filtering features that are most suitable for the current task during training, thereby improving the modeling ability of image details and structures; 3) introducing asymmetric directional convolution and void convolution structures, which are used for feature modeling of high-frequency and low-frequency components respectively, thereby enhancing the model's perception of edges, directional information and long-distance dependencies.

[0149] Through the above structural improvements, MA-WTConv not only retains the multi-level structural information of the image in the spatial dimension, but also realizes fine-grained, learnable multi-scale feature extraction in the frequency dimension, effectively improving the model's recognition accuracy and robustness for tomato fruits in complex agricultural scenarios, overcoming the problem of limited feature expression capabilities of existing WTConv, and has obvious practical value and engineering application prospects.

[0150] Embodiment 3: The tomato picking method described in this embodiment is implemented based on the real-time tomato image detection method described in any one of Embodiments 1 and 2, and includes the following steps:

[0151] Step S4: Input the tomato image to be detected into the SWMD-YOLO model to complete real-time detection of the tomato, record the bounding box coordinates of the tomato output by the SWMD-YOLO model, and obtain the center coordinates of the tomato;

[0152] In step S5, the obtained tomato center coordinates are used as the target position for the robot arm path planning. The RRT algorithm is used to randomly sample and expand the tree structure from the current position of the robot arm to explore the feasible path from the current position to the target position, and complete the tomato picking.

[0153] Existing tomato picking methods rely on manual judgment, often subject to subjectivity and inefficiency. However, this method is inevitably limited by environmental factors and human resources, which can easily lead to inaccurate grasp of picking timing, thus affecting crop quality and yield.

[0154] In a real greenhouse environment, tomato fruits often overlap and are heavily occluded, with varying sparsity and size, posing a challenge to rapid and accurate identification. Several deep learning-based tomato detection models have been developed to address the efficiency challenges of manual harvesting. However, these models suffer from either limited real-time performance, resulting in slow detection due to complex processing, or low recognition accuracy, making it difficult to accurately identify harvestable tomatoes. Furthermore, these models are bulky and difficult to deploy.

[0155] In order to solve the above technical problems, this embodiment is based on the real-time detection method of tomato images described in any one of embodiments one and two, which can circumvent or reduce these problems to a certain extent. First, the YOLO series of algorithms are favored for their high efficiency and accuracy, and therefore become the benchmark model of this embodiment; second, this embodiment improves the model performance by optimizing the model neck network; third, this embodiment adds MSCA to the network structure to enable the model to capture more representative greenhouse tomato features; fourth, this embodiment more effectively fuses multi-scale tomato features by replacing DySample. Finally, compared with some existing tomato recognition models based on deep learning, such as RCNN (target detection model), this embodiment has higher accuracy and a lighter design, making it more suitable for deployment in a picking robot. Thus, a tomato picking method is proposed, comprising the following steps:

[0156] Step S6, as Figure 6 As shown in the figure, the tomato image data to be detected is used as input, and the trained SWMD-YOLO model is applied to perform real-time detection of tomatoes. The bounding box coordinates of each tomato output by the SWMD-YOLO model are recorded, and the detected tomato is numbered as i. The bounding box coordinates of each tomato are (x imin ,y imin ,x imax ,y imax ), where min is the minimum value, max is the maximum value, x is the horizontal pixel coordinate of the image, and y is the vertical pixel coordinate of the image, and the center coordinate of tomato i is obtained.

[0157] In step S7, the center coordinates of tomato i obtained in step S6 are used as the target position for the robot arm's path planning. The RRT (Rapid Random Tree Algorithm) is used to randomly sample and expand the tree structure from the current position of the robot arm to explore feasible paths from the current position to the target position and achieve the picking of the tomato. Specifically,

[0158] First, initialize the robot arm end effector to its current position and set the center coordinates of the tomato i identified by the SWMD-YOLO model to the target position. The center coordinates of the tomato i are calculated as follows:

[0159]

[0160] Among them, C i is the center coordinate of tomato i in the image coordinate system.

[0161] Then randomly sample a point q rand In the working space of the robot, this point can be anywhere to guide the path planning towards the target area. In the tree T, find the distance from the random point q randThe nearest node q near , which is the node in the current tree closest to the random sampling point. The calculation formula is as follows:

[0162] d(q near ,q rand )=||q rand -q near ||;

[0163] Among them, d(q near ,q rand ) is the Euclidean distance between two points, q near is an existing node in the tree, q rand is the randomly sampled target point.

[0164] In q near and q rand A new node q is generated between new , by choosing a suitable step size Δq, along the neer to q rand The expansion formula is as follows:

[0165]

[0166] Among them, Δq is the step size of each expansion, which means the distance of tree expansion, q new For the newly generated node.

[0167] For the newly generated node q new Perform collision detection to ensure that the path of the robot arm does not collide with obstacles in the environment. Repeat the above steps until a part of the tree reaches the target position q goal At this point, the RRT generates a path from its current position to the target position. The robotic arm follows the planned path to the target tomato position and uses force and tactile sensors to control the end effector of the robotic arm to grasp the tomato.

[0168] Therefore, this embodiment uses the SWMD-YOLO model to perform real-time detection of tomato images. This model can quickly and accurately identify greenhouse tomato fruits, addressing some of the limitations currently faced in the harvesting field and providing new technical support for robotic tomato harvesting vision systems. This embodiment focuses on solving the problem of object detection in automated tomato harvesting in greenhouse environments. The core of this problem is to develop and optimize a real-time tomato object detection model, aiming to enhance the accuracy of tomato detection in complex environments. Therefore, compared with existing tomato detection models, the real-time tomato detection model based on the SWMD-YOLO model proposed in this embodiment is not only more efficient but also easier to deploy.

[0169] Introducing RRT for robotic arm path planning allows for real-time path adjustments, ensuring the robotic arm can accurately reach the target tomato despite environmental changes. This eliminates reliance on manual judgment, improves picking efficiency and accuracy, and reduces losses caused by improper picking timing.

[0170] Implementation 4: The real-time tomato image detection system described in this implementation includes the following modules:

[0171] The acquisition module obtains the tomato image dataset, preprocesses the tomato image dataset, and obtains the tomato image training set;

[0172] Construct the module and build the SWMD-YOLO model, specifically:

[0173] The standard convolution in the YOLO11 backbone network is replaced with SAConv, and MA-WTConv is introduced. The downsampling in the YOLO11 neck network is replaced with DySample, and MSCA is introduced.

[0174] The detection module inputs the tomato image to be detected into the SWMD-YOLO model to complete real-time detection of tomatoes.

[0175] Embodiment 5: The tomato picking system described in this embodiment is implemented based on the real-time tomato image detection system described in Embodiment 4 and includes the following modules:

[0176] The recording module inputs the tomato image to be detected into the SWMD-YOLO model to complete real-time detection of the tomato, and records the bounding box coordinates of the tomato output by the SWMD-YOLO model to obtain the center coordinates of the tomato;

[0177] The picking module uses the obtained tomato center coordinates as the target position for the robot arm's path planning. It uses the RRT algorithm to start from the current position of the robot arm, randomly sample and expand the tree structure, explore feasible paths from the current position to the target position, and complete the tomato picking.

[0178] The above describes in detail the real-time tomato image detection method and system, and the tomato picking method and system proposed in the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only intended to help understand the method and core concept of the present invention. At the same time, for those skilled in the art, according to the concept of the present invention, there may be changes in the specific implementation methods and application scope. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A real-time tomato image detection method, characterized in that: The following steps are involved: Step S1, obtaining a tomato image dataset, preprocessing the tomato image dataset, and obtaining a tomato image training set; Step S2: Build the SWMD-YOLO model, specifically: The standard convolution in the YOLO11 backbone network is replaced with SAConv, and MA-WTConv is introduced. The downsampling in the YOLO11 neck network is replaced with DySample, and MSCA is introduced. Step S3: Input the tomato image to be detected into the SWMD-YOLO model to complete the real-time detection of tomatoes; In the step S2, the MA-WTConv is specifically: The tomato image is divided into four wavelet subbands. The low-frequency component, horizontal high-frequency component, vertical high-frequency component and diagonal high-frequency component are input into the four wavelet subbands for convolution operation respectively. The four wavelet subbands after the convolution operation are integrated through inverse wavelet transform and the integration result is output. The convolution kernels of low-frequency components, horizontal high-frequency components, vertical high-frequency components and diagonal high-frequency components are designed separately, and the convolution paths of high-frequency components, vertical high-frequency components and diagonal high-frequency components are independent. The convolution method of the low-frequency component is specifically as follows: ； in, To use the void ratio 3×3 multi-scale dilated convolution operation, For different expansion coefficients, for The low-frequency component of is the input feature map, It is the low-frequency feature response image extracted using different void rates, which represents the input at the current scale. The structural information modeling results, is the low-frequency component; ； in, It is the intermediate feature map after the multi-scale dilated convolution output features are spliced ​​in the channel dimension, which is used to fuse the low-frequency feature responses under different receptive fields. is the feature splicing module, are the results of three groups of multi-scale dilated convolutions. It is a 1×1 convolution; ； in, Enhance the feature map for the final low-frequency component; The convolution method of the horizontal high-frequency component is specifically as follows: ； in, It is the horizontal high-frequency component enhancement feature map extracted after the 1×5 asymmetric convolution operation. is a 1×5 asymmetric convolution, for The horizontal high-frequency component of is the input feature map, is the horizontal high-frequency component; The convolution method of the vertical high-frequency component is specifically as follows: ； in, It is the vertical high-frequency component enhancement feature map extracted after the 5×1 asymmetric convolution operation. is a 5×1 asymmetric convolution, for The vertical high-frequency component of is the input feature map, is the vertical high-frequency component; The convolution method of the diagonal high-frequency component is specifically as follows: ； in, for The enhanced feature map obtained by fusing 3×3 lightweight group convolution with 1×1 convolution, To use 3×3 lightweight group convolution processing , for The diagonal high-frequency components of is the input feature map, is a 1×1 convolution, is the diagonal high frequency component.

2. Tomato image real-time detection system, characterized by: Includes the following modules: The acquisition module obtains the tomato image dataset, preprocesses the tomato image dataset, and obtains the tomato image training set; Construct the module and build the SWMD-YOLO model, specifically: The standard convolution in the YOLO11 backbone network is replaced with SAConv, and MA-WTConv is introduced. The downsampling in the YOLO11 neck network is replaced with DySample, and MSCA is introduced. The detection module inputs the tomato image to be detected into the SWMD-YOLO model to complete real-time detection of tomatoes; In the building blocks, the MA-WTConv is specifically: The tomato image is divided into four wavelet subbands. The low-frequency component, horizontal high-frequency component, vertical high-frequency component and diagonal high-frequency component are input into the four wavelet subbands for convolution operation respectively. The four wavelet subbands after the convolution operation are integrated through inverse wavelet transform and the integration result is output. The convolution kernels of low-frequency components, horizontal high-frequency components, vertical high-frequency components and diagonal high-frequency components are designed separately, and the convolution paths of high-frequency components, vertical high-frequency components and diagonal high-frequency components are independent. The convolution method of the low-frequency component is specifically as follows: ； in, To use the void ratio 3×3 multi-scale dilated convolution operation, For different expansion coefficients, for The low-frequency component of is the input feature map, It is the low-frequency feature response image extracted using different void rates, which represents the input at the current scale. The structural information modeling results, is the low-frequency component; ； in, It is the intermediate feature map after the multi-scale dilated convolution output features are spliced ​​in the channel dimension, which is used to fuse the low-frequency feature responses under different receptive fields. is the feature splicing module, are the results of three groups of multi-scale dilated convolutions. It is a 1×1 convolution; ； in, Enhance the feature map for the final low-frequency component; The convolution method of the horizontal high-frequency component is specifically as follows: ； in, It is the horizontal high-frequency component enhancement feature map extracted after the 1×5 asymmetric convolution operation. is a 1×5 asymmetric convolution, for The horizontal high-frequency component of is the input feature map, is the horizontal high-frequency component; The convolution method of the vertical high-frequency component is specifically as follows: ； in, It is the vertical high-frequency component enhancement feature map extracted after the 5×1 asymmetric convolution operation. is a 5×1 asymmetric convolution, for The vertical high-frequency component of is the input feature map, is the vertical high-frequency component; The convolution method of the diagonal high-frequency component is specifically as follows: ； in, for The enhanced feature map obtained by fusing 3×3 lightweight group convolution with 1×1 convolution, To use 3×3 lightweight group convolution processing , for The diagonal high-frequency components of is the input feature map, is a 1×1 convolution, is the diagonal high frequency component.

Images