Tomato image real-time detection method, tomato image real-time detection system, tomato picking method and tomato picking system

Abstract

The invention discloses a tomato image real-time detection method and system, and a tomato picking method and system, relates to the technical field of intelligent crop picking and deep learning, and solves the problems of adaptability, real-time performance, accuracy and the like of an existing tomato detection technology in a complex environment. Acquiring a tomato image data set, and preprocessing the tomato image data set to obtain a tomato image training set; constructing an SWMD-YOLO model, specifically, replacing a standard convolution in a YOLO11 backbone network with SAConv, introducing MA-WTConv, replacing a down-sampling in a YOLO11 neck network with DySample, and introducing MSCA (Maximum Session Counter Analysis); a tomato image to be detected is input into the SWMD-YOLO model, and real-time detection of tomatoes is completed.

Description

Technical Field

[0001] The present invention relates to the technical fields of intelligent crop picking and deep learning, and particularly relates to a method and system for real-time detection of tomato images, and a method and system for tomato picking. Background Art

[0002] Harvesting robots first use a computer vision system to detect fruits, and then guide the robotic arm to perform harvesting operations according to the detection results. Therefore, fruit detection is a key link in the entire picking process, and its accuracy and speed directly affect the efficiency of the picking robot. However, tomato fruits exhibit different growth postures, overlap with each other, and are severely blocked by leaves, branches, and stems, which poses certain challenges to robot recognition. Quickly and accurately identifying tomato fruits in a complex greenhouse environment is an urgent problem in the development of tomato picking robots.

[0003] With the development of information technology, the widespread application of digital information transmission and industrial Internet of Things has made the management and monitoring of agricultural production more efficient and intelligent. In the field of crop picking, traditional manual picking methods are difficult to meet the requirements of modern agriculture for efficiency and accuracy. For this reason, automated tomato detection systems based on computer vision and deep learning technologies have emerged. Different from traditional methods, these systems can analyze video streams or image data in real time, accurately identify ripe tomatoes, and guide the picking operation, thereby greatly improving the picking efficiency and reducing labor costs.

[0004] With the progress of artificial intelligence and machine learning technologies, tomato detection technologies based on deep learning have emerged. This technology provides a more efficient and accurate means for tomato detection, which can not only analyze image data in real time with little impact on crop growth, but also resist a certain degree of environmental interference. This is crucial for improving the picking efficiency and ensuring the quality of crops. Through an intelligent detection system, farmers can better grasp the harvesting time, thereby maximizing the yield and economic benefits, and promoting the development of agricultural production towards a more refined and intelligent direction. However, existing technologies still have deficiencies in terms of adaptability, real-time performance, and accuracy in complex environments. Summary of the Invention

[0005] The present invention solves the problems of adaptability, real-time performance, and accuracy of existing tomato detection technologies in complex environments.

[0006] The method for real-time detection of tomato images according to the present invention includes the following steps: Step S1, obtaining a tomato image data set, preprocessing the tomato image data set, and obtaining a tomato image training set; Step S2, constructing a SWMD-YOLO model, specifically: Replace the standard convolution in the YOLO11 backbone network with SAConv, introduce MA-WTConv, replace the downsampling in the YOLO11 neck network with DySample, and introduce MSCA; Step S3, input the tomato image to be detected into the SWMD-YOLO model to complete the real-time detection of tomatoes.

[0007] Further, in an embodiment of the present invention, in the step S2, the MA-WTConv is specifically: Divide the tomato image into four wavelet subbands, and input the four wavelet subbands into the low-frequency component, horizontal high-frequency component, vertical high-frequency component, and diagonal high-frequency component respectively for convolution operations. After the four wavelet subbands after convolution operations are integrated through inverse wavelet transform, the integrated result is output; Design the convolution kernels of the low-frequency component, horizontal high-frequency component, vertical high-frequency component, and diagonal high-frequency component respectively, and the convolution paths of the high-frequency component, vertical high-frequency component, and diagonal high-frequency component are all independent.

[0008] Further, in an embodiment of the present invention, the convolution kernel of the low-frequency component is specifically: ; Among them, is the 3×3 multi-scale dilated convolution operation with a dilation rate of , is different dilation coefficients, is of the low-frequency component, is the input feature map, is the low-frequency feature response image extracted using different dilation rates, representing the modeling result of the structural information of the input at the current scale, is the low-frequency component; ; Among them, is the intermediate feature map after concatenating the multi-scale dilated convolution output features in the channel dimension, used to fuse the low-frequency feature responses under different receptive fields, is the feature concatenation module, is the result of three groups of multi-scale dilated convolutions, is the 1×1 convolution; ; Among them, is the final enhanced feature map of the low-frequency component.

[0009] Further, in an embodiment of the present invention, the convolution kernel of the horizontal high-frequency component is specifically: ; Among them, is the enhanced feature map of the horizontal high-frequency component extracted after the 1×5 asymmetric convolution operation, is the 1×5 asymmetric convolution, is 's horizontal high-frequency component, is the input feature map, is the horizontal high-frequency component.

[0010] Furthermore, in an embodiment of the present invention, the convolution kernel of the vertical high-frequency component is specifically: ; Among them, is the enhanced feature map of the vertical high-frequency component extracted after the 5×1 asymmetric convolution operation, is the 5×1 asymmetric convolution, is 's vertical high-frequency component, is the input feature map, is the vertical high-frequency component.

[0011] Furthermore, in an embodiment of the present invention, the convolution kernel of the diagonal high-frequency component is specifically: ; Among them, is the enhanced feature map obtained after the fusion of 3×3 lightweight group convolution and 1×1 convolution, is the processing of using 3×3 lightweight group convolution, is 's diagonal high-frequency component, is the input feature map, is the 1×1 convolution, is the diagonal high-frequency component.

[0012] Furthermore, in an embodiment of the present invention, the four wavelet subbands after the convolution operation are integrated by inverse wavelet transform, specifically: ; Among them, is the low-frequency component feature map obtained by wavelet decomposition of the th layer, is the horizontal high-frequency component feature map obtained by wavelet decomposition of the th layer, is the vertical high-frequency component feature map obtained by wavelet decomposition of the th layer, is the diagonal high-frequency component feature map obtained by the -layer wavelet decomposition, is the hierarchical index of the current wavelet decomposition, is the wavelet transform operation, is the low-frequency component feature map obtained by the -layer wavelet decomposition, is the horizontal high-frequency component feature map obtained by the -layer wavelet decomposition, is the vertical high-frequency component feature map obtained by the -layer wavelet decomposition, is the diagonal high-frequency component feature map obtained by the -layer wavelet decomposition, is the low-frequency component, is the horizontal high-frequency component, is the vertical high-frequency component, is the diagonal high-frequency component, is the input feature map, is the low-frequency component, is the horizontal high-frequency component, is the vertical high-frequency component, is the diagonal high-frequency component.

[0013] The tomato picking method described in the present invention is implemented based on the tomato image real-time detection method described in any one of the above methods, and includes the following steps: Step S4: Input the tomato image to be detected into the SWMD-YOLO model, complete the real-time detection of tomatoes, record the bounding box coordinates of the tomatoes output by the SWMD-YOLO model, and obtain the center coordinates of the tomatoes; Step S5: Use the obtained center coordinates of the tomatoes as the target position for the robotic arm path planning. Starting from the current position of the robotic arm, randomly sample and expand the tree structure, explore the feasible path from the current position to the target position, and complete the tomato picking.

[0014] The tomato image real-time detection system described in the present invention includes the following modules: An acquisition module that acquires a tomato image dataset, preprocesses the tomato image dataset, and obtains a tomato image training set; A construction module that constructs the SWMD-YOLO model, specifically: Replace the standard convolution in the YOLO11 backbone network with SAConv, introduce MA-WTConv, replace the downsampling in the YOLO11 neck network with DySample, and introduce MSCA; The detection module inputs the tomato image to be detected into the SWMD-YOLO model to complete the real-time detection of tomatoes.

[0015] The tomato picking system described in the present invention is implemented based on the above tomato image real-time detection system and includes the following modules: The recording module inputs the tomato image to be detected into the SWMD-YOLO model to complete the real-time detection of tomatoes, records the bounding box coordinates of the tomatoes output by the SWMD-YOLO model, and obtains the center coordinates of the tomatoes. The picking module uses the obtained center coordinates of the tomatoes as the target position for the robotic arm path planning. Starting from the current position of the robotic arm, it randomly samples and expands the tree structure using the RRT algorithm, explores the feasible path from the current position to the target position, and completes the picking of tomatoes.

[0016] The present invention solves the problems of adaptability, real-time performance, and accuracy of existing tomato detection technologies in complex environments. The specific beneficial effects include: 1. For the tomato image real-time detection method described in the present invention, existing tomato detection technologies have problems in terms of adaptability, real-time performance, and accuracy in complex environments. To solve the above technical problems, SAConv and MA-WTConv are introduced into the backbone network of YOLO11, and the original standard convolution module is replaced with the enhanced SAConv. By dynamically adjusting the receptive field size and decomposing local and global features, the detection ability for small targets is enhanced. Especially in complex backgrounds or when the target size is small, it can effectively improve the capture and classification accuracy of the model for small tomatoes. And in the neck network of YOLO11, the downsampling operation is replaced with DySample. DySample can adaptively adjust the upsampling strategy according to the input features, reduce feature loss while more efficiently restoring the detailed information of the feature map. Through multi-scale feature fusion, the model's recognition ability for occluded tomatoes is significantly enhanced. Thus, it copes with the occlusion problem and the feature fusion challenge in complex scenarios; 2. For the tomato image real-time detection method described in the present invention, MSCA is introduced into the YOLO11 neck network. This module dynamically adjusts the attention weights by combining multi-scale context information, effectively strengthens the feature expression of key regions, and at the same time suppresses background noise interference, further improving the target detection accuracy in occlusion and complex scenarios; 3. For the tomato image real-time detection method described in the present invention, by combining DySample and MSCA, the model can more accurately restore the feature edges of small targets, thereby improving the robustness and accuracy of detection; 4. The tomato picking method described in the present invention. Although the current models can solve the efficiency problem of manual tomato picking, these models either lack real-time performance, with complex processing leading to slow detection speed, or have low recognition accuracy and cannot accurately identify pickable tomatoes. Moreover, these models are large in size and difficult to deploy. To solve the above technical problems, the present invention uses the SWMD-YOLO model to perform real-time detection on the targets of tomato images. This model solves the problems of the existing models, which either lack real-time performance, with complex processing leading to slow detection speed, or have low recognition accuracy and cannot accurately identify pickable tomatoes, and are large in size and difficult to deploy. And the RRT is introduced for robotic arm path planning, which can adjust the path in real time to ensure that the robotic arm can still accurately reach the target tomato under changing environmental conditions. This dynamic path adjustment ability greatly improves the robustness and stability of the system in practical applications; The real-time detection method of tomato images described in the present invention is of great significance for the field of intelligent picking. The real-time tomato detection method based on the SWMD-YOLO model effectively solves the problem of low recognition accuracy of small occluded tomatoes, can accurately and quickly identify tomato information, and has advantages in terms of accuracy, robustness, and implementation convenience. This provides an efficient tomato picking solution for farmers and is expected to be popularized in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and / or additional aspects and advantages of the present invention will become apparent and be readily understood from the following description of the embodiments in conjunction with the accompanying drawings, where: Figure 1 is the structural diagram of the SWMD-YOLO model described in Embodiment 1; Figure 2 is the structural diagram of SAConv described in Embodiment 1; Figure 3 is the schematic diagram of DySample described in Embodiment 1; Figure 4 is the structural diagram of MSCA described in Embodiment 1; Figure 5 is a comparison example diagram of the original image and the image after data augmentation in the dataset described in Embodiment 1; Figure 6 is an example diagram of the detection effect in the actual scenario described in Embodiment 3; Figure 7 is the ablation data diagram described in Embodiment 1. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] The following will clearly and completely describe various embodiments of the present invention in conjunction with the accompanying drawings. The embodiments described by referring to the accompanying drawings are exemplary and are intended to explain the present invention, and should not be construed as limiting the present invention.

[0019] Embodiment 1. The real-time tomato image detection method described in this embodiment includes the following steps: Step S1, obtain a tomato image dataset, preprocess the tomato image dataset, and obtain a tomato image training set; Step S2, construct the SWMD-YOLO model, specifically: Replace the standard convolution in the YOLO11 backbone network with SAConv, introduce MA-WTConv, replace the downsampling in the YOLO11 neck network with DySample, and introduce MSCA; Step S3, input the tomato image to be detected into the SWMD-YOLO model to complete the real-time detection of tomatoes.

[0020] Existing tomato detection technologies have problems in terms of adaptability, real-time performance, and accuracy in complex environments.

[0021] To solve the above technical problems, based on the improvement of the YOLO11 model (the 11th version of the YOLO (You Only Look Once) model), this embodiment proposes a real-time tomato image detection method, including the following steps: Step S1, obtain an open-source dataset of tomatoes, and the dataset is marked with category labels and occlusion degree labels, specifically: Adopt the tomato dataset publicly available on Kaggle (website name), and this dataset contains 895 images. The category label is tomato, and the occlusion degree labels include unoccluded, partially occluded, and fully occluded. Step S2, configure the model training environment, specifically: Accelerated computing structure cuda12.4, deep learning architecture pytorch2.4.1, Intel core i7-12700kf CPU, 128G of memory, GPU (graphics card) is NVIDIA GeForce RTX 4070 Ti SUPER, and the video memory is 16G.

[0022] Step S3, preprocess the tomato image dataset to obtain a tomato image training set; As Figure 5 shown, the preprocessing is as follows: Consider using brightness change, saturation change, contrast change, Gaussian noise, and salt-and-pepper noise to perform data augmentation on the tomato image dataset. Finally, obtain 4475 tomato image datasets, and divide the training set, validation set, and test set according to the ratio of 7:2:1. Use 3200 tomato images for model training, 895 tomato images for model validation, and 380 tomato images for model testing.

[0023] In data augmentation, the noise transformation is specifically as follows: Gaussian noise and salt-and-pepper noise with different intensities are randomly added to the image to simulate the image quality fluctuations that may occur during high-temperature periods in the greenhouse. The Gaussian noise transformation is specifically as follows: with a mean of 0 and a standard deviation of 25, noise is randomly added to different regions of the image to simulate the light changes and equipment noise in the greenhouse environment. The salt-and-pepper noise transformation is specifically as follows: white and black pixels are randomly added to random positions in the image with a probability of 0.1 to simulate the failures or other abnormal interferences of the image sensor.

[0024] Step S4, construct the SWMD-YOLO model (image target detection model), specifically as follows: As Figure 1 shown, the SWMD-YOLO model uses the YOLO11 model as the basic model. The YOLO11 model includes a backbone network, a neck network, and a head network. Replace the standard convolution in the YOLO11 backbone network with SAConv (switchable atrous convolution layer), and introduce MA-WTConv (multi-scale adaptive wavelet convolution layer). Replace the downsampling in the YOLO11 neck network with DySample (dynamic upsampling module), and introduce MSCA (multi-scale attention module); For SAConv, as Figure 2 shown, it is divided into three parts. The first part is Pre-Global Context. The feature map passes through a 1×1 pointwise convolution layer and a global average pooling layer for inter-channel information compression and enhancement to extract global context information. The second part is Switchable Atrous Convolution, which has two parallel branches. The atrous convolution branch: is processed using 3×3 atrous convolutions with different dilation rates of 1. One branch has a dilation rate of 1, and the other branch has a dilation rate of 3, which helps to expand the receptive field of the convolution without increasing the computational cost and capture richer multi-scale information. The average pooling branch: performs a 5×5 average pooling layer operation to further obtain local information. Finally, the results of the two branches are integrated through a 1×1 pointwise convolution layer. The third part is Post-Global Context, which is processed through a 1×1 pointwise convolution layer and a global average pooling layer to integrate global information again. The switchable mechanism: through a dynamic weighting parameter , it is possible to switch between the results of the 3×3 atrous convolution and the 5×5 average pooling layer. According to different requirements of the task, the module can adaptively adjust the attention to local details and global features. The specific calculation formula for the 3×3 atrous convolution is: ; where, is the output of a 3×3 dilated convolution operation, is the input feature map, is the weight matrix (convolution kernel) for the 3×3 dilated convolution operation. is the offset of the 3×3 dilated convolution kernel, is the dilation rate of the 3×3 dilated convolution, is a function that calculates a value based on the input feature map to control whether to use the standard convolution kernel or the modified convolution kernel , is the convolution path control function that dynamically switches between the standard convolution and the offset dilated convolution paths according to the value of, is the 3×3 dilated convolution.

[0025] As Figure 3 shown, DySample implements dynamic upsampling using point sampling. First, the input feature map of size C×H×W is transformed into a sampling set S of size 2×sH×sW through a sampling point generator. Among them, the 2 in the first dimension of the sampling set S represents the x and y coordinates. Then, the input features are resampled using the gridsample module (network sampling module) to obtain the upsampled feature map of size C×sH×sW. In the design of DySample, the weights are dynamically adjusted using the following formula: ; where, is the DySample convolution kernel weight applied at position , is the basic convolution kernel weight, is the smooth convolution kernel weight, is the response of the local image.

[0026] As Figure 4 shown, the specific process expression of MSCA is as follows: ; ; where, is the input feature, and are the input and output of MSCA respectively, is the element-wise matrix multiplication operation, , is the th branch, is the depth convolution.

[0027] Scale 0 uses two depthwise bar convolutions for identity connection to approximate the standard depthwise convolution with a large kernel, and the kernel sizes of each branch are set to 7, 11, and 21 respectively.

[0028] To verify whether each module plays a role in the SWMD-YOLO model, as Figure 7 shown, the ablation experiment results will be analyzed as follows: From the ablation experiment results, it can be seen that the SWMD-YOLO model after introducing SAConv has an improvement of 0.69% in mAP50, from 91.36% to 92.04%, and at the same time, the FPS increases from 65.48 to 76.52. On this basis, introducing MA-WTConv further improves mAP50 by 0.55%, from 91.67% to 92.22%, while the FPS decreases from 68.95 to 65.48. Subsequently, introducing MSCA further improves mAP50 by 0.61%, from 92.24% to 92.85%, and at the same time, mAP50-95 increases from 61.46% to 65.74%, and the FPS decreases from 70.21 to 63.91. To further improve the performance while controlling the consumption of parameters and computing resources, MA-WTConv and MSCA are combined and used, and mAP50 reaches 93.47%, which is an improvement of 1.25% compared to using MA-WTConv alone and an improvement of 0.63% compared to using MSCA alone. mAP50-95 increases from 61.46% to 65.74%, and the FPS increases from 70.21 to 72.03. Through this combination, the SWMD-YOLO model achieves good computational efficiency and detection speed while maintaining high performance.

[0029] Step S5, load the constructed SWMD-YOLO model into the configured model training environment, modify the parameter file of the model, train the SWMD-YOLO model using the training set, and evaluate the trained SWMD-YOLO model using the validation set and the test set. Specifically: Step S501, use the backbone network in the YOLO11 model constructed in step S3 to extract the feature vectors of tomato images in the training set, and output at the 5th, 7th, and 11th layers. The features output at the 11th layer are input into the C2PSA (Spatial Pyramid Pooling Layer) for processing, and then the processed features are output to the neck network. The DySample is used to combine the Concat (Feature Concatenation Module) and C3k2 (Three-Layer 2×2 Convolution Module) to perform fusion processing on multi-scale features. The MSCA is introduced into the head network to optimize the attention of the P3, P4, and P5 feature layers, and finally, the Detect (Output Detection Module) is used for object classification and bounding box regression; The number of training rounds is 300, and 16 tomato images are input in each round. During the training process, the training log is observed in real time through Tensorboard (a data visualization tool), and the weight file is saved after the training is completed.

[0030] Step S502, after the training is completed, the trained SWMD-YOLO model is evaluated using the validation set and the test set. The evaluation metrics used include accuracy, recall, PR curve (precision-recall curve), AP (average precision), mAP50-95, and FPS (frames per second). Specifically: ; ; ; ; ; Among them, is the number of images in which the SWMD-YOLO model correctly detects tomato targets, is the number of images in which the SWMD-YOLO model incorrectly detects non-tomato targets, is the number of images in which the SWMD-YOLO model fails to detect tomato targets, is the accuracy, is the recall. The PR curve is constructed using the accuracy and recall, and the area under the curve is denoted as AP. is the AP with the IoU (intersection over union) threshold ranging from 0.5 to 0.95, is the detection time of a single image.

[0031] Step S6, using the tomato image to be detected as the input, the trained SWMD-YOLO model is applied for real-time detection of tomatoes.

[0032] Therefore, this embodiment develops an SWMD-YOLO model for tomato image detection that can identify in the case of occlusion. It can not only improve the accuracy and efficiency of tomato image detection, but also promote the development of intelligent agriculture, optimize the agricultural production process, enhance the market competitiveness of crops, and contribute to the realization of the goal of smart agriculture.

[0033] Embodiment 2: This embodiment further limits the tomato image real-time detection method described in Embodiment 1. In the step S2, the MA-WTConv is specifically: The tomato image is divided into four wavelet subbands, and the four wavelet subbands are respectively input into the low-frequency component, the horizontal high-frequency component, the vertical high-frequency component, and the diagonal high-frequency component for convolution operations. After the four wavelet subbands after the convolution operations are integrated through inverse wavelet transform, the integrated result is output; The convolution kernels of the low-frequency component, the horizontal high-frequency component, the vertical high-frequency component, and the diagonal high-frequency component are designed respectively, and the convolution paths of the high-frequency component, the vertical high-frequency component, and the diagonal high-frequency component are all independent.

[0034] In this embodiment, the convolution kernel of the low-frequency component is specifically:

[0035] Among them, is to use a 3×3 multi-scale dilated convolution operation with a dilation rate of , is different dilation coefficients, is of the low-frequency component, is the input feature map, is the low-frequency feature response image extracted using different dilation rates, indicating the modeling result of the structural information of the input at the current scale, is the low-frequency component; ; Among them, is the intermediate feature map after the multi-scale dilated convolution output features are concatenated in the channel dimension, used to fuse the low-frequency feature responses under different receptive fields, is the feature concatenation module, is the results of three groups of multi-scale dilated convolutions, is a 1×1 convolution; ; Among them, is the final low-frequency component enhanced feature map.

[0036] In this embodiment, the convolution kernel of the high-frequency component is specifically: ; Among them, is the horizontally high-frequency component enhanced feature map extracted after a 1×5 asymmetric convolution operation, is a 1×5 asymmetric convolution, is of the horizontally high-frequency component, is the input feature map, is the horizontally high-frequency component.

[0037] In this embodiment, the convolution kernel of the vertical high-frequency component is specifically: ; where is the enhanced feature map of the vertical high-frequency component extracted after a 5×1 asymmetric convolution operation, is a 5×1 asymmetric convolution, is 's vertical high-frequency component, is the input feature map, is the vertical high-frequency component.

[0038] In this embodiment, the convolution kernel of the diagonal high-frequency component is specifically: ; where is the enhanced feature map obtained after fusing 3×3 lightweight group convolution and 1×1 convolution, is to process using 3×3 lightweight group convolution, is 's diagonal high-frequency component, is the input feature map, is a 1×1 convolution, is the diagonal high-frequency component.

[0039] In this embodiment, the four wavelet subbands after the convolution operation are integrated through inverse wavelet transform, specifically: ; where is the low-frequency component feature map obtained from the -th layer of wavelet decomposition, is the horizontal high-frequency component feature map obtained from the -th layer of wavelet decomposition, is the vertical high-frequency component feature map obtained from the -th layer of wavelet decomposition, is the diagonal high-frequency component feature map obtained from the -th layer of wavelet decomposition, is the hierarchical index of the current wavelet decomposition, is the wavelet transform operation, is the low-frequency component feature map obtained from the -th layer of wavelet decomposition, is the horizontal high-frequency component feature map obtained from the -th layer of wavelet decomposition, is the vertical high-frequency component feature map obtained from the -th layer of wavelet decomposition, is the The diagonal high-frequency component feature map obtained by layer wavelet decomposition is the low-frequency component of is the horizontal high-frequency component of is the vertical high-frequency component of is the diagonal high-frequency component of is the input feature map is the low-frequency component is the horizontal high-frequency component is the vertical high-frequency component is the diagonal high-frequency component

[0040] In this embodiment, the existing WTConv (wavelet convolution) uses a two-dimensional Haar wavelet transform to perform multi-level decomposition on the input image. The two-dimensional Haar wavelet transform uses four filters to decompose the image into four sub-bands: Low-frequency component: Captures the low-frequency information of the image, such as the overall shape or contour. Horizontal high-frequency component: Captures the horizontal edge information in the image. Vertical high-frequency component: Captures the vertical edge information in the image. Diagonal high-frequency component: Captures the diagonal details of the image. The convolution method is as shown in the following formula

[0041] where is the low-frequency filter in the two-dimensional Haar wavelet decomposition is the horizontal high-frequency filter is the vertical high-frequency filter is the diagonal high-frequency filter is the low-frequency component is the horizontal high-frequency component is the vertical high-frequency component is the diagonal high-frequency component

[0042] The convolution operation is as shown in the following formula ; where is the low-frequency component of is the horizontal high-frequency component of is the vertical high-frequency component of is the diagonal high-frequency component of is the input feature map is the convolution operation

[0043] However, the existing WTConv mainly uses two-dimensional Haar wavelet transform to perform multi-level decomposition on images. Although it can extract low-frequency and high-frequency information of images to a certain extent, due to the fixed filter kernel parameters, it is difficult to adapt to the complex changes of different image contents. Especially in tomato images taken in greenhouse environments, due to factors such as strong background interference, blurred target edges, and variable fruit shapes, the frequency features extracted by two-dimensional Haar wavelet transform have problems such as limited expression ability, insufficient direction information, and unstable feature response, resulting in technical bottlenecks such as insufficient feature extraction and decreased detection accuracy in subsequent detection tasks.

[0044] For example, Chinese Patent CN104951783A discloses "An underwater sea cucumber image processing method and system based on wavelet transform algorithm", which denoises underwater sea cucumber images through wavelet transform algorithm to obtain high-clarity images, laying a foundation for the recognition and capture of underwater sea cucumbers. However, this wavelet transform algorithm has the following defects: 1) The wavelet kernel is fixed and non-learnable, and lacks adaptability. In complex environments (such as occlusion, variable light, local blur), the feature extraction ability is limited; 2) The processing dimension is single, only one-layer wavelet decomposition is performed, and the multi-scale features in the image cannot be fully exploited, which is not conducive to the simultaneous perception of detailed textures and global structures.

[0045] To solve the above technical problems, in this embodiment, the convolution kernel of WTConv is improved using conventional convolution. However, after the convolution kernel of WTConv is improved using conventional convolution, when detecting the target of tomato images, there are the following defects: 1) WTConv divides the input features into multiple frequency components. Especially the high-frequency component features are relatively sparse, and the edge response is vulnerable to noise interference. If conventional convolution is directly applied, it may lead to weakening of feature information or over-smoothing, and it is difficult to effectively retain the key texture structure. 2) The low-frequency component, as the structural backbone of the image, contains the overall contour information of most targets. In this channel, fixed kernels or standard convolutions are usually used, and it is difficult to take into account the modeling requirements of targets at different scales.

[0046] To further improve the model's modeling ability and directional response ability for different component features, a differentiated convolution kernel structure is designed for the four components to adapt to their respective frequency structure characteristics. In this embodiment, the convolution kernel of WTConv is improved again, and MA-WTConv (Multi-scale Adaptive Wavelet Convolution) is designed, specifically as follows: 1) Design of convolution kernel for low-frequency component: This component contains the main structural contour information. To enhance the modeling ability for the overall target morphology (such as tomato edges, occlusion relationships), multi-scale dilated convolution is introduced, and the convolution method is shown in the following formula: ; Among them, is the 3×3 multi-scale dilated convolution operation with a dilation rate of . is different dilation coefficients (such as 1, 2, 3). The dilation rate is not set randomly, but designed in combination with the common size scales in tomato images to effectively cover small fruits, large fruits, and blurred structures. is the low-frequency feature response image extracted using different dilation rates, representing the modeling result of the structural information of the input at the current scale. This operation can expand the receptive field without increasing the number of parameters and enhance the perception of target structures at different scales.

[0047] Next, the results output by the multi-scale dilated convolution are feature fused, as shown in the following formula: ; Among them, is the intermediate feature map after concatenating the output features of the multi-scale dilated convolution in the channel dimension, used to fuse the low-frequency feature responses under different receptive fields. is the feature concatenation module. are the results of three groups of multi-scale dilated convolutions. is the 1×1 convolution. Finally, 1×1 convolution is used for channel compression and fusion, as shown in the following formula: ; Among them, is the final enhanced feature map of the low-frequency component.

[0048] The final enhanced result of the low-frequency feature is obtained through the 1×1 convolution operation.

[0049] 2) Design of the convolution kernel for the horizontal high-frequency component: The horizontal high-frequency component mainly expresses the edge textures in the horizontal direction, such as the fruit boundaries arranged horizontally and the horizontal extension of the vines. For its directionality, an asymmetric convolution kernel 1×k is used for direction enhancement to enhance the ability to extract horizontal structures. The convolution method is shown in the following formula: ; Among them, is the enhanced feature map of the horizontal high-frequency component extracted after the 1×5 asymmetric convolution operation. is the 1×5 asymmetric convolution, which can enhance the ability to extract features of horizontal long edges.

[0050] 3) Design of the convolution kernel for the vertical high-frequency component: The vertical high-frequency component responds to vertical edge information, such as vertically growing fruit stalks and vines. To enhance the longitudinal structure modeling, a 5×1 asymmetric convolution is adopted, and the convolution method is shown as follows: ; Among them, is the enhanced feature map of the vertical high-frequency component extracted after the 5×1 asymmetric convolution operation, is the 5×1 asymmetric convolution. The 5×1 asymmetric convolution has a larger receptive field in the vertical direction and can strengthen the vertical direction detail extraction.

[0051] 4) Convolution kernel design of the diagonal high-frequency component: The diagonal high-frequency component mostly contains fine-grained textures and edge details. By adopting a lightweight convolution structure to enhance its expression ability, a 1×1 convolution is subsequently used for channel compression and fusion, and the calculation is shown as follows: ; Among them, is the enhanced feature map obtained after the fusion of the 3×3 lightweight grouped convolution and the 1×1 convolution, is the processing using the 3×3 lightweight grouped convolution for , which can reduce the interference of redundant features.

[0052] In addition, due to the specific filter kernel structure and overlapping window mechanism of wavelet decomposition and reconstruction, how to embed the above convolution operation without destroying its frequency band separation characteristics and keep the network end-to-end trainable is an important implementation difficulty.

[0053] To solve the above technical problems, in this embodiment, independent convolution paths are designed for the four components of wavelet decomposition respectively. Each component convolution operation is carried out inside the component and will not change the feature distribution or channel structure of other components. That is, through structural decoupling, the directional convolution and the dilated convolution are respectively inserted into the feature extraction paths of each sub-band, maintaining the independence of the wavelet transform kernel, and at the same time ensuring that the gradient of the entire module is transmittable and the structure is integrable, thus realizing the effective fusion of the convolution enhancement structure and the wavelet transform.

[0054] After completing the above adaptive convolution operation, the feature maps of the four components are restored to the original spatial dimension through IWT (Inverse Wavelet Transform), and the calculation process is shown as follows: ; Among them, is the transposed convolution operation, using the same filter as the forward wavelet transform to reconstruct the sub-band into the complete image features, It is the reconstructed feature map, representing the feature expression result restored after performing IWT on the four-component enhanced feature maps.

[0055] After completing the convolution, the cascaded wavelet decomposition is obtained by recursively decomposing the four components. Finally, the convolution results of each component are recombined into a complete output using IWT. The following formula shows the calculation process of IWT: ; where, is the low-frequency component feature map obtained from the wavelet decomposition of the th layer, is the horizontal high-frequency component feature map obtained from the wavelet decomposition of the th layer, is the vertical high-frequency component feature map obtained from the wavelet decomposition of the th layer, is the diagonal high-frequency component feature map obtained from the wavelet decomposition of the th layer, is the level index of the current wavelet decomposition, is the wavelet transform operation, is the low-frequency component feature map obtained from the wavelet decomposition of the th layer, is the horizontal high-frequency component feature map obtained from the wavelet decomposition of the th layer, is the vertical high-frequency component feature map obtained from the wavelet decomposition of the th layer, is the diagonal high-frequency component feature map obtained from the wavelet decomposition of the th layer.

[0056] The final output result is shown in the following formula: ; where, is the set of trainable convolution kernels with a size of k×k, and its input channel number is four times that of the original input, corresponding to the four-component channels, is the final output feature map, is to perform wavelet transform on .

[0057] Therefore, by improving the existing WTConv, this embodiment mainly includes the following aspects: 1) replacing the original fixed wavelet convolution kernel with a convolution kernel with trainable parameters, enabling the model to adaptively learn the frequency filtering features most suitable for the current task during training, and enhancing the modeling ability for image details and structures; 2) on the basis of the conventional single-wavelet transform, further performing a second wavelet decomposition on all components (including low-frequency and high-frequency), significantly enhancing the feature expression ability in multiple scales and directions; 3) introducing an asymmetric direction convolution and a dilated convolution structure, which are respectively used for feature modeling of high-frequency and low-frequency components, thereby enhancing the model's perception ability for edge, direction information, and long-range dependencies.

[0058] Through the above structural improvements, MA-WTConv not only retains the multi-level structural information of the image in the spatial dimension but also realizes fine-grained and learnable multi-scale feature extraction in the frequency dimension, effectively improving the recognition accuracy and robustness of the model for tomato fruits in complex agricultural scenarios, overcoming the problem of limited feature expression ability of the existing WTConv, and having obvious practical value and engineering application prospects.

[0059] Embodiment 3. The tomato picking method described in this embodiment is implemented based on the tomato image real-time detection method described in any one of Embodiments 1-2, and includes the following steps: Step S4, input the tomato image to be detected into the SWMD-YOLO model, complete the real-time detection of tomatoes, record the bounding box coordinates of the tomatoes output by the SWMD-YOLO model, and obtain the center coordinates of the tomatoes. Step S5, use the obtained center coordinates of the tomatoes as the target position for the robotic arm path planning. Starting from the current position of the robotic arm, use the RRT algorithm to randomly sample and expand the tree structure, explore the feasible path from the current position to the target position, and complete the picking of tomatoes.

[0060] In the prior art, tomato picking methods rely on manual judgment, often suffering from subjectivity and low efficiency. However, this method is inevitably restricted by environmental factors and human resources, easily leading to inaccurate grasping of the picking time, thus affecting the quality and yield of crops.

[0061] In the actual greenhouse environment, tomato fruits often overlap and are severely occluded, with different sparsity levels and sizes, posing challenges to the rapid and accurate identification of tomato fruits. In fact, there are already some deep learning-based tomato detection models that can solve the efficiency problem of manual picking, but these models are either not real-time, with complex processing resulting in slow detection speed, or have low recognition accuracy and cannot accurately identify pickable tomatoes, and these models are large in size and difficult to deploy.

[0062] To solve the above technical problems, this embodiment is based on the tomato image real-time detection method described in any one of Embodiments 1-2, which can avoid or reduce these problems to a certain extent. First, the YOLO series of algorithms are favored for their high efficiency and accuracy, so they become the benchmark models of this embodiment; second, this embodiment improves the model performance by optimizing the model neck network; third, this embodiment makes the model capture more representative greenhouse tomato features by adding MSCA to the network structure; fourth, this embodiment more effectively fuses multi-scale tomato features by replacing DySample. Finally, compared with some existing tomato recognition models based on deep learning, such as RCNN (object detection model), this embodiment has higher accuracy and a more lightweight design, so it is more suitable for deployment to picking robots. Thus, a tomato picking method is proposed, including the following steps: Step S6, as Figure 6 shown, the tomato image data to be detected is used as input, and the trained SWMD-YOLO model is applied for real-time detection of tomatoes. Record the bounding box coordinates of each tomato output by the SWMD-YOLO model, and number the detected tomatoes as , and the bounding box coordinates of each tomato are , where is the minimum value, is the maximum value, is the pixel coordinate in the horizontal direction of the image, is the pixel coordinate in the vertical direction of the image, and the center coordinates of tomato are obtained.

[0063] Step S7, take the center coordinates of the tomato obtained in Step S6 as the target position for the robotic arm path planning. Use RRT (Rapidly-exploring Random Tree algorithm) to start from the current position of the robotic arm, randomly sample and expand the tree structure, explore the feasible path from the current position to the target position, and achieve the picking of tomatoes, specifically: First, initialize. Set the position of the end effector of the robotic arm as the current position, and set the center coordinates of the tomato identified by the SWMD-YOLO model as the target position. The calculation formula for the center coordinates of tomato is as follows: ; where is the center point coordinates of tomato in the image coordinate system.

[0064] Then randomly sample a point in the workspace of the robotic arm. This point can be any position to guide the path planning to advance towards the target area. In the tree find the node closest to the random point in , which is the node in the current tree closest to the randomly sampled point. The calculation formula is as follows: ; where is the Euclidean distance between two points, is an existing node in the tree, is the target point of random sampling.

[0065] Generate a new node between and , by selecting an appropriate step size and extending the tree along the direction from to . The extension formula is as follows: ; where is the step size for each extension, representing the distance of tree extension, is the newly generated node.

[0066] Perform collision detection on the newly generated node to ensure that the path of the robotic arm does not collide with obstacles in the environment. Repeat the above steps until a part of the tree reaches the target position . At this time, RRT generates a path from the current position to the target position. The robotic arm reaches the tomato target position according to the planned path, and uses the force sensor and tactile sensor to control the end effector of the robotic arm to grasp the tomato.

[0067] Therefore, this embodiment performs real-time detection on tomato images based on the SWMD-YOLO model. This model can quickly and accurately identify greenhouse tomato fruits, solves some limitations faced in the current picking field, and provides new technical support for the vision system of tomato picking robots. This embodiment focuses on solving the target detection problem of automated tomato harvesting in greenhouse environments. The core of this problem is to develop and optimize a real-time tomato target detection model, aiming to enhance the accuracy of tomato detection in complex environments. Therefore, compared with existing tomato detection models, the real-time tomato detection model based on the SWMD-YOLO model proposed in this embodiment is not only more efficient but also more convenient to deploy.

[0068] Introducing RRT for robotic arm path planning can adjust the path in real time to ensure that the robotic arm can still accurately reach the target tomato under changing environmental conditions. Avoid relying on manual judgment to improve picking efficiency and accuracy and reduce losses caused by improper picking timing.

[0069] Embodiment 4. The tomato image real-time detection system described in this embodiment includes the following modules: An acquisition module that acquires a tomato image data set, preprocesses the tomato image data set, and obtains a tomato image training set; A construction module that constructs a SWMD-YOLO model, specifically: Replace the standard convolution in the YOLO11 backbone network with SAConv, introduce MA-WTConv, replace the downsampling in the YOLO11 neck network with DySample, and introduce MSCA; A detection module that inputs the tomato image to be detected into the SWMD-YOLO model to complete the real-time detection of tomatoes.

[0070] Embodiment 5. The tomato picking system described in this embodiment is implemented based on the tomato image real-time detection system described in Embodiment 4 and includes the following modules: A recording module that inputs the tomato image to be detected into the SWMD-YOLO model to complete the real-time detection of tomatoes, records the bounding box coordinates of the tomatoes output by the SWMD-YOLO model, and obtains the center coordinates of the tomatoes; A picking module that uses the obtained center coordinates of the tomatoes as the target position for the robotic arm path planning, starts from the current position of the robotic arm, randomly samples and expands the tree structure using the RRT algorithm, explores the feasible path from the current position to the target position, and completes the picking of tomatoes.

[0071] The above has introduced in detail the tomato image real-time detection method and its system, the tomato picking method and its system proposed by the present invention. Specific examples are used in this article to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A real-time detection method for tomato images, characterized in that, It includes the following steps: Step S1: Obtain a tomato image dataset, preprocess the tomato image dataset, and obtain a tomato image training set; Step S2: Construct an SWMD-YOLO model, specifically: Replace the standard convolution in the YOLO11 backbone network with SAConv, introduce MA-WTConv, replace the downsampling in the YOLO11 neck network with DySample, and introduce MSCA; Step S3: Input the tomato image to be detected into the SWMD-YOLO model to complete real-time detection of tomatoes.

2. The real-time tomato image detection method according to claim 1, characterized in that In the said Step S2, the said MA-WTConv is specifically: Divide the tomato image into four wavelet subbands, and input the four wavelet subbands into the low-frequency component, horizontal high-frequency component, vertical high-frequency component, and diagonal high-frequency component respectively for convolution operations. After the convolution operations on the four wavelet subbands are integrated through inverse wavelet transform, the integrated result is output; Design the convolution kernels of the low-frequency component, horizontal high-frequency component, vertical high-frequency component, and diagonal high-frequency component respectively, and the convolution paths of the high-frequency component, vertical high-frequency component, and diagonal high-frequency component are all independent.

3. The real-time tomato image detection method according to claim 2, wherein The convolution kernel of the said low-frequency component is specifically: ； Among them, is to use a 3×3 multi-scale dilated convolution operation with a dilation rate of , is different dilation coefficients, is low-frequency component, is the input feature map, is the low-frequency feature response image extracted using different dilation rates, representing the modeling result of the structural information of the input at the current scale, is the low-frequency component; ； Among them, is the intermediate feature map after concatenating the output features of multi-scale dilated convolutions in the channel dimension, which is used to fuse the low-frequency feature responses under different receptive fields, is the feature concatenation module, are the results of three groups of multi-scale dilated convolutions, is a 1×1 convolution; ； Among them, is the final enhanced feature map of the low-frequency component.

4. The real-time detection method of tomato images according to claim 2, wherein The convolution kernel of the said horizontal high-frequency component is specifically: ； Among them, is the enhanced feature map of the horizontal high-frequency component extracted after the 1×5 asymmetric convolution operation, is the 1×5 asymmetric convolution, is the horizontal high-frequency component of is the input feature map, is the horizontal high-frequency component.

5. The real-time tomato image detection method according to claim 2, wherein, The convolution kernel of the said vertical high-frequency component is specifically: ； Among them, is the enhanced feature map of the vertical high-frequency component extracted after the 5×1 asymmetric convolution operation, is the 5×1 asymmetric convolution, is the vertical high-frequency component of is the input feature map, is the vertical high-frequency component.

6. The real-time detection method of tomato images according to claim 2, characterized in that The convolution kernel of the said diagonal high-frequency component is specifically: ； Among them, is the enhanced feature map obtained after the fusion of 3×3 lightweight grouped convolution and 1×1 convolution, is the result of processing with 3×3 lightweight grouped convolution , is the diagonal high-frequency component of, is the input feature map, is 1×1 convolution, is the diagonal high-frequency component.

7. The real-time tomato image detection method according to claim 2, wherein The integration of the four wavelet subbands after the convolution operations through inverse wavelet transform is specifically: ； Among them, is the low-frequency component feature map obtained by the -layer wavelet decomposition, is the horizontal high-frequency component feature map obtained by the -layer wavelet decomposition, is the vertical high-frequency component feature map obtained by the -layer wavelet decomposition, is the diagonal high-frequency component feature map obtained by the -layer wavelet decomposition, is the level index of the current wavelet decomposition, is the wavelet transform operation, is the low-frequency component feature map obtained by the -layer wavelet decomposition, is the horizontal high-frequency component feature map obtained by the -layer wavelet decomposition, is the vertical high-frequency component feature map obtained by the -layer wavelet decomposition, is the diagonal high-frequency component feature map obtained by the -layer wavelet decomposition, is the low-frequency component, is the horizontal high-frequency component, is the vertical high-frequency component, is the diagonal high-frequency component, is the input feature map, is the low-frequency component, is the horizontal high-frequency component, is the vertical high-frequency component, is the diagonal high-frequency component.

8. A tomato picking method, which is implemented based on the tomato image real-time detection method according to any one of claims 1-7, characterized in that, It includes the following steps: Step S4: Input the tomato image to be detected into the SWMD-YOLO model to complete real-time detection of tomatoes, record the bounding box coordinates of the tomatoes output by the SWMD-YOLO model, and obtain the center coordinates of the tomatoes; Step S5: Use the obtained center coordinates of the tomatoes as the target position for the robotic arm path planning. Starting from the current position of the robotic arm, use the RRT algorithm to randomly sample and expand the tree structure, explore the feasible path from the current position to the target position, and complete the picking of tomatoes.

9. A real-time tomato image detection system, characterized in that, It includes the following modules: An acquisition module that acquires a tomato image dataset, preprocesses the tomato image dataset, and obtains a tomato image training set; A construction module that constructs an SWMD-YOLO model, specifically: Replace the standard convolution in the YOLO11 backbone network with SAConv, introduce MA-WTConv, replace the downsampling in the YOLO11 neck network with DySample, and introduce MSCA; A detection module that inputs the tomato image to be detected into the SWMD-YOLO model to complete real-time detection of tomatoes.

10. A tomato picking system, which is implemented based on the tomato image real-time detection system described in claim 9, and is characterized in that, It includes the following modules: A recording module that inputs the tomato image to be detected into the SWMD-YOLO model to complete real-time detection of tomatoes, records the bounding box coordinates of the tomatoes output by the SWMD-YOLO model, and obtains the center coordinates of the tomatoes; A picking module that uses the obtained center coordinates of the tomatoes as the target position for the robotic arm path planning. Starting from the current position of the robotic arm, use the RRT algorithm to randomly sample and expand the tree structure, explore the feasible path from the current position to the target position, and complete the picking of tomatoes.

Images