Deep learning-based tomato maturity detection method

Abstract

The invention discloses a tomato maturity detection method based on deep learning, and is applied to the field of tomato maturity detection in a complex scene. Comprising the following steps: acquiring tomato picture data sets at different maturity stages in a complex scene; manually marking the position and the type of the tomato in the image, and dividing the data set; the method comprises the following steps: introducing a Swin Transform into a backbone network; dSConv is introduced into the neck part; constructing a loss function of the improved YOLOv8 model; an improved YOLOv8 model is trained; according to the method, improvement is carried out based on YOLOv8, richer global context information can be provided, the model is more excellent in the aspect of processing complex scenes and details, and therefore the accuracy of the model for detecting the tomato maturity in the complex scenes is improved.

Description

Technical Field

[0001] The present invention is applied to the field of tomato maturity detection in complex scenarios, and specifically is a tomato maturity detection method based on deep learning. Background Art

[0002] With the development of science and technology, the detection of tomato maturity has gradually changed from manual subjective judgment to computer detection. At present, computer detection of tomato maturity is mainly based on the target detection model of deep learning. Compared with the traditional image processing method for detecting tomato maturity, the deep learning method is more suitable for real-time detection. At present, the target detection algorithms based on deep learning for tomato maturity detection are mainly divided into two-stage target detection and one-stage target detection; the two-stage detection algorithms mainly include FasterR-CNN, maskR-CNN, etc.; in 2021, Zu et al. used MaskRCNN and RestNet50-FPN backbone network to detect and segment green tomatoes in greenhouses; the algorithm uses ROIAlign bilinear interpolation to calculate the target area generated by the training feature map, and realizes the detection and segmentation of ripe green tomatoes through the parallel operation of ROI target category, bounding box regression and mask; since they only experimented with green tomatoes, the different maturity stages of tomatoes were not considered; and the data used in the experiment was obtained in a greenhouse, and the influence of various factors in actual complex scenes on the experiment was not considered; therefore, this method is not suitable for the detection of different maturity stages of tomatoes in complex scenes; in addition, due to the high computational resource consumption of the two-stage algorithm, it is not suitable for real-time detection; the one-stage detection algorithm mainly includes the YOLO series of algorithms; Currently, the more mature YOLO algorithms are YOLOv5, YOLOv7, and YOLOv8; and researchers have applied them to the inspection of tomato maturity; in 2023, Li et al. proposed a tomato maturity grading and counting model based on YOLOv8; this model improved the backbone network of YOLOv8 and improved the network's ability to extract diversified features; this method showed good performance for tomato counting, but the effect of maturity detection was not well improved. Therefore, this application provides a tomato maturity detection method based on deep learning. Summary of the invention

[0003] In order to address the deficiencies in the above-mentioned prior art, the object of the present invention is to provide a tomato maturity detection method based on deep learning, which is improved based on YOLOv8 and can provide richer global context information, so that the model performs better in processing complex scenes and details, thereby improving the accuracy of the model in detecting tomato maturity in complex scenes.

[0004] The technical solution adopted by the present invention to solve the technical problem is: A tomato maturity detection method based on deep learning is provided, comprising the following steps: Step 1: Obtain a dataset of tomato images at different maturity stages in complex scenes; Step 2: Manually mark the location and type of tomatoes in the image and divide the dataset; Step 3: Improve the backbone network; Step 4: Improve the neck network; Step 5: Construct the loss function of the improved YOLOv8 model; Step 6: Train the improved YOLOv8 model; Step 7: Use the trained model for testing.

[0005] In the step 1, the tomato image dataset used includes large tomato images and small tomato images; and according to the maturity of the tomatoes, the tomatoes are divided into three types: unripe, half-ripe and fully ripe.

[0006] In the step 2, the tomatoes in the collected pictures are labeled using the labeling software Labelimg to obtain the labeling information of the tomatoes and save it in a file; the file contains the category information and location information of each target; the data set we use has a total of six categories: unripe large tomatoes (b_green), half-ripe large tomatoes (b_halfripened), fully ripe large tomatoes (b_fullyripened), unripe small tomatoes (l_green), half-ripe small tomatoes (l_halfripened), and fully ripe small tomatoes (l_fullyripened); after the labeling is completed, the data set is randomly divided into a training set, a validation set, and a test set in a ratio of 7:2:1.

[0007] In the step 3, Swin Transformer is introduced into the third C2f layer of the original backbone to obtain an improved backbone. The tomato image data enters the improved backbone network. After passing through the first two layers of C2F modules, a preliminary low-level feature representation is obtained. The low-level feature representation is downsampled and passed to the Swin Transformer layer as input. In the Swin Transformer layer, the feature map is divided into multiple windows. The size of each window can be adjusted by hyperparameters. The window size of 4x4 or 8x8 can be selected to meet different application requirements. This method not only reduces the computational complexity, but also enables the self-attention mechanism to efficiently calculate the similarity between features in each window. Through this mechanism, important tomato features are emphasized, while irrelevant information is suppressed. In addition, in order to enhance the interaction between windows, the windows are shifted to ensure that the features between different windows can be effectively fused. This process can enhance the model's ability to understand global information. After the self-attention calculation of the features in each window, it is then processed by linear transformation and activation function, and finally a new feature representation is output. After feature extraction is completed, the feature map of the improved backbone network module will enter the feature fusion layer for further processing.

[0008] In the step 4, the feature map sizes of five different scales are obtained through feature extraction of the backbone network, which are respectively recorded as P1, P2, P3, P4, and P5; then, P3, P4, and P5 are transferred to the feature pyramid network of the neck for fusion; compared with the original neck network, the improved network replaces the ordinary convolution in C2f with the distributed shift convolution (DSConv) to obtain the module C2f_DSConv; The improved neck network uses DSConv. The overall goal of DSConv is to simulate the behavior of convolutional layers by using quantization and distribution shift. In the original C2F module, the connection of convolutional layers is linear, and each layer depends only on the output of the previous layer. By introducing distribution shift convolution, we can achieve feature diversity and interactivity between each convolutional layer. This method can not only improve the effectiveness of feature extraction, but also reduce the risk of information loss and ensure the stability of the model when facing complex images.

[0009] After entering C2f_DSConv, the feature map first undergoes a distribution shift convolution operation. We translate the feature map according to the preset step size, usually including shifts in multiple directions (such as up, down, left, and right). This translation allows each pixel to focus on the features of the surrounding adjacent areas, thereby enhancing the ability to capture contextual information in the tomato image. After the distribution shift convolution operation, the feature map is divided into two parts, one part is directly passed to the concatenation layer (Concat), and the other part is processed by multiple Bottleneck_DSConv2d modules and concatenated with the previously directly passed part. Feature fusion is achieved by concatenating features from different branches in the channel dimension. The concatenated features contain information from different branches and have a more comprehensive feature representation. The feature map processed by the above operations will be passed to the subsequent network layer for further processing.

[0010] In the step 5, the loss function of the improved YOLOv8 consists of several parts, including classification loss (VFLLoss) and regression loss (Focal-EIOU Loss+DFL Loss); the classification loss is consistent with the original YOLOv8 model; and in the regression loss, Focal-EIoU is used to replace the CIoU loss function used by the original YOLOv8.

[0011] In step six, during model training, the size of the input image is adjusted, SGD is used as the optimizer, the entire training process has 400 epochs, the batch size is set to 16, the learning rate is set to 0.01, and the momentum is set to 0.937.

[0012] In the step 7, the prepared tomato images with different maturity stages to be detected are input into the trained model, and the reasoning process is performed to obtain the result of tomato maturity recognition.

[0013] Compared with the prior art, the present invention has the following beneficial effects: 1. The tomato maturity detection method based on deep learning in the example of the present invention introduces a distributed shift convolution at the neck. By dynamically adjusting the weight of the convolution kernel, the model's adaptability to the target shape and deformation is improved, thereby improving the generalization ability of the model. This also improves the performance of the model in complex scenes. The distributed shift convolution decomposes the convolution operation into depth convolution and point-by-point convolution, reducing the computational complexity and the number of parameters, enabling the model to make more accurate judgments in a shorter time.

[0014] 2. The tomato maturity detection method based on deep learning in the example of the present invention improves the backbone network and strengthens the attention to the global and local information in the tomato image through a multi-level self-attention mechanism. It can flexibly capture features of different scales and provide richer feature representations, thereby improving the model's recognition ability to extract tomato features in complex scenes.

[0015] 3. The tomato maturity detection method based on deep learning in the present invention uses Focal-EIOU instead of the original CIOU, combines the class imbalance processing capability of Focal Loss and the bounding box regression improvement of EIOU to improve the detection performance under class imbalance and enhance the attention to difficult samples. Improve the performance and accuracy of the model in object detection tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings: Figure 1 This is a flow chart of the tomato maturity detection method based on deep learning; Figure 2 This is the improved YOLOv8 structure diagram; Figure 3 This is the Swin Transformer structure diagram; Figure 4 For C2f_DSConv module; Figure 5 This is the DSConv structure diagram. DETAILED DESCRIPTION

[0017] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.

[0018] The components of the embodiments of the present invention generally described and shown in the drawings herein may be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention.

[0019] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.

[0020] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0021] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0022] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings.

[0023] In the prior art, most deep learning models used to detect tomato maturity use standard convolutional neural networks. Standard convolutional neural networks usually require a large amount of data to learn effective feature representations, which means that insufficient training data may lead to model overfitting or performance degradation. In addition, the training process of standard convolutional neural networks requires a large amount of computing resources. This places very high demands on hardware resources, which will lead to increased computing resources and time costs. For real-time detection and resource-constrained devices, this high computing demand may become a bottleneck. The C2f module in the original YOLOv8 performs well in local feature extraction, but lacks in capturing global background information. In addition, C2f may be limited to a fixed hierarchy when extracting features, and is less effective when dealing with complex backgrounds or lighting changes. In the original YOLOv8, the CIoU loss function optimizes the regression of the target box by comprehensively considering factors such as IoU, center point distance, aspect ratio, etc., but this may lead to limited performance when dealing with complex scenes or small targets; CIOU mainly focuses on the accuracy of the target box and pays less attention to the problem of class imbalance; in the target detection task, the imbalance of positive and negative samples will affect the training effect.

[0024] The present invention is improved based on YOLOv8, which can provide richer global context information, so that the model performs better in processing complex scenes and details, thereby improving the accuracy of the model in detecting the maturity of tomatoes in complex scenes. The specific steps are as follows: (The present application Figure 5 ⊙ denotes the Hadamard product. ): Step 1: Obtain a dataset of tomato images at different maturity stages in complex scenes.

[0025] In this step, we use the publicly available tomato dataset, which contains 804 tomato fruit images, including 442 large tomato images and 362 small tomato images. This image dataset shows tomatoes at different stages of maturity, which can be divided into three types according to maturity: unripe, half-ripe, and fully ripe. It includes 1854 unripe large tomatoes (b_green), 636 half-ripe large tomatoes (b_halfripened), 420 fully ripe large tomatoes (b_fullyripened), 4596 unripe small tomatoes (l_green), 1020 half-ripe small tomatoes (l_halfripened), and 1251 fully ripe small tomatoes (l_fullyripened).

[0026] Step 2: Manually mark the location and type of tomatoes in the image and divide the dataset.

[0027] Use the labeling software Labelimg to label the tomatoes in the collected pictures, obtain the labeling information of the tomatoes and save it in a file. The file contains the category information and location information of each target. The dataset we use has six categories: b_green, b_halfripened, b_fullyripened, l_green, l_halfripened, l_fullyripened; after labeling, the dataset is randomly divided into training set, validation set, and test set in a ratio of 7:2:1.

[0028] Step 3: Improve the backbone network. Improving the backbone network specifically refers to introducing Swin Transformer into the backbone network.

[0029] The improved YOLOv8 model consists of three parts: backbone network (Backbone), neck network (Neck) and detection head (Head). The improved structure is as follows Figure 2As shown in the figure. The backbone network includes CBS module, C2f module, SPPF module and SwinTransformer module. In the new backbone network, the third C2F module of YOLOv8 is replaced by SwinTransformer module to improve the feature extraction capability and the overall performance of the model. This improved backbone network structure aims to more effectively capture the contextual information in the image by introducing the self-attention mechanism, thereby enhancing the accuracy and robustness of object detection. The new network structure still maintains the efficient design of YOLOv8, but realizes a more flexible processing method in the feature extraction process.

[0030] The input layer of the new backbone network remains unchanged to adapt to different image sizes, thus ensuring that the model can flexibly handle inputs of various resolutions. Before inputting the image, the image is resized to 640×640 pixels and normalized so that the pixel values ​​are between [0,1] to meet the input requirements of the model. The 640×640×3 image data then enters the improved backbone network. The first two layers of C2F modules are used to extract low-level features to capture the edges and textures in the image. These low-level features provide basic information for subsequent high-level features. Low-level features can be efficiently transmitted to higher-level semantic information layers, ensuring the continuity and accuracy of information.

[0031] Improved backbone network such as Figure 2 As shown in Figure 2. After the first two layers of C2F modules, preliminary low-level feature representations are obtained. After a certain amount of downsampling, these representations are passed to the Swin Transformer module as input. Swin Transformer consists of four stages, and its structure is as follows: Figure 3 As shown in Figure 1. In this module, the feature map is divided into multiple windows. The size of each window can be adjusted through hyperparameters. The window size of 4x4 or 8x8 can be selected to meet different application requirements. This method not only reduces the computational complexity, but also enables the self-attention mechanism to efficiently calculate the similarity between features in each window. Through this mechanism, important tomato features are emphasized, while irrelevant information is suppressed. In addition, in order to enhance the interaction between windows, the model shifts the windows to ensure that the features between different windows can be effectively fused. This process can improve the model's ability to understand global information. After the self-attention calculation of the features in each window, it is then processed through linear transformation and activation function, and finally a new feature representation is output. After the feature extraction is completed, the feature map that has passed the Swin Transformer module will enter the feature fusion layer for further processing.

[0032] Step 4: Improve the neck network. Improving the neck network specifically refers to introducing DSConv into the neck network.

[0033] Through the feature extraction of the backbone network, five feature map sizes of different scales are obtained, which are recorded as P1, P2, P3, P4, and P5. Subsequently, P3, P4, and P5 are passed to the feature pyramid network of the neck for fusion. Compared with the original neck network, the improved network replaces the ordinary convolution in C2f with the distributed shift convolution (DSConv) to obtain the module C2f_DSConv, as shown in Figure 4 The overall goal of DSConv is to simulate the behavior of convolutional layers by using quantization and distribution shift. Its structure is shown in Figure 5 As shown in the figure. In the original C2F module, the connection mode of convolutional layers is linear, and each layer only depends on the output of the previous layer. By introducing distributed shift convolution, we can achieve feature diversity and interactivity between each convolutional layer. This method can not only improve the effectiveness of feature extraction, but also reduce the risk of information loss and ensure the stability of the model when facing complex images.

[0034] After entering C2f_DSConv, the feature map first undergoes a distribution shift convolution operation. We translate the feature map according to the preset step size, usually including shifts in multiple directions (such as up, down, left, and right). This translation allows each pixel to focus on the features of the surrounding adjacent areas, thereby enhancing the ability to capture contextual information in the tomato image. After the distribution shift convolution operation, the feature map is divided into two parts, one part is directly passed to the concatenation layer (Concat), and the other part is processed by multiple Bottleneck_DSConv2d modules and concatenated with the previously directly passed part. Feature fusion is achieved by concatenating features from different branches in the channel dimension. The concatenated features contain information from different branches and have a more comprehensive feature representation. The feature map processed by the above operations will be passed to the subsequent network layer for further processing.

[0035] Step 5: Construct the loss function of the improved YOLOv8 model.

[0036] The loss function of the improved YOLOv8 consists of several parts, including classification loss (VFLLoss) and regression loss (Focal-EIOULoss+DFLLoss). The classification loss is consistent with the original YOLOv8 model. In the regression loss, Focal-EIoU is used to replace the CIoU loss function used by the original YOLOv8. Compared with the CIoU loss function, the advantage of the Focal-EIoU loss function is that it comprehensively considers multiple factors such as category imbalance, bounding box positioning and shape. This can more comprehensively optimize the target detection task and make the model perform better in complex environments. The Focal-EIoU loss function combines the FocalL1 loss and the EIoU loss function. The EIoU loss function includes IOU loss ( ), distance loss( ) and direction loss ( ), its formula is as follows: ; in, represents any two boxes. and Represent the center points of the anchor box and the ground-truth box respectively. and Represent the width of the anchor box and the ground-truth box respectively. and Represent the height of the anchor box and the height of the ground-truth box respectively. and are the width and height of the smallest box that covers the anchor box and the ground-truth box, respectively. Represents the Euclidean distance. The formula for IOU is as follows: ; in Represents any two boxes. The formula for FocalL1 loss is as follows: ; in , is a parameter, Indicates the horizontal coordinate of the point. Represents a constant. The Focal-EIoU loss function is obtained by integrating FocalL1 and EIoU losses, and the formula is as follows: ; in, It is a parameter that controls the degree of outlier suppression. In this method, we use the Focal-EIOU loss function instead of the CIOU loss function in the original model to improve the convergence speed and positioning accuracy of the model and solve the problem of imbalance between difficult and easy samples.

[0037] Step 6: Train the improved YOLOv8 model.

[0038] In model training, the size of the input image is set to 640×640, SGD is used as the optimizer, the entire training process is 400 epochs, the batch size is set to 16, the learning rate is set to 0.01, and the momentum is set to 0.937.

[0039] Step 7: Use the trained model for testing.

[0040] The prepared tomato images with different maturity stages to be detected are input into the trained model, and the inference process is performed to obtain the result of tomato maturity recognition.

[0041] The comparative experimental results of the method proposed in the present invention and other methods are shown in Table 1: Model Mean Average Precision (mAP) YOLOv5n 0.826 YOLOv7 0.845 YOLOv8n 0.824 YOLOv8s 0.846 YOLOv8m 0.848 The method proposed by the present invention 0.869 Table 1 From the comparative experimental results, the detection accuracy of the improved model is higher than that of other models. The mean average precision (mAP) of the improved model reaches 0.869, which is 4.3%, 2.4%, 4.5%, 2.3% and 2.1% higher than YOLOv5n, YOLOv7-tiny, YOLOv8n, YOLOv8s and YOLOv8m respectively. These results show that the improved YOLOv8s model has significant advantages in the tomato ripening detection task, and the adopted improvement strategy plays an important role in the model performance.

[0042] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the present application.

[0043] Except for the technical features described in the specification, the remaining technical features are known technologies to those skilled in the art. In order to highlight the innovative features of the present invention, the remaining technical features will not be described here in detail.

Claims

1. A tomato maturity detection method based on deep learning, characterized in that: The following steps are involved: Step 1: Obtain a dataset of tomato images at different maturity stages in complex scenes; Step 2: Manually mark the location and type of tomatoes in the image and divide the dataset; Step 3: Improve the backbone network; Step 4: Improve the neck network; Step 5: Construct the loss function of the improved YOLOv8 model; Step 6: Train the improved YOLOv8 model; Step 7: Use the trained model for testing.

2. The tomato maturity detection method based on deep learning according to claim 1, characterized in that: In the step 1, the tomato image dataset used includes large tomato images and small tomato images; and according to the maturity of the tomatoes, the tomatoes are divided into three types: unripe, half-ripe and fully ripe.

3. The tomato maturity detection method based on deep learning according to claim 2, characterized in that: In the step 2, the tomatoes in the collected pictures are labeled using the labeling software Labelimg to obtain the labeled information of the tomatoes and save it in a file; The file contains the category information and location information of each target; the dataset we use has six categories: unripe large tomatoes, half-ripe large tomatoes, fully ripe large tomatoes, unripe small tomatoes, half-ripe small tomatoes, and fully ripe small tomatoes; after labeling, the dataset is randomly divided into training set, validation set, and test set in a ratio of 7:2:

1.

4. The tomato maturity detection method based on deep learning according to claim 3 is characterized in that: In the step three, Swin Transformer is introduced into the third C2f layer of the original backbone to obtain an improved backbone. The tomato picture data enters the improved backbone network, and after passing through the first two layers of C2F modules, a preliminary low-level feature representation is obtained. The low-level feature representation is downsampled and passed to the Swin Transformer layer as input. In the Swin Transformer layer, the feature map is divided into multiple windows, the size of each window can be adjusted by hyperparameters, and the windows are shifted so that the features between different windows can be effectively fused. After the self-attention calculation of the features in each window, they are processed by linear transformation and activation function, and finally a new feature representation is output. After the feature extraction is completed, the feature map of the improved backbone network module will enter the feature fusion layer for further processing.

5. The tomato maturity detection method based on deep learning according to claim 4, characterized in that: In the step 4, the feature map sizes of five different scales are obtained through feature extraction of the backbone network, which are respectively recorded as P1, P2, P3, P4, and P5; then, P3, P4, and P5 are transferred to the feature pyramid network of the neck for fusion; compared with the original neck network, the improved network replaces the ordinary convolution in C2f with the distributed shift convolution to obtain the module C2f_DSConv; The improved neck network uses DSConv. The overall goal of DSConv is to simulate the behavior of the convolution layer by using quantization and distribution offset. After the feature map enters C2f_DSConv, it first undergoes a distribution shift convolution operation to translate the feature map according to a preset step size, including shifts in multiple directions. After the distribution shift convolution operation, the feature map is divided into two parts, one part is directly passed to the concatenation layer, and the other part is processed by multiple Bottleneck_DSConv2d modules and then concatenated with the part that was directly passed before. By concatenating the features of different branches in the channel dimension, the features are realized. The feature map after the above operation will be passed to the subsequent network layer for further processing.

6. The tomato maturity detection method based on deep learning according to claim 5, characterized in that: In the step 5, the loss function of the improved YOLOv8 consists of several parts, including classification loss and regression loss; the classification loss is consistent with the original YOLOv8 model; and in the regression loss, Focal-EIoU is used to replace the CIoU loss function used by the original YOLOv8.

7. The tomato maturity detection method based on deep learning according to claim 6, characterized in that: In step six, during model training, the size of the input image is adjusted, SGD is used as the optimizer, the entire training process has 400 epochs, the batch size is set to 16, the learning rate is set to 0.01, and the momentum is set to 0.

937.

8. The tomato maturity detection method based on deep learning according to claim 7, characterized in that: In the step 7, the prepared tomato images with different maturity stages to be detected are input into the trained model, and the reasoning process is performed to obtain the result of tomato maturity recognition.

Images