Method for detecting maturity of greenhouse tomatoes based on improved YOLOv8n

Abstract

The invention discloses a greenhouse tomato maturity detection method based on improved YOLOv8n, and relates to the technical field of target detection. Comprising the following steps: acquiring image data of tomatoes in a greenhouse within a preset time range, performing expansion preprocessing and enhanced expansion processing by using a SinGAN network to obtain processed tomato image data, and dividing the processed tomato image data into a training set and a test set; yOLOv8n is used as a backbone network, an SPD-Conv module is introduced into the front section of the backbone network, BoTNet and an attention mechanism are introduced into the tail end of the backbone network, a greenhouse tomato maturity detection model is constructed, and the trained greenhouse tomato maturity detection model is obtained through a training set and a test set; and inputting the processed tomato image data into the trained greenhouse tomato maturity detection model to obtain an indoor tomato maturity detection result.

Description

Technical Field

[0001] The present invention relates to the technical field of object detection, and particularly to a method for detecting the maturity of greenhouse tomatoes based on improved YOLOv8n. Background Art

[0002] Tomatoes, as a common and easily grown vegetable, play a very important role in global agricultural activities. Real-time monitoring of the growth and development process of tomatoes and ultimately predicting their yield are of great significance for tomato cultivation. This can not only help growers understand the expected harvest this year, but also help them effectively adjust cultivation methods and sales strategies. Traditional prediction methods often use relevant biological and mathematical knowledge to build tomato growth models. This requires modelers to collect a large amount of environmental data, such as temperature, humidity, and carbon dioxide concentration, and accurately measure plant indicators to simulate photosynthesis and respiration. The verification of the model requires destructive weighing of the dry weight of the fruits. However, with the rapid development of machine learning, artificial intelligence related to computer vision has gradually been applied to various industries, and precision agriculture and smart agriculture have become popular concepts. This provides growers with a new way to predict crop yields.

[0003] In recent years, scholars at home and abroad have conducted extensive research on tomato recognition and ripeness detection. The image processing technology for fruit detection generally goes through three development stages, namely traditional digital image processing, machine learning-based image processing, and deep learning-based image processing. In recent years, with the rapid development of machine learning and deep learning, especially the rapid development of convolutional neural networks in the field of image processing, deep learning-based object detection methods have gradually occupied an important position in the agricultural field. Among them, the mainstream deep learning-based object detection methods include RCNN, Faster R-CNN, Mask R-CNN, SDD, and the YOLO series. With the rapid iteration of the YOLO series, while maintaining its fast characteristics, its accuracy is also gradually improving. Long et al. and Wang et al. proposed an improved Mask R-CNN for identifying and segmenting apples with three different maturities in the orchard; Wang Z et al. improved Faster R-CNN for identifying and detecting greenhouse tomatoes. Although both of these networks can effectively detect fruits, due to the double-layer network structure of both networks, there is a problem of sacrificing speed for accuracy. The latter two networks, SSD and the YOLO series, belong to single-stage object detection networks and are faster than the two-stage networks. Xue et al. improved YOLOV2 for detecting immature mangoes, and this model is mainly used to overcome the difficulty of detecting occluded or overlapping mangoes. Liu el et al. used an improved YOLOv3 tomato recognition model to enable the model to correctly identify yellow tomatoes in a shaded environment. Yan et al. improved YOLOV5 for detecting apple fruits according to the problems that apple fruits are easily occluded and the fruits directly overlap with each other.

[0004] Although these studies have all played a good detection role, some problems have still been found. Since the training of deep neural networks requires data support, and it often takes a long time to obtain training images to balance the samples, the training images obtained in the short term usually have a certain degree of sample imbalance. In addition, the training images in these studies are usually close-up images, and in a complex greenhouse environment, not only the problem of overlapping occlusion in the near field needs to be solved, but also the problem of small targets in the distance needs to be noted. Summary of the Invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art in tomato detection in a complex greenhouse environment, such as small targets, many occlusions, and dense fruits, which make it difficult to identify, the main purpose of the present invention is to provide a method for detecting the maturity of greenhouse tomatoes based on improved YOLOv8n.

[0006] To achieve the above object, the present invention adopts the following technical solutions. A method for detecting the maturity of greenhouse tomatoes based on improved YOLOv8n includes: Obtain the tomato image data in the greenhouse within a preset time range, and use the SinGAN network to preprocess and enhance and expand the tomato image data in the greenhouse within the preset time range to obtain the processed tomato image data, which is divided into a training set and a test set; Take YOLOv8n as the backbone network, introduce the SPD-Conv module into the front section of the backbone network, introduce BoTNet and the attention mechanism into the end of the backbone network to construct a greenhouse tomato maturity detection model. The greenhouse tomato maturity detection model is trained through the training set and the test set to obtain a trained greenhouse tomato maturity detection model; Preprocess the tomato image data to be detected to obtain the preprocessed tomato image data to be detected, and input the preprocessed tomato image data to be detected into the trained greenhouse tomato maturity detection model to obtain the indoor tomato maturity detection result.

[0007] The backbone network of the YOLOv8n has 10 layers; The introduction of the SPD-Conv module into the front section of the backbone network means that the SPD-Conv module is inserted between the second and third layers, the fourth and fifth layers, the sixth and seventh layers, and the eighth and ninth layers of the backbone network, and the parameter dimension is set to 1; The attention mechanism is a multi-head self-attention mechanism.

[0008] The backbone network of the YOLOv8n includes: The first-layer standard Conv convolution module, the number of channels = 64, and the convolution kernel size = 3*3; The second-layer standard Conv convolution module, the number of channels = 128, and the convolution kernel size = 3*3; The third-layer C2f module, the number of channels = 128; The fourth-layer Conv standard convolution module, the number of channels = 256, and the convolution kernel size = 3*3; The fifth-layer C2f module, the number of channels = 256, and further fuse the feature maps; The sixth-layer Conv standard convolution module, the number of channels = 512, and the convolution kernel size = 3*3; The seventh-layer C2f module, the number of channels = 512; The eighth-layer Conv standard convolution module, the number of channels = 1024, and the convolution kernel size = 3*3; The ninth-layer C2f module, the number of channels = 1024; The tenth-layer SPPF module, the number of channels = 1024, and the parameter = 5.

[0009] The obtaining of the processed tomato image data includes the following steps: Randomly divide the obtained tomato image data in the greenhouse within the preset time range into a training set and a test set at a ratio of 9:1; Obtain the unbalanced sample data in the training set and the test set, input it into SinGAN for data augmentation, obtain the augmented data, randomly scale, flip, and rotate the augmented data at any angle to obtain processed data, and combine the processed data with the training set to obtain processed samples; Label the processed samples to obtain labeled samples; Perform enhancement processing on the expression samples, and the enhancement processing includes flipping, randomly adding noise, and brightness adjustment to obtain processed tomato image data.

[0010] A greenhouse tomato maturity detection system based on improved YOLOv8n, comprising: A data processing module, configured to obtain tomato image data in the greenhouse within a preset time range, perform preprocessing and enhancement and augmentation processing on the tomato image data in the greenhouse within the preset time range by using a SinGAN network, obtain processed tomato image data, and divide it into a training set and a test set; A model construction module, configured to use YOLOv8n as the backbone network, introduce the SPD-Conv module into the front section of the backbone network, introduce the BoTNet and attention mechanism into the end of the backbone network, construct a greenhouse tomato maturity detection model, and train the greenhouse tomato maturity detection model through the training set and the test set to obtain a trained greenhouse tomato maturity detection model; A maturity detection module, configured to preprocess the tomato image data to be detected, obtain the preprocessed tomato image data to be detected, input the preprocessed tomato image data to be detected into the trained greenhouse tomato maturity detection model, and obtain the indoor tomato maturity detection result.

[0011] Compared with the prior art, the beneficial effects of the present invention are as follows: By introducing the SPD-Conv module, BoTNet, and self-attention mechanism, the improved YOLOv8n model of the present invention can more accurately detect the maturity of tomatoes in a complex greenhouse environment. The SPD-Conv module helps improve the detection performance for small targets and low-pixel images, while BoTNet and the self-attention mechanism help the network better capture key feature information. Using the SinGAN generative adversarial network to augment unbalanced samples effectively increases the diversity and quantity of training data, helps the model learn richer features, and thus improves the generalization ability. The images generated by the SinGAN network can effectively solve the problem of sample imbalance in the training images obtained in the short term, enabling the model to better learn the features of tomatoes at various maturities during the training process. By introducing the SPD-Conv module and optimizing the network structure in the YOLOv8n model, the detection efficiency is further improved, making it suitable for real-time detection. It can better adapt to problems such as small targets, occlusions, and dense fruits in a complex greenhouse environment, and improves the robustness in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present application, form a part of the present application, and the schematic embodiments and descriptions thereof are used to explain the present application and do not constitute an improper limitation to the present application.

[0013] Figure 1 is a schematic structural diagram of the process of the present invention; Figure 2 is a schematic diagram of the process of equalizing data using the SinGAN network in the present invention; Figure 3 is a schematic structural diagram of the improved YOLOv8 network of the present invention; Figure 4 is a schematic diagram of a sample of the captured data in the embodiment of the present invention; Figure 5 is a schematic diagram of the process based on the SinGAN network structure of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0014] The present invention will be further described below in conjunction with the drawings and embodiments.

[0015] Embodiment: In order to solve the problem of long period of tomato image data collection in complex greenhouse environment, this paper proposes a method to balance and expand the data acquired in a short period of time by using SinGAN generative adversarial network. The difference between this method and other data expansion and enhancement methods is that it uses SinGAN generative adversarial network, which can train multiple generators to generate new samples similar to the input image in terms of spatial features, shape, etc. through a single image input. Figure 1 - Figure 5 , the specific steps of the technical method are:

[0016] Step 1: Sample unbalanced category samples from the training set divided by images collected in a short period of time. For example, the number of ripe tomatoes in tomato images collected in a short period of time is often greater than that of tomatoes of other maturity levels, resulting in category imbalance. From these, images of tomatoes of other maturity levels are cropped.

[0017] Step 2: Input the sampled imbalanced samples into the SinGAN generation network training, and generate multiple new samples with the help of the trained generator model.

[0018] Step 3: Combine (paste) the new sample into the original image acquired in the short term in step 1.

[0019] 2. In order to solve the problems of small targets, many occlusions, and dense fruits faced by tomato detection in complex greenhouse environments, the present invention proposes a greenhouse tomato detection model that improves YOLOv8n. This model is different from other models in that it introduces the SPD-Conv module, space-to-depth convolution, to improve the detection performance of low-pixel images and small targets. It introduces BoTNet, a bottleneck transformer for visual recognition, which introduces the self-attention mechanism while reducing parameters and delays. The specific steps of this method are:

[0020] Step 1: Enhance and expand the dataset of complex environment by flipping, mirroring, changing contrast, changing brightness, adding random noise and other traditional methods.

[0021] Step 2: Use the front-end backbone network of the YOLOv8n model that introduces the SPD-Conv module to extract feature information of the image data.

[0022] Step 3: Add BoTNet to the end of the YOLOv8n backbone network and add a self-attention mechanism to enable the network to better grasp the key information in the feature map and further improve the model performance. At this point, the model framework is complete.

[0023] Step 4: Input the training set into the improved model network for training. With the help of various metrics, conduct ablation experiments to verify the effectiveness of each module, and conduct comparative experiments with other versions of the YOLO series models at the same scale and different scales. Finally, obtain a greenhouse tomato detection model for complex environments.

[0024] Figure 1 In this, SinGAN represents a variant of the generative adversarial network (GAN), which is specifically designed to handle periodic data or data with periodic characteristics. In the standard GAN framework, SinGAN introduces the characteristics of the sine function to generate images or other types of data with periodic patterns.

[0025] Figure 3 In this, Conv represents the convolutional layer, which is used to extract local features of the input data. SPD-Conv represents the convolutional layer, and C2f represents the conversion layer (Channel to Feature Layer), which is used to convert channel information into feature information. Concat represents the concatenation layer, which is used to concatenate multiple tensors along the specified dimension. Upsample represents the upsampling layer, which is used to increase the resolution of the image, usually achieved by interpolation. SPPF represents the spatial pyramid pooling layer, which is used to perform pooling operations on regions of different sizes.

[0026] BoTNet represents a neural network structure that introduces the self-attention mechanism in the Transformer architecture into the bottleneck layer of the traditional convolutional neural network (CNN).

[0027] Example 2 In this example, a greenhouse tomato maturity detection method based on the improved YOLOv8n is applied. Among them, the data balancing method based on the SinGAN generation network includes the following steps: Step (1): Use image capture tools such as digital cameras, surveillance cameras, and mobile phone cameras to capture tomato-related images within a short period of 1 - 7 days. The image content includes tomatoes of various maturities, and the images are required to have various perspectives, not limited to close distances. Image examples are shown as a, b, and c in Picture 4.

[0028] Step (2): Randomly divide the image data obtained in Step (1) into a training set and a test set according to a ratio of 9:1.

[0029] Step (3): Identify which label samples have a small number and cause imbalance, and determine the target samples. Sample the target category samples, which generally means cutting them out from the original images. This method can be assisted by image processing tools such as PS, and the samples are shown in the figure. The number of samples collected should not be too much, about 20% of the number of samples of this type is sufficient. After sampling, perform background removal on these samples, which can also be completed with the help of tools such as PS and Pixian AI.

[0030] Step (4): Input the background-removed samples after sampling into the SinGAN generation network one by one for training.

[0031] SinGAN is a generative model based on generative adversarial networks (GANs). It can generate realistic images by only using a single natural image for training. The core of this model lies in its multi-scale generation strategy, that is, gradually generating high-resolution images from image patches of different resolutions. The network structure of SinGAN mainly consists of two core parts, namely the generator and the discriminator. The role of the generator is to convert random noise into images with natural image features, while the discriminator is responsible for distinguishing between the generated images and real images. In SinGAN, both parts of the network use multiple convolutional layers and have the same basic structure. The specific structure and working process are shown in Figure 5.

[0032] To reduce computational resource consumption and time cost, the parameter configuration of the SinGAN network is as follows: min_size == 25, where min_size refers to the minimum size of image cropping; max_size = 130, where max_size refers to the maximum size of image cropping and also the maximum size of the generated images; scale_factor == 0.75, where scale_factor refers to the scaling scale factor of each layer of the image; alpha == 10, where alpha refers to the weight of the reconstruction loss; niter == 1000, where niter refers to the number of training cycles for each scale.

[0033] Step (5), after each image is generated, select 10-20 generated images as balanced materials from the greenhouse tomato maturity detection method based on improved YOLOv8n. Use a python program to batch add these materials into the training images obtained in step (2) of the greenhouse tomato maturity detection method based on improved YOLOv8n. The programming criterion for this step requires that each material be added randomly and without repetition to a random position in the original image. During this process, the greenhouse tomato maturity detection method based on improved YOLOv8n will set a size range for the materials for random scaling. In addition, random flipping operations and random rotation operations of 0-180° will also be performed to ensure the diversity of the materials. A specific example is shown in Figure 6, and the newly generated samples are marked with red circles.

[0034] Step (6), use the labelme image annotation tool to annotate the tomatoes in the image, which can be divided into three categories, such as ripe tomatoes, semi-ripe tomatoes, and unripe tomatoes. When annotating, it is required that the annotation points enclose the tomato contour, or use the built-in ai annotation tool of labelme for annotation. The annotation example is shown in Figure 1d.

[0035] Step (7), perform traditional enhancement on the new data obtained in step (6) and the validation set obtained in step (2) again. In this step, the greenhouse tomato maturity detection method based on improved YOLOv8n still performs batch operations through python programming. The specific operations include flipping each image up and down once, adding random noise to each image, and adjusting the random brightness of each image. Merge the images obtained in step (6) to obtain the balanced dataset for training the greenhouse tomato maturity detection method based on improved YOLOv8n. A specific example is shown in Figure 7.

[0036] Similarly, obtain the validation set for training. So far, the data balancing method based on the SinGAN generation network ends.

[0037] For the greenhouse tomato maturity detection method based on the improved YOLOv8n, it specifically includes the following steps: Step (1), configure the programming environment for the computer. The programming environment is created using python3 with Anaconda3, and the network is built with the pytorch framework. Download the baseline model of YOLOv8 from the ultralytics official website and install the corresponding dependency libraries into the programming environment.

[0038] Step (2), introduce SPD-Conv into the backbone network of the YOLOv8n model to transfer the spatial information of the feature map to the depth. The backbone network of YOLOv8n has 10 layers, which are:

[0039] The first - layer standard Conv convolution module, with the number of channels = 64 and the convolution kernel size = 3*3, performs a convolution operation on the input image. The number of channels becomes 64, and the feature map is reduced to half of its original size; The second - layer standard Conv convolution module, with the number of channels = 128 and the convolution kernel size = 3*3, continues to perform a convolution operation on the feature map. The number of channels becomes 128, and the feature map is reduced to half of its original size again; The third - layer C2f module, with the number of channels = 128, fuses feature maps of different scales; The fourth - layer Conv standard convolution module, with the number of channels = 256 and the convolution kernel size = 3*3, performs a convolution operation on the feature map again. The number of channels becomes 256, and the feature map is reduced to half of its original size again; The fifth - layer C2f module, with the number of channels = 256, further fuses the feature map; The sixth - layer Conv standard convolution module, with the number of channels = 512 and the convolution kernel size = 3*3, continues to perform a convolution operation on the feature map. The number of channels becomes 512, and the feature map is reduced to half of its original size again; The seventh - layer C2f module, with the number of channels = 512, further fuses the feature map; The eighth - layer Conv standard convolution module, with the number of channels = 1024 and the convolution kernel size = 3*3, performs a convolution operation on the feature map again. The number of channels becomes 1024, and the feature map is reduced to half of its original size again; The ninth - layer C2f module, with the number of channels = 1024, further fuses the feature map; The tenth - layer SPPF module, with the number of channels = 1024 and parameter = 5, performs a spatial pyramid pooling operation to extract features at different scales, thereby capturing the detailed information of the target object at different scales.

[0040] The SPD - Conv module of the greenhouse tomato maturity detection method based on the improved YOLOv8n is inserted between the second and third, fourth and fifth, sixth and seventh, eighth and ninth layers as Figure 3 shown. The parameter dimension is set to 1. It performs a method of taking one pixel every other pixel for the height and width of the feature map tensor passed in from the previous layer, obtaining four parts, and then performing a concatenation in the depth space to achieve the operation of reducing the spatial dimension and increasing the depth dimension.

[0041] Step (3): Introduce BoTNet into the backbone network of the YOLOv8n model and add a self-attention mechanism. This module is inserted after the tenth-layer SPPF, with the number of channels = 1024. The specific process is to input the output of the tenth-layer SPPF module into two identical Conv modules respectively for convolution to obtain c1 and c2, input c1 into the multi-head self-attention mechanism for processing to emphasize important features and obtain m, and then perform a concatenation operation on m and c2. The concatenated feature map is input into a convolutional layer again, with the input number of channels being 2048 and the output number of channels being 1024, and finally the final output of the backbone network is obtained.

[0042] Step (4): Input the image into the improved YOLOv8n network for training, and verify the effectiveness of the modules introduced in steps (2) and (3) through ablation experiments.

[0043] Step (5): Compare with other YOLO series models, and finally obtain a tomato detection method that can handle complex greenhouse environments.

[0044] It should be noted that in the present invention, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device.

[0045] The above embodiments are only illustrative examples of the present invention and do not constitute a limitation on the protection scope of the present invention. Any design identical or similar to the present invention falls within the protection scope of the present invention.

Claims

1. A greenhouse tomato maturity detection method based on improved YOLOv8n, characterized in that: include: Acquire tomato image data in a greenhouse within a preset time range, use a SinGAN network to preprocess and enhance the tomato image data in the greenhouse within the preset time range, obtain processed tomato image data, and divide it into a training set and a test set; YOLOv8n is used as the backbone network, the SPD-Conv module is introduced into the front end of the backbone network, BoTNet and the attention mechanism are introduced into the end of the backbone network, and a greenhouse tomato maturity detection model is constructed. The greenhouse tomato maturity detection model is trained by a training set and a test set to obtain a trained greenhouse tomato maturity detection model; The tomato image data to be detected is preprocessed to obtain the preprocessed tomato image data to be detected, and the preprocessed tomato image data to be detected is input into the trained greenhouse tomato maturity detection model to obtain the indoor tomato maturity detection result.

2. The greenhouse tomato maturity detection method based on improved YOLOv8n as claimed in claim 1, characterized in that: The backbone network of YOLOv8n has 10 layers; The front end of introducing the SPD-Conv module into the backbone network is to insert the SPD-Conv module between the second layer and the third layer, between the fourth layer and the fifth layer, between the sixth layer and the seventh layer, and between the eighth layer and the ninth layer of the backbone network, wherein the parameter dimension dimension is set to 1; The attention mechanism is a multi-head self-attention mechanism.

3. The acquisition method of the greenhouse tomato maturity detection method based on improved YOLOv8n as claimed in claim 2 is characterized in that, The backbone network of YOLOv8n includes: The first layer is a standard Conv convolution module with 64 channels and 3*3 convolution kernel size. The second layer is a standard Conv convolution module with 128 channels and 3*3 convolution kernel size. The third layer C2f module, the number of channels = 128; The fourth layer Conv standard convolution module, number of channels = 256, convolution kernel size = 3*3; The fifth layer C2f module, with 256 channels, further fuses the feature maps; The sixth layer Conv standard convolution module, number of channels = 512, convolution kernel size = 3*3; The seventh layer C2f module, the number of channels = 512; The eighth layer Conv standard convolution module, number of channels = 1024, convolution kernel size = 3*3; Ninth layer C2f module, number of channels = 1024; The tenth layer SPPF module, number of channels = 1024, parameter = 5.

4. The greenhouse tomato maturity detection method based on improved YOLOv8n as claimed in claim 1, characterized in that: The step of obtaining the processed tomato image data comprises the following steps: The acquired greenhouse tomato image data within a preset time range is randomly divided into a training set and a test set in a ratio of 9:1; Obtain unbalanced sample data in the training set and the test set, input them into SinGAN for data expansion, obtain expanded data, randomly scale, flip and rotate the expanded data at any angle in sequence to obtain processed data, combine the processed data with the training set to obtain processed samples; Annotate the processed samples to obtain annotated samples; The expression sample is enhanced, and the enhancement processing includes flipping, randomly adding noise and adjusting brightness to obtain processed tomato image data.

5. A greenhouse tomato maturity detection system based on improved YOLOv8n, characterized in that: include: A data processing module is used to obtain tomato image data in a greenhouse within a preset time range, and use a SinGAN network to pre-process and enhance the tomato image data in the greenhouse within the preset time range to obtain processed tomato image data, and divide it into a training set and a test set; A model building module is used to use YOLOv8n as a backbone network, introduce an SPD-Conv module into the front end of the backbone network, introduce BoTNet and an attention mechanism into the end of the backbone network, and build a greenhouse tomato maturity detection model. The greenhouse tomato maturity detection model is trained by a training set and a test set to obtain a trained greenhouse tomato maturity detection model; The maturity detection module is used to preprocess the tomato image data to be detected, obtain the preprocessed tomato image data to be detected, input the preprocessed tomato image data to be detected into the trained greenhouse tomato maturity detection model, and obtain the indoor tomato maturity detection result.

6. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

7. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

Images