Tomato maturity detection system based on multi-scale feature fusion and implementation method
By introducing multi-scale feature fusion and improvements in small object detection heads in the YOLOv5s model, combined with the accelerated processing of ZCU104 platform and Vitis AI tool, the problem of low detection accuracy of tomato ripening in the existing technology is solved, and more efficient tomatoes are achieved Detection and management.
Patent Information
- Application Number
- CN202411468749.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-21
- Publication Date
- 2025-02-14
AI Technical Summary
In the prior art, the tomato ripening degree detection method has low recognition accuracy, and it is difficult to accurately detect the position and ripening degree of tomatoes in complex environments.
Using a tomato ripening detection system with multi-scale feature fusion, through the improvement of the YOLOv5s model, a new feature layer is added to the FPN+PAN structure, and a small object detection head is added to the Detect part, combining the ZCU104 computing platform and Vitis AI tool for neural network Network acceleration processing.
It improves the accuracy and accuracy of tomato ripening detection, and can detect the location and ripening of tomatoes in real time, helping farmers accurately control picking timing, reduce fruit loss, and improve the management efficiency and production quality of greenhouses.
Smart Images
Figure CN119445561A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a tomato maturity detection system and an implementation method, and in particular to a multi-scale feature fusion tomato maturity detection system and an implementation method, belonging to the technical field of target detection. Background Art
[0002] In recent years, China's tomato industry has flourished, with rapid growth in production capacity and a trend of diversified flavors becoming the industry's duet. In order to better increase tomato yields, it is necessary to dynamically and accurately detect the number, location and maturity of tomatoes. However, tomato fruits have different growth postures, overlap between fruits, and are severely blocked by leaves, branches, and fruit stalks. Unripe tomatoes are similar in color to leaves and vines, and the greenhouse lighting environment is complex. These factors have brought certain difficulties to robot recognition, resulting in problems such as imbalanced samples of different categories, small targets, and easy occlusion at different maturity stages in tomato fruit detection and segmentation in complex environments.
[0003] Based on this, the present invention proposes a tomato maturity detection system and implementation method that realizes real-time positioning and maturity judgment of tomatoes, and the improved algorithm can be deployed on edge computing devices for multi-scale feature fusion. Summary of the invention
[0004] A brief overview of the present invention is provided below in order to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention. It is not intended to identify key or important parts of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to a more detailed description discussed later.
[0005] In view of this, in order to solve the problem of low recognition accuracy of traditional tomato maturity detection methods in the prior art, the present invention provides a tomato maturity detection system and implementation method with multi-scale feature fusion.
[0006] Technical solution 1 is as follows: A method for detecting tomato maturity by multi-scale feature fusion, comprising the following steps:
[0007] S1. Collect tomato images at different maturity levels to construct a tomato dataset, and use open source software to annotate the tomato images in the tomato dataset with anchor boxes and classify them into categories to obtain the annotated tomato dataset;
[0008] S2. The labeled tomato dataset is divided into training set, validation set and test set. YOLOv5s is used as the basic model for improvement. A new feature layer is added to the FPN+PAN structure, and a small target detection head is added to the Detect part to obtain the improved YOLOv5s model.
[0009] S3. Activate the Pytorch environment, input the labeled tomato dataset into the improved YOLOv5s model for training, predict the position and maturity of the tomato fruit target, use the loss function to converge to obtain the optimal model weight, update the improved YOLOv5s model, and obtain a trained tomato maturity detection model;
[0010] S4. Using the trained tomato maturity detection model, the tomato maturity detection result is obtained, that is, the real-time position of the tomato and the real-time maturity of the tomato.
[0011] Furthermore, in S1, the categories include Mature, Semi-Mature and Immature. When the skin color of a tomato is dark red, the corresponding tomato image is labeled as Mature. When the skin color is orange-red, the corresponding tomato image is labeled as Semi-Mature. When the skin color is green, the corresponding tomato image is labeled as Immature.
[0012] Furthermore, in S2, the improvement process is as follows: through the top-down path of FPN, the 20×20 deep features are upsampled layer by layer and fused to the low-level features of 40×40, 80×80 and 160×160, and at the same time, through the bottom-up path of PAN, the 160×160 low-level high-resolution features are transferred layer by layer and fused to the high-level features of 80×80, 40×40 and 20×20, and a 160*160 small target detection head is added on the basis of the three detection heads of the original Detect part.
[0013] Furthermore, in the S3, the training process of the improved YOLOv5s model is as follows: the annotated tomato data set is subjected to feature extraction by the Backbone part, and the 2nd, 4th, 6th, and 9th network layers of the Backbone part output four feature maps of different scales and input them into the Neck part. Through the top-down path of the FPN structure of the Neck part, the feature maps of four different scales are subjected to multi-scale feature fusion. Through the bottom-up path of the PAN structure, multiple multi-scale feature maps from low layers to high layers are subjected to feature aggregation. Finally, the 21st, 24th, 27th, and 30th network layers output four feature maps of different scales and input them into the Detect part, and the position and maturity of the tomatoes are predicted. The model parameters and the number of iterations are adjusted using the loss function to converge to obtain the optimal model weights.
[0014] Technical solution 2 is as follows: a multi-scale feature fusion tomato maturity detection system, used to implement a multi-scale feature fusion tomato maturity detection method described in technical solution 1, including a ZCU104 computing platform and a Vitis AI tool;
[0015] The ZCU104 computing platform includes an ARM processing subsystem and an FPGA programmable logic part;
[0016] The ARM processing subsystem is used for image acquisition, storage, reading, scaling format and processing result display;
[0017] The FPGA programmable logic part is used to perform neural network acceleration processing on the improved YOLOv5s model;
[0018] The Vitis AI tool is used for neural network model format processing and size reduction.
[0019] The beneficial effects of the present invention are as follows: the present invention proposes a tomato maturity detection model based on the improved YOLOv5s model. When a small target detection head is added, although the overall calculation amount and parameters are increased, the accuracy is improved while ensuring the accuracy; the present invention can detect the position and maturity of tomatoes in real time, help farmers accurately control the picking time, thereby reducing fruit loss and improving the management efficiency and production quality of greenhouses; the present invention proposes a tomato maturity detection system with multi-scale feature fusion, uses the Vitis AI tool to convert the format of the YOLOv5s basic model, and uses the ZCU104 platform as the mounting platform, combining the computing power of ARM and the parallel processing capability of FPGA, and accelerates the improved YOLOv5s model through DPU, which meets the requirements of the greenhouse operation platform for real-time target recognition;
[0020] According to the tomato maturity detection results, the accuracy of the improved model is judged from the precision, recall, and mean average precision (mAP) indicators. Referring to Table 1, the comparison results of the model proposed in the present invention and the YOLOv5s basic model, and referring to Table 2, the quantitative comparison results of the model proposed in the present invention and the YOLOv5s basic model, it can be obviously obtained that the results obtained by the model of the present invention are more accurate.
[0021] Table 1
[0022] Model Precision Recall mAP50 mAP50-95 YOLOv5s 0.993 0.989 0.994 0.864 The present invention 0.994 0.989 0.994 0.872
[0023] Table 2
[0024] Model Precision Recall mAP50 mAP50-95 YOLOv5s 0.974 0.922 0.960 0.718 The present invention 0.967 0.920 0.959 0.769 BRIEF DESCRIPTION OF THE DRAWINGS
[0025] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0026] Figure 1 It is a flowchart of a tomato maturity detection system and implementation method based on multi-scale feature fusion;
[0027] Figure 2 This is a schematic diagram of the overall network structure of the improved YOLOv5s model;
[0028] Figure 3 It is a schematic diagram of the structure of a tomato maturity detection system based on multi-scale feature fusion;
[0029] Figure 4 The present invention is a flowchart of an embodiment of a tomato maturity detection system and implementation method based on multi-scale feature fusion;
[0030] Figure 5 Schematic diagram of the types of bottleneck modules for the backbone and neck parts. DETAILED DESCRIPTION
[0031] In order to make the technical solutions and advantages of the embodiments of the present invention more clearly understood, the exemplary embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.
[0032] Example 1: Reference Figure 1-Figure 4 The present embodiment is described in detail, a method for detecting tomato maturity by multi-scale feature fusion, which specifically includes the following steps:
[0033] S1. Collect tomato images at different maturity levels to construct a tomato dataset, and use open source software to annotate the tomato images in the tomato dataset with anchor boxes and classify them into categories to obtain the annotated tomato dataset;
[0034] S2. The labeled tomato dataset is divided into training set, validation set and test set. YOLOv5s is used as the basic model for improvement. A new feature layer is added to the FPN+PAN structure, and a small target detection head is added to the Detect part to obtain the improved YOLOv5s model.
[0035] S3. Activate the Pytorch environment, input the labeled tomato dataset into the improved YOLOv5s model for training, predict the position and maturity of the tomato fruit target, use the loss function to converge to obtain the optimal model weight, update the improved YOLOv5s model, and obtain a trained tomato maturity detection model;
[0036] S4. Using the trained tomato maturity detection model, the tomato maturity detection result is obtained, that is, the real-time position of the tomato and the real-time maturity of the tomato.
[0037] Specifically, refer to Figure 2 , Backbone part (backbone network), Neck part (neck network), Detect part (detection layer);
[0038] CBS: The CBS layer consists of three modules: Convolution, Batch Normalization, and Activation. In CBS, the activation function is the SiLU activation function. x is the CBS layer input;
[0039] C3: C3 layer (CSPBottleneck with 3convolutions, three convolutions cross-stage local bottleneck, CSP is cross-stage partial network, Bottleneck is bottleneck structure, the difference between C3 in the backbone part and C3 in the neck part is mainly the type of internal bottleneck module, the former is bottleneck1, the latter is bottleneck2, C3_X in the backbone part, X represents the number of times bottleneck1 is repeated, C3_X_F in the neck part, X represents the number of times bottleneck2 is repeated, and F represents bottleneck2;
[0040] SPPF: Fast Spatial Pyramid Pooling Module;
[0041] Upsample: upsampling module;
[0042] Concat: concatenation module;
[0043] Conv: convolution module;
[0044] Detect: detection layer;
[0045] The 2nd, 4th, 6th, and 9th layer networks are C3_1, C3_2, C3_3, and SPPF respectively, and the 21st, 24th, 27th, and 30th layer networks are C3_1_F, C3_1_F, C3_1_F, and C3_1_F respectively;
[0046] Furthermore, in S1, the categories include Mature, Semi-Mature and Immature. When the skin color of a tomato is dark red, the corresponding tomato image is labeled as Mature (mature); when the skin color is orange-red, the corresponding tomato image is labeled as Semi-Mature (semi-mature); when the skin color is green, the corresponding tomato image is labeled as Immature (immature).
[0047] Specifically, the specific process of constructing the tomato dataset is as follows: in a greenhouse environment, tomato images are collected using smart devices (such as mobile phones, cameras, etc.) in the morning and afternoon respectively. Data enhancement technology is used to pre-process the collected tomato images using methods such as flipping, rotating, brightness adjustment, and adding Gaussian noise and salt and pepper noise, obtaining a total of 2700 images.
[0048] Furthermore, in S2, the improvement process is as follows: through the top-down path of FPN, the 20×20 deep features are upsampled layer by layer and fused to the low-level features of 40×40, 80×80 and 160×160, and at the same time, through the bottom-up path of PAN, the 160×160 low-level high-resolution features are transferred layer by layer and fused to the high-level features of 80×80, 40×40 and 20×20, and a 160*160 small target detection head is added on the basis of the three detection heads of the original Detect part.
[0049] Specifically, the Neck part of the YOLOv5s basic model is composed of FPN and PAN structures. The FPN structure realizes feature fusion from high-level to low-level feature maps through a top-down path, that is, firstly extracting multi-layer feature maps through the backbone network Backbone part, namely CSPDarknet53, and then splicing multiple feature maps from top to bottom through upsampling and feature fusion modules. The PAN structure passes the low-level spatial information upward to the high-level feature map through a bottom-up path aggregation method, that is, firstly reducing the resolution of the low-level feature map through the downsampling module, and then splicing multiple feature maps from bottom to top through the feature aggregation module, so that the low-level feature map and the high-level feature map are fused, and finally the feature map is processed through the convolution layer to further compress the fused multi-scale information.
[0050] Furthermore, in the S3, the training process of the improved YOLOv5s model is as follows: the annotated tomato data set is subjected to feature extraction by the Backbone part, and the 2nd, 4th, 6th, and 9th layers of the Backbone part output four feature maps of different scales and input them into the Neck part, and the four feature maps of different scales are subjected to multi-scale feature fusion through the top-down path of the FPN structure of the Neck part, and the multiple multi-scale feature maps from the low layer to the high layer are subjected to feature aggregation through the bottom-up path of the PAN structure, and finally the 21st, 24th, 27th, and 30th layers of the network output four feature maps of different scales and input them into the Detect part, and the position and maturity of the tomato are predicted, and the loss function Loss is used to adjust the model parameters and the number of iterations, and the optimal model weights are obtained by convergence;
[0051] Specifically, in this embodiment, training is started after activating the pytorch environment, the pixel size of the tomato image in the annotated tomato data set is 3*3*640*640, the batch size is 4, and the number of iterations is 100. During the training process of the improved YOLOv5s model, the training weights are automatically saved after each iteration, the training loop is exited according to the number of iterations, the model parameters, i.e., the weights, are saved, the optimal model weights are selected, and the trained tomato maturity detection model is updated;
[0052] In the Backbone part, in addition to the three feature maps of different sizes generated by the original 4th, 6th, and 9th layers of the network being input into the Neck part, the feature map generated by the 2nd layer of the network is also input into the Neck part. Step S4 can effectively reduce missed detections and ensure that more tomato instances are correctly identified and classified;
[0053] The loss function Loss is expressed as:
[0054] Loss = l box +l cls +l obj
[0055] Among them, l box is the position loss of the target box and the predicted box, l cls is the classification loss, l obj is the confidence loss;
[0056] The position loss of the target box and the predicted box l box It is expressed as:
[0057]
[0058] Among them, b is the center point of the prediction box, b gtis the center point of the target box, ρ is the Euclidean distance, c is the diagonal length of the minimum bounding box covering the two boxes, and v represents the consistency of the measured aspect ratio;
[0059] Classification loss cls It is expressed as:
[0060]
[0061] Among them, S 2 is the number of grid units, that is, each image is divided into S×S grids, is the first indicator variable. When there is an object in the i-th grid cell, the value is 1, otherwise it is 0. The probability of predicting category c for the i-th grid cell, P i (c) is the label of the true category c;
[0062] Confidence loss l obj It is expressed as:
[0063]
[0064] Where B represents the number of predicted bounding boxes for each grid cell, is the second indicator variable, which takes the value of 1 when there is no object in the i-th grid cell and the j-th boundary. is the confidence of the prediction, that is, the probability of whether the object exists, C i is the true confidence, that is, whether there is an object label, a value of 1 means there is an object, a value of 0 means there is no object, λ noobj To adjust the weight factor of the confidence loss of no object, it is usually set to a smaller value to reduce the impact of the background area on the loss.
[0065] Example 2: Reference Figure 3 and Figure 4 This embodiment is described in detail, a tomato maturity detection system with multi-scale feature fusion, which is used to implement a tomato maturity detection method with multi-scale feature fusion described in Example 1, including a ZCU104 computing platform and a VitisAI tool;
[0066] The ZCU104 computing platform includes an ARM processing subsystem and an FPGA programmable logic part;
[0067] The ARM processing subsystem is used for image acquisition, storage, reading, scaling format and processing result display;
[0068] The FPGA programmable logic part is used to perform neural network acceleration processing on the improved YOLOv5s model;
[0069] The VitisAI tool is used for neural network model format processing and size reduction.
[0070] Specifically, the tomato maturity detection system is built based on the SoCFPGA detection system, namely the ZCU104 platform. The Vitis AI tool includes an AI quantizer, an AI compiler, and an AI runtime. The AI quantizer is used to convert a floating-point neural network model into a fixed-point neural network model, and the AI compiler is used to convert a fixed-point neural network model into DPU instructions. The AI compiler is used to accelerate the reasoning of the neural network model.
[0071] refer to Figure 3 The ZCU104 platform includes FPGA programmable logic part PL (Programmable Logic), ARM processing subsystem PS (Processing System), 504K system logic unit, quad-core ARM CortexA53 application processor and dual-core Cortex-R5 real-time processor, and includes Deep Learning Processor Unit (DPU), which can support different neural network structures through reconfigurable hardware architecture;
[0072] To achieve real-time processing of video acquisition data, the ZCU104 platform is equipped with USB, Micro SD, DDR and DisplayPort, as well as a UART interface for debugging. The ARM processing subsystem and the FPGA programmable logic part can communicate through the AXI bus. The PS side is mainly responsible for the acquisition, storage, reading, scaling format conversion, and processing result display of the target image. The processed image will be stored in the DDR, and the DPU on the PL side will be called to perform neural network acceleration processing on the trained tomato maturity detection model.
[0073] Based on the trained tomato maturity detection model, the real-time image is recognized on the tomato maturity detection system to obtain the tomato maturity detection result;
[0074] VitisAI tool is a comprehensive set of software tools provided by Xilinx, which can accelerate the reasoning of deep learning models on Xilinx FPGA platform. The present invention uses AI quantizer (Quantizer), AI compiler (Compiler) and AI runtime (Runtime) in VitisAI tool. AI quantizer reduces the calculation and storage overhead of the model by converting 32-bit floating point numbers into INT8 fixed point numbers. AI compiler converts the quantized deep learning model into a hardware executable format and optimizes it to adapt to the computing architecture of FPGA. AI runtime (Runtime) loads and executes the compiled deep learning model on Xilinx FPGA to ensure the efficient operation and performance optimization of the model on hardware.
[0075] Reference Figure 4 In the process of quantizing the model, the tomato images in the validation set are used as the calib calibration data set. The quantization configuration file is generated by the written Python quantization script, and the calibration data set is used to calculate the quantization parameters of the model, determine the scaling factors of the activation value and weight, and use the test test data set to perform reasoning verification on the quantized model to evaluate its accuracy and performance. Finally, the quantized .xmodel file is generated to achieve fixed-point and efficient reasoning of the model. The compilation process of the model is the process of converting the fixed-point model into DPU instructions. According to the model of the SoC FPGA platform, the ZCU104.json file containing the specific configuration information of the DPU is added, and the AI compiler is used for compilation to output the .xmodel instruction file required by the DPU. In the quantization process, the weights and activation values of the model are converted from 32-bit floating point numbers to lower-precision INT8 fixed-point numbers, which will cause a decrease in accuracy.
[0076] Connect the USB camera of the camera to the USB port of ZCU104, use the Python program to call the OpenCV library, DPU library functions and APIs in the ARM processor on the PL side for image preprocessing, and then use the AI runtime of the VitisAI tool to perform DPU-accelerated improved YOLOv5s neural network inference to achieve real-time detection of tomato maturity.
[0077] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is selected primarily for readability and teaching purposes, rather than for explaining or defining the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is illustrative, not restrictive, with respect to the scope of the present invention, which is defined by the appended claims.
Claims
1. A method for detecting tomato maturity based on multi-scale feature fusion, characterized in that: The following steps are involved: S1. Collect tomato images at different maturity levels to construct a tomato dataset, and use open source software to annotate the tomato images in the tomato dataset with anchor boxes and classify them into categories to obtain the annotated tomato dataset; S2. The labeled tomato dataset is divided into training set, validation set and test set. YOLOv5s is used as the basic model for improvement. A new feature layer is added to the FPN+PAN structure, and a small target detection head is added to the Detect part to obtain the improved YOLOv5s model. S3. Activate the Pytorch environment, input the labeled tomato dataset into the improved YOLOv5s model for training, predict the position and maturity of the tomato fruit target, use the loss function to converge to obtain the optimal model weight, update the improved YOLOv5s model, and obtain a trained tomato maturity detection model; S4. Using the trained tomato maturity detection model, the tomato maturity detection result is obtained, that is, the real-time position of the tomato and the real-time maturity of the tomato.
2. The method for realizing tomato maturity detection based on multi-scale feature fusion according to claim 1, characterized in that: In S1, the categories include Mature, Semi-Mature and Immature. When the skin color of a tomato is dark red, the corresponding tomato image is labeled as Mature. When the skin color is orange-red, the corresponding tomato image is labeled as Semi-Mature. When the skin color is green, the corresponding tomato image is labeled as Immature.
3. The method for realizing tomato maturity detection based on multi-scale feature fusion according to claim 2, characterized in that: In S2, the improvement process is as follows: through the top-down path of FPN, the 20×20 deep features are upsampled layer by layer and fused to the low-level features of 40×40, 80×80 and 160×160. At the same time, through the bottom-up path of PAN, the low-level high-resolution features of 160×160 are transferred layer by layer and fused to the high-level features of 80×80, 40×40 and 20×20. A 160*160 small target detection head is added on the basis of the three detection heads of the original Detect part.
4. The method for realizing tomato maturity detection by multi-scale feature fusion according to claim 3 is characterized in that: In the S3, the training process of the improved YOLOv5s model is as follows: the annotated tomato data set is subjected to feature extraction by the Backbone part, and the 2nd, 4th, 6th, and 9th network layers of the Backbone part output four feature maps of different scales and input them into the Neck part. Through the top-down path of the FPN structure of the Neck part, the feature maps of four different scales are subjected to multi-scale feature fusion. Through the bottom-up path of the PAN structure, multiple multi-scale feature maps from low layers to high layers are subjected to feature aggregation. Finally, the 21st, 24th, 27th, and 30th network layers output four feature maps of different scales and input them into the Detect part, and the position and maturity of the tomatoes are predicted. The model parameters and the number of iterations are adjusted using the loss function to converge to obtain the optimal model weights.
5. The tomato maturity detection system based on multi-scale feature fusion according to claim 4, characterized in that: A method for realizing tomato maturity detection by multi-scale feature fusion as described in any one of claims 1 to 4, comprising a ZCU104 computing platform and a Vitis AI tool; The ZCU104 computing platform includes an ARM processing subsystem and an FPGA programmable logic part; The ARM processing subsystem is used for image acquisition, storage, reading, scaling format and processing result display; The FPGA programmable logic part is used to perform neural network acceleration processing on the improved YOLOv5s model; The Vitis AI tool is used for neural network model format processing and size reduction.