Deep learning-based tomato maturity detection system and method

Abstract

The invention discloses a tomato maturity detection system and method based on deep learning, and relates to the technical field of fruit and vegetable detection models. Comprising the following steps: acquiring data from a robot perspective based on a natural scene of a tomato plantation, and making a tomato maturity detection data set; on the basis of a YOLO v5s model, an improved model YOLOv5s BCD model fusing a CA attention mechanism, a BiFPN neck feature fusion network and a Decup led Head head decoupling network is provided and serves as a tomato maturity detection model of the invention; experimental research of model training and effect comparison is introduced, the conditions of model evaluation and optimization are performed through a model comparison test, the tomato maturity grade is judged on the basis of the fruit red surface coloring proportion and in combination with fruit appearance characteristics, tomatoes with the required maturity grade are collected, the light weight and accuracy of edge side deployment are adapted, and the tomato maturity grade evaluation method is suitable for the tomato maturity grade evaluation and the tomato maturity grade evaluation and the tomato maturity grade evaluation and the tomato maturity grade evaluation and the tomato maturity grade evaluation. The system can be used as a visual system of a tomato harvesting robot, and technical support is provided for automatic picking.

Description

Technical Field

[0001] The present invention relates to the technical field of fruit and vegetable detection models, and in particular to a tomato maturity detection system and method based on deep learning. Background Art

[0002] With the development of smart agricultural technology, robots are gradually entering the fruit and vegetable planting industry, carrying out operations such as fruit and vegetable transplanting, plant protection and harvesting. Visual technology is a key technology for environmental perception of fruit and vegetable harvesting robots. It can solve the positioning and posture of fruit and vegetable targets in natural scenes, fruit and vegetable maturity and quality detection, weed and pest detection, and robot visual navigation in fruit and vegetable garden environments. Tomatoes are one of the most popular fruits and vegetables in my country. They are widely planted in my country because of their high nutritional value and delicious taste. The quality of tomatoes depends on their harvesting period and storage and transportation conditions. If they are picked at low maturity, although the storage and transportation time can be extended, the quality and taste are not good. If they are picked at high maturity, they need to be sold quickly, otherwise they will affect the taste due to over-ripening. Before the robot automatically harvests tomatoes, it is necessary to perform maturity grading and differentiated harvesting according to market demand and storage and transportation plans, which is of great significance to improving the quality and taste of tomatoes and increasing farmers' income.

[0003] At present, the main methods for detecting the maturity of fruits and vegetables include machine vision detection, spectral imaging detection, electronic nose detection, and comprehensive evaluation methods such as multi-source information fusion, which are applied to the harvesting of common fruits and vegetables such as tomatoes, strawberries, and green peppers. In the existing technology, research is carried out from the visual perspective of the color of fruits and vegetables. The above methods mostly use traditional feature extraction and classification for maturity detection, which is difficult to adapt to complex situations in natural scenes due to lighting, occlusion of branches and leaves, and superposition of fruits in clusters. It is also difficult for electronic noses to perform maturity detection through the external odor characteristics of fruits and vegetables in natural scenes.

[0004] Fruit and vegetable maturity detection based on deep learning can achieve positioning and classification accuracy in complex natural scenes. However, fruit and vegetable maturity detection is used to realize automatic harvesting by robots. The model is deployed on the robot side. The model is large in scale and has many parameters. The inference time is long and the computing resources are high, making it difficult to implement terminal deployment. The existing maturity detection model design must consider lightweight design while improving the detection accuracy. A compromise must be made between model accuracy and lightweight, and a high-precision system can be judged from visual perspectives such as the color of fruits and vegetables.

[0005] In the prior art, the publication number CN117975442A discloses a lightweight tomato maturity detection system and method based on improved YOLOv5s, the system includes a backbone network, a neck structure and a head structure; the backbone network includes a Ghostconv module, a C3Ghost module and an SPPF module, and the Bott leneck module in the original C3 module is replaced by a GhostBott leneck module; the PANet of the neck structure is replaced by a weighted bidirectional feature pyramid network BiFPN; the head structure uses an EIoU loss function to replace the CIoU loss function. This solution solves the problem that the PANet structure of the YOLOv5s model in the prior art does not consider the different contributions of different feature information in the fusion process during the feature fusion process, and does not horizontally connect the fused feature information, resulting in a weakened expression ability of multi-scale features and a reduced detection accuracy, but does not mention the use of the attention mechanism, and is weak in focusing on important features, especially in the case of complex backgrounds or occlusions. The attention mechanism can significantly improve the robustness and accuracy of the model. At the same time, the head structure design uses the EIoU loss function to replace the CIoU loss function, which has poor flexibility and efficiency when processing classification and positioning tasks. Therefore, the detection system not only wastes resources, but also reduces the accuracy and effectiveness of the detection system.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute the prior art that is already known to one of ordinary skill in the art. Summary of the invention

[0007] The purpose of the present invention is to provide a tomato maturity detection system and method based on deep learning to solve the problems raised in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] A tomato maturity detection system based on deep learning, specifically comprising:

[0010] The sample image module is used to collect a number of sample images of natural scenes of a tomato garden under different weather conditions and at different time periods during the day, and mark the position of each single tomato fruit and the proportion of the fruit with red surface in each sample image, wherein each sample image contains at least one tomato, and the proportion of the fruit with red surface is the proportion of the area of ​​the fruit that changes from cyan to red to the entire area of ​​the single tomato fruit;

[0011] Model building module, used to build the YOLOv5s BCD model. The YOLOv5s BCD model consists of four parts: input end, backbone network, neck network and head network. A bidirectional pyramid network is added to the neck network, a coordinate attention mechanism is introduced into the head network of the YOLOv5s model, and a decoupled head design is adopted in the head network of the YOLOv5s model.

[0012] The model training module is used to take the labeled sample images as the input of the improved YOLOv5s BCD model, and take the position of the tomato fruit and the proportion of the red surface of the fruit in the image as labels to train the YOLOv5s BCD model;

[0013] The model detection module is used to input the tomato image to be identified into the trained YOLOv5s BCD model, output the tomato position information and the proportion of the red surface of the fruit, judge the tomato maturity level based on the appearance characteristics of the fruit, and collect tomatoes of the required maturity level.

[0014] Further, the collected images are used as sample images to form a sample image set; the position of each tomato fruit and the red area of ​​the fruit in each sample image are marked, wherein the Labelme annotation software is used for image annotation, and the specific annotation rules are as follows: the position of each single fruit is marked and the proportion of the red surface of the fruit is marked; the fruit that is blocked by stems, leaves or other fruits and whose maturity cannot be identified is not marked; the fruit that is blocked by stems, leaves or other fruits and whose blocked area exceeds two-thirds is not marked; the fruit that is too far away to identify its maturity is not marked. If there is no single tomato fruit that meets the annotation rules in a sample image, the image is removed from the sample image set;

[0015] The logic for obtaining the red surface ratio of the fruit is: converting the sample image from the RGB color space to the HSV color space, specifically: calculating the image feature data according to the intensity values ​​of the red, green and blue channels of each pixel. The specific calculation formula is as follows:

[0016]

[0017] In the formula, atan2 is the inverse tangent function, which is used to calculate the inverse tangent value of two variables, H p , S p and V p are the hue, saturation and brightness of the pth pixel in the visible light image, R p Represents the intensity value of the red channel of the pth pixel, G p Indicates the intensity value of the green channel of the pth pixel, B p Represents the intensity value of the blue channel of the p-th pixel.

[0018] After converting to the HSV color space, set the red color threshold range, that is:

[0019] H min ≤H p ≤H max

[0020] S min ≤S p ≤S max

[0021] V min ≤V p ≤V max

[0022] In the formula, H min and H max are the minimum and maximum values ​​of the red color hue threshold range, S min and S max are the minimum and maximum values ​​of the red color saturation threshold range, V min and V max are the minimum and maximum values ​​of the red color brightness threshold range respectively; the pixels in the image that meet the set red color threshold range are recorded, and the area formed by all recorded pixels is calculated. The specific formula is:

[0023] A rad =N rad *A PRIX

[0024] In the formula, A rad is the red fruit area, N rad To record the total number of pixels, A PRIX is the actual area represented by each pixel;

[0025] The red surface ratio of the fruit is calculated based on the red surface area of ​​the fruit. The specific formula is:

[0026]

[0027] In the formula, R rad is the red surface ratio of the fruit, A total is the total area of ​​a single tomato fruit in the image plane.

[0028] Furthermore, a bidirectional pyramid network is added to the neck network for multi-scale feature fusion. The specific logic is as follows: the neck network only uses three feature layers, namely P3, P4 and P5 layers of the Backbone network, to achieve cross-layer connection and weighted feature fusion of three-layer features. The network is simplified by removing nodes with only one input edge, adding an edge between feature layers at different layers but with the same resolution to achieve cross-layer connection and finally form a BiFPN layer, and stacking the BiFPN layer three times. The BiFPN is a weighted bidirectional feature pyramid network. BiFPN adopts a weighted feature fusion method to obtain weights of different layers through learning, and the formula for limiting the weights of different layers is as follows:

[0029]

[0030] Where O is the limiting threshold of the weights of different layers, ω i ∈(ω1,ω2,ω3),ω j ∈(ω1,ω2,ω3), where ω1 is the weight of the P4 feature layer in the Backbone network, ω2 is the weight of the P5 feature layer in the Backbone network, and ω3 is the weight of the P3 feature layer in the Backbone network. i ∈(I1,I2,I3), I1 is the learnable weight of the P4 feature layer in the Backbone network, I2 is the learnable weight of the P5 feature layer in the Backbone network, I3 is the learnable weight of the P3 feature layer in the Backbone network, ε is the initial learning rate, where i and j are the indexes of the feature layers in the Backbone network, and i∈(1,2,3), and j∈(1,2,3).

[0031] Furthermore, feature fusion is performed in the BiFPN layer, where the formula for feature fusion is:

[0032]

[0033] In the formula, represents the output of node m4 in the BiFPN layer, is the output of node o4 in the BiFPN layer, Conv represents the convolution operation, ω1 is the weight of the P4 feature layer in the Backbone network, ω2 is the weight of the P5 feature layer in the Backbone network, is the output of node i5 in the BiFPN layer, resize represents upsampling and downsampling operations, is the output of node o3 in the BiFPN layer, ω′1, ω′2 and ω′3 represent the weights of node i4, node m4 and node o3 on the output of node o4 respectively; after the feature fusion of three stacked BiFPN layers, a multi-scale fusion feature map is output;

[0034] Among them, the nodes i3, i4 and i5 are the feature input nodes of the feature layers P3, P4 and P5 respectively, the node m4 is the feature fusion node of nodes i4 and i5, the node o3 is the output node after the features of the feature input node i3 and the feature fusion node m4 are fused, and the node o4 represents the output node after the features of the feature fusion node m4, the feature input node i4 and the output node o3 are fused.

[0035] Furthermore, the attention weights are generated through the coordinate attention mechanism, and the logic of obtaining the weighted feature map is as follows: the input multi-scale fusion feature map is first encoded in each channel in the width and height directions through the H×1 and 1×W pooling kernels respectively, the global receptive field is obtained and the channel and position information are encoded. The formula for the encoding output along the height and width directions is:

[0036]

[0037] In the formula, is the output after the pooling operation in the width direction, representing the encoding of the Cth channel in the height direction, is the output after the pooling operation in the height direction, representing the encoding of the Cth channel in the width direction. W and H represent the width and height of the multi-scale fusion feature map, respectively. c (H, p) represents the element value at the height H and width p position on the Cth channel of the feature map, X c (W,q) represents the element value at the width W and height q position on the Cth channel of the feature map, where C is the index of the multi-scale fusion feature map channel, and p and q are the indexes of the width and height of the multi-scale fusion feature map respectively;

[0038] The two encoded outputs are then concatenated and convolved according to the following formula:

[0039]

[0040] In the formula, f represents the tensor obtained by cascading and convolution operations on the encoded output, δ(·) represents the nonlinear Relu operation, F1 is the 1×1 convolution Conv operation, Represents the cascade Concatenate operation along the channel dimension, where the channel dimension is controlled to N / r, N is the number of channel dimensions, and r is the scale factor;

[0041] Then perform a Split operation on the output result to get f H and f W Two tensors, where f H and f WThey represent the tensor containing the feature information in the height direction and the tensor containing the feature information in the width direction respectively. H and f W The two tensors are respectively subjected to 1×1 convolution and sigmoid function activation operations, and finally two weighted tensors g with the same latitude as the multi-scale fusion feature map are obtained. H and g W , as the attention weight, perform element-by-element multiplication on each channel of the multi-scale fusion feature map based on the attention weight of each channel. The final output of the coordinate attention mechanism module is expressed as:

[0042]

[0043] In the formula, y C (p,q) represents the value of the Cth channel at position (p,q) in the final output feature map. and They represent the weight of the Cth channel at the height position q and the weight at the width position p respectively.

[0044] Furthermore, a decoupled head design is adopted, in which the network output of the decoupled model design of the head includes three parts: Classification, Regression and Objectness. After fusion, the P3-5 layers of the feature map enter the coordinate attention mechanism module of the head respectively, and are output to the head decoupled model with three different granularities. The specific workflow of the head decoupled model is as follows: first, the number of feature channels is reduced to 256 using the CBS1×1 network, and then it is divided into two parallel branches to learn the classification and regression tasks respectively through the CBS3×3 network, and the Objectness branch is added to the regression task. Finally, the Conv1×1 convolution layer is used to achieve dimensional matching of the output parameters. The model concatenates the output tensors into a tensor of H×W×(N+1+4) dimensions, and finally concatenates the output tensors of the three decoupled heads with different granularities, where CBS represents the combination of three operations: convolutional layer, batch normalization and activation function.

[0045] Furthermore, the logic for judging the maturity level of tomatoes based on the red surface ratio of the fruit and the appearance characteristics of the fruit is as follows: extracting the appearance characteristic data of the fruit, wherein the appearance characteristic data of the fruit includes the roundness of the fruit and the roughness of the fruit texture, wherein the formula for calculating the roundness and the roughness is as follows:

[0046]

[0047]

[0048] Where YD is the roundness of the fruit, ZP is the circumference of the fruit, AAR is the fruit texture roughness, M is the total number of pixels in the single tomato image, and Zβ is the gray value of the β-th pixel, and Z mean is the average value of the gray values of all pixels, and β is the index of the pixels in the single tomato fruit image;

[0049] Generate a maturity coefficient based on the proportion of the red-colored surface of the fruit, the roundness of the fruit, and the roughness of the fruit texture. The formula for calculating the maturity coefficient is as follows:

[0050]

[0051] In the formula, SD is the maturity coefficient, and k1, k2, and k3 are the weight coefficients of the proportion of the red-colored surface of the fruit, the roundness of the fruit, and the roughness of the fruit texture respectively, where k1 > k2 > k3 and k1, k2, and k3 are all greater than 0;

[0052] The logic for judging the tomato maturity level based on the generated maturity coefficient is as follows:

[0053] When 0 < SD ≤ 0.1 * yz, judge that the tomato maturity level is the unripe stage;

[0054] When 0.1 * yz < SD ≤ 0.3 * yz, judge that the tomato maturity level is the green-ripe stage;

[0055] When 0.3 * yz < SD ≤ 0.5 * yz, judge that the tomato maturity level is the color-changing stage;

[0056] When 0.5 * yz < SD ≤ 0.6 * yz, judge that the tomato maturity level is the early red-ripe stage;

[0057] When 0.6 * yz < SD ≤ 0.8 * yz, judge that the tomato maturity level is the mid red-ripe stage;

[0058] When 0.8 * yz < SD ≤ 1.0 * yz, judge that the tomato maturity level is the late red-ripe stage.

[0059] The present invention also provides a tomato maturity detection method based on deep learning. The tomato maturity detection method based on deep learning is used to control the above-mentioned tomato maturity detection system based on deep learning. The specific steps include:

[0060] Collect sample images of the natural scene of the tomato orchard at different times of the day under different weather conditions, and mark the position and the proportion of the red-colored surface of each single tomato fruit in each sample image. Each sample image contains at least one tomato, and the proportion of the red-colored surface of the fruit is the proportion of the area where the fruit changes from green to red in the area of the whole single tomato fruit;

[0061] Construct a YOLOv5s BCD model. The YOLOv5s BCD model consists of four parts: input end, backbone network, neck network and head network. A bidirectional pyramid network is added to the neck network, a coordinate attention mechanism is introduced into the head network of the YOLOv5s model, and a decoupled head design is adopted in the head network of the YOLOv5s model.

[0062] The labeled sample images are used as inputs to the improved YOLOv5s BCD model. The positions of individual tomatoes and the proportion of the red surface of the fruits in the images are used as labels to train the YOLOv5s BCD model.

[0063] The tomato image to be identified is input into the trained YOLOv5s BCD model, which outputs the tomato location information and the proportion of the red surface of the fruit. The tomato maturity level is determined based on the appearance characteristics of the fruit, and tomatoes of the required maturity level are collected.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] First, a bidirectional pyramid network (BiFPN) is introduced into the neck network, which has significant advantages in multi-scale feature fusion. Through bidirectional feature flow and weighted fusion strategy, BiFPN can effectively aggregate feature maps of different scales and improve the accuracy of the model in multi-scale object detection. This improvement enables the model to more accurately extract features and locate targets when facing tomatoes of different sizes and maturity levels, solving the shortcomings of traditional models in multi-scale object detection.

[0066] Secondly, the coordinate attention mechanism is introduced in the head network. This mechanism can enhance the model's feature selection ability in complex backgrounds by generating attention weights. The coordinate attention mechanism enables the model to focus more on the target area by capturing information in important areas of the image and suppressing irrelevant background noise. This mechanism not only improves the accuracy of target detection, but also enhances the robustness of the model, especially in natural scenes with changing lighting and complex backgrounds.

[0067] In addition, the decoupling head design is adopted to optimize the learning ability of the model through the separation calculation of classification and regression. The decoupling head design allows the model to handle classification tasks and positioning tasks more flexibly, and perform feature processing and information integration at different granularities. This not only improves the detection accuracy of the model, but also enables the model to converge more effectively during the training process. At the same time, according to the visual features of the red surface ratio of the fruit, the roundness of the fruit, and the roughness of the fruit texture, the recognition accuracy and accuracy are improved, and the lightweight and real-time performance of edge-side deployment are adapted. It can be used as the visual system of the tomato harvesting robot to provide technical support for automated picking. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a schematic diagram of the overall system structure of the present invention;

[0069] Figure 2 It is a schematic diagram of the overall method flow of the present invention;

[0070] Figure 3 It is a schematic diagram of the structure of the weighted bidirectional feature pyramid model of the present invention. DETAILED DESCRIPTION

[0071] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.

[0072] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0073] Example:

[0074] See also Figure 1-3 , the present invention provides a technical solution:

[0075] A tomato maturity detection system based on deep learning, specifically comprising:

[0076] The sample collection module is used to collect a number of sample images of natural scenes of a tomato garden under different weather conditions and at different time periods during the day, and mark the position of each single tomato fruit and the proportion of the fruit with red surface in each sample image, wherein each sample image contains at least one tomato, and the proportion of the fruit with red surface is the proportion of the area of ​​the fruit that changes from cyan to red to the entire area of ​​the single tomato fruit;

[0077] The data collection location is a modern fruit and vegetable industry planting base. All samples were collected in greenhouses under natural scenes from September to October 2023, and were taken with mobile phones in different weather conditions and at different times of the day. Blurred and redundant images were manually removed, and the remaining images were used as sample images to form a sample image set, and finally 3,890 original images were obtained. The typical characteristics of the sample images in the natural scene of the tomato garden are complex backgrounds, overlapping clusters of eggplants, and occlusion by branches and leaves. In the eggplant orchard with pruned branches and leaves, there are many small targets in the distance.

[0078] Labelme annotation software was used for image annotation. The annotation rules are as follows: ① Annotate the position and the red surface ratio of each single fruit; ② For fruits that are blocked by stems, leaves or other fruits and whose maturity cannot be identified, no annotation is given; ③ For fruits that are blocked by stems, leaves or other fruits and whose occlusion area exceeds two-thirds, no annotation is given; ④ For fruits that are far away and whose maturity cannot be identified, no annotation is given. For automatic picking robots, the closest target is always the focus. The tomato maturity detection dataset includes 3890 annotated images, and a total of 19453 tomatoes with different maturity levels are annotated. Due to the small amount of data, in order to increase the generalization ability of the model, data augmentation is performed in the data preprocessing stage. Five data augmentation methods are used: left-right flipping, random translation, random scaling, adding appropriate noise, and changing brightness. Through pre-data enhancement, the data volume is expanded five times. If there is no single tomato fruit that meets the annotation rules in a sample image, the image is removed from the sample image set.

[0079] The tomato dataset is mainly collected from natural scene images in vegetable greenhouses. Its data characteristics are as follows: ① The maturity of fruits and vegetables mainly depends on the phenotypic characteristics of the fruit surface, and cannot be classified according to the flesh color, seed morphology and odor characteristics; ② The background under natural conditions is complex, tomato fruits are easy to cluster, and branches and leaves are blocked and fruits are superimposed. ③ For long-distance tomatoes whose maturity level cannot be determined, the robot only focuses on near-field targets and ignores the labeling and detection of small long-distance targets; ④ The difference between adjacent maturity levels is small, resulting in classification errors; ⑤ Abnormal fruits such as deformed fruits, hollow fruits, cracked fruits, overripe and damaged fruits interfere with maturity detection. Tomato maturity detection is mainly based on phenotypic characteristics such as fruit surface gloss and skin color to locate tomatoes and classify maturity, and ultimately serve robot automated harvesting. The tomato visual detection model needs to be designed according to the application scenario and data characteristics.

[0080] The logic for obtaining the red surface ratio of the fruit is: converting the sample image from the RGB color space to the HSV color space, specifically: calculating the image feature data according to the intensity values ​​of the red, green and blue channels of each pixel. The specific calculation formula is as follows:

[0081]

[0082]

[0083] In the formula, atan2 is the inverse tangent function, which is used to calculate the inverse tangent value of two variables, H p , S p and V p are the hue, saturation and brightness of the pth pixel in the visible light image, R p Represents the intensity value of the red channel of the pth pixel, G p Indicates the intensity value of the green channel of the pth pixel, B p Represents the intensity value of the blue channel of the p-th pixel.

[0084] After converting to the HSV color space, set the red color threshold range, that is:

[0085] H min ≤H p ≤H max

[0086] S min ≤S p ≤S max

[0087] V min ≤V p ≤V max

[0088] In the formula, H min and H max are the minimum and maximum values ​​of the red color hue threshold range, S min and S max are the minimum and maximum values ​​of the red color saturation threshold range, V min and V max are the minimum and maximum values ​​of the red color brightness threshold range respectively; the pixels in the image that meet the set red color threshold range are recorded, and the area formed by all recorded pixels is calculated. The specific formula is:

[0089] A rad =N rad *A PRIX

[0090] In the formula, A rad is the red fruit area, N rad To record the total number of pixels, A PPIX is the actual area represented by each pixel;

[0091] The red surface ratio of the fruit is calculated based on the red surface area of ​​the fruit. The specific formula is:

[0092]

[0093] In the formula, R rad is the red surface ratio of the fruit, A total is the total area of ​​a single tomato fruit in the image plane.

[0094] For the HSV color space, the hue of red is usually between 0° to 15° and 345° to 360°. The specific red color threshold range of the image can be set according to expert experience. The total area of ​​a single tomato in a plane can be obtained by edge detection algorithm, extracting the edge contour and calculating the specific area.

[0095] The model building module is used to build the YOLOv5s BCD model. The YOLOv5s BCD model consists of four parts: input end, backbone network, neck network and head network. A bidirectional pyramid network is added to the neck network, a coordinate attention mechanism is introduced into the head network of the YOLOv5s model, and a decoupled head design is adopted in the head network of the YOLOv5s model. Specifically,

[0096] The traditional YOLOv5s model is used as the base model. The traditional YOLOv5s model includes four parts: input end, backbone network, neck network and head network. The YOLOv5s model is improved by adding a bidirectional pyramid network to the neck network, and multi-scale feature fusion is performed on the input image to obtain a multi-scale fusion feature map.

[0097] In natural scenes, distant objects guide the robot to walk, and near objects guide picking. The visual model needs to fuse multi-scale features to improve the accuracy of object detection. The neck network of the model uses a weighted bidirectional feature pyramid network (BiFPN), which realizes multi-scale feature fusion through top-down and bottom-up bidirectional paths, and reuses the bidirectional path for cross-layer weighted feature fusion, combining the high resolution of the underlying features with the rich semantic information of the high-level features, improving the stability and accuracy of object detection.

[0098] Since automatic tomato picking focuses on the near-field targets and ignores the far-field targets whose maturity cannot be identified, the neck network only uses three feature layers, namely P3, P4 and P5 layers of the Backbone network, to achieve cross-layer connection and weighted feature fusion of the three-layer features. By removing nodes with only one input edge to simplify the network, an edge is added between feature layers at different layers but with the same resolution to achieve cross-layer connection and finally form a BiFPN layer, and the BiFPN layer is stacked three times. The BiFPN is a weighted bidirectional feature pyramid network. BiFPN adopts weighted feature fusion to obtain weights of different layers through learning. In order to limit the weights without affecting the training speed, the formula for limiting the weights of different layers is as follows:

[0099]

[0100] Where O is the limiting threshold of the weights of different layers, ω i ∈(ω1,ω2,ω3),ω j ∈(ω1,ω2,ω3), where ω1 is the weight of the P4 feature layer in the Backbone network, ω2 is the weight of the P5 feature layer in the Backbone network, and ω3 is the weight of the P3 feature layer in the Backbone network. i ∈(I1,I2,I3), I1 is the learnable weight of the P4 feature layer in the Backbone network, I2 is the learnable weight of the P5 feature layer in the Backbone network, I3 is the learnable weight of the P3 feature layer in the Backbone network, ε is the initial learning rate, where i and j are the indexes of the feature layers in the Backbone network, and i∈(1,2,3), and j∈(1,2,3).

[0101] In order to aggregate features of different levels and resolutions, feature fusion is performed in the BiFPN layer, where the formula for feature fusion is:

[0102]

[0103] In the formula, represents the output of node m4 in the BiFPN layer, is the output of node o4 in the BiFPN layer, Conv represents the convolution operation, ω1 is the weight of the P4 feature layer in the Backbone network, ω2 is the weight of the P5 feature layer in the Backbone network, is the output of node i5 in the BiFPN layer, resize represents upsampling and downsampling operations, is the output of node o3 in the BiFPN layer, ω′1, ω′2 and ω′3 represent the weights of node i4, node m4 and node o3 on the output of node o4 respectively; after the feature fusion of three stacked BiFPN layers, a multi-scale fusion feature map is output;

[0104] Among them, the nodes i3, i4 and i5 are the feature input nodes of the feature layers P3, P4 and P5 respectively, the node m4 is the feature fusion node of nodes i4 and i5, the node o3 is the output node after the feature input node i3 and the feature fusion node m4 are fused, and the node o4 represents the output node after the feature fusion node m4, the feature input node i4 and the output node o3 are fused. The BiFPN module and the fusion nodes m4 and o4 implement the add operation. For the specific location distribution of the nodes, please refer to Figure 3 .

[0105] The coordinate attention mechanism is introduced into the head network of the YOLOv5s model. The attention weight is generated through the coordinate attention mechanism. Based on the attention weight of each channel, each channel of the multi-scale fusion feature map is element-by-element multiplied to obtain a weighted feature map.

[0106] The tomato maturity detection task includes two subtasks: target positioning and classification. Accurate target positioning can improve classification accuracy. Introducing the coordinate attention mechanism (CA) in the visual task can improve the effect of tomato positioning and maturity detection. CA combines spatial attention and channel attention. Spatial attention focuses on the location information and key feature areas of the target, while channel attention strengthens the feature layer with the most target positioning and classification, which can be understood as a certain spectral feature layer of the visible light band of tomatoes or the color and texture feature layer of tomatoes, etc. Finally, the feature layer with the most positioning and classification effect is weighted and focused. Adding the attention mechanism to the head network and combining it with the decoupled head network design can improve the accuracy of target positioning and recognition.

[0107] The logic of generating attention weights through the coordinate attention mechanism and obtaining weighted feature maps is as follows: the input multi-scale fusion feature map first encodes each channel in the width and height directions through H×1 and 1×W pooling kernels respectively, obtains the global receptive field and encodes the channel and position information. The formula for encoding output along the height and width directions is:

[0108]

[0109] In the formula, is the output after the pooling operation in the width direction, representing the encoding of the Cth channel in the height direction, is the output after the pooling operation in the height direction, representing the encoding of the Cth channel in the width direction. W and H represent the width and height of the multi-scale fusion feature map, respectively. c (H, p) represents the element value at the height H and width p position on the Cth channel of the feature map, X c (W,q) represents the element values ​​at the width E and height q positions on the Cth channel of the feature map, where C is the index of the multi-scale fusion feature map channel, and p and q are the indexes of the width and height of the multi-scale fusion feature map respectively;

[0110] The two encoded outputs are then concatenated and convolved according to the following formula:

[0111]

[0112] In the formula, f represents the tensor obtained by cascading and convolution operations on the encoded output, δ(·) represents the nonlinear Relu operation, F1 is the 1×1 convolution Conv operation, Represents the cascade Concatenate operation along the channel dimension, where the channel dimension is controlled to N / r, N is the number of channel dimensions, and r is the scale factor;

[0113] Then perform a Split operation on the output result to get f H and f W Two tensors, where f H and f W They represent the tensor containing the feature information in the height direction and the tensor containing the feature information in the width direction respectively. H and f W The two tensors are respectively subjected to 1×1 convolution and sigmoid function activation operations, and finally two weighted tensors g with the same latitude as the multi-scale fusion feature map are obtained. H and g W , as the attention weight, each channel of the multi-scale fusion feature map is element-by-element multiplication based on the attention weight of each channel. The final output of the coordinate attention mechanism module is expressed as:

[0114]

[0115] In the formula, y C (p,q) represents the value of the Cth channel at position (p,q) in the final output feature map. and They represent the weight of the Cth channel at the height position q and the weight at the width position p respectively.

[0116] The decoupling head design is adopted in the head network of the YOLOv5s model. The weighted feature map is calculated separately for classification and regression through decoupling heads of different granularities. The tensors of the weighted feature map are concatenated into tensors of fixed dimensions, and finally the output tensors of the concatenation of decoupling heads of different granularities are obtained.

[0117] The detection head of YOLO v5s is a coupled head. The bounding box regression and target classification share feature parameters. However, in the target detection task, the features of the classification and regression tasks are different. The classification task pays more attention to the maturity features of tomatoes, such as texture features and color features, while the bounding box regression task pays more attention to the boundary information and background of the target. If too much attention is paid to the location information of the tomato, the network will increase the response value of the foreground and background and ignore the classification features. If too much attention is paid to the maturity information of the tomato, the network will strengthen the texture and color features related to maturity and ignore the location information. To solve this problem, improve the classification and regression accuracy at the same time and improve the overall performance of the model, the head network is changed to a decoupled head to achieve separate calculations of classification and regression, so as to improve the classification accuracy without affecting the positioning accuracy.

[0118] The decoupled head network design was proposed by the YOLOX network. Through the decoupled calculation of the classification and regression branches, the model performance is improved and the convergence speed of training is increased. The network output of the head decoupled model design includes three parts: Classification, Regression and Objectness. After fusion, the P3-5 layers of the feature map enter the coordinate attention mechanism module of the head respectively, and are output to the head decoupled model in three different granularities. The specific workflow of the head decoupled model is as follows: first, the number of feature channels is reduced to 256 using the CBS1×1 network, and then it is divided into two parallel branches to learn the classification and regression tasks respectively through the CBS3×3 network, and the Objectness branch is added to the regression task. Finally, the Conv1×1 convolution layer is used to achieve dimensional matching of the output parameters. The model splices the output tensor into a tensor of H×W×(N+1+4) dimensions, and finally splices the output tensors of the three decoupled heads with different granularities, where CBS represents the combination of three operations: convolution layer, batch normalization and activation function.

[0119] The model training module is used to take the labeled sample images as the input of the improved YOLOv5s BCD model, and take the position of the tomato fruit and the proportion of the red surface of the fruit in the image as labels to train the YOLOv5s BCD model;

[0120] The YOLOv5s BCD model is a YOLO v5s+BiFPN+CA+Decoupled Head tomato maturity detection model, referred to as the YOLOv5s BCD model;

[0121] The training, verification and testing environment configuration of the model is as follows: the processor model is Intel(R) Core(TM) i7-14700K, 3.6GHz, the graphics card model is NVIDIA GeForce RTX 3080Ti, the experimental software environment uses Ubuntu18.04 operating system, PaddlePaddle v2.0 deep learning framework, CUDA 12.0 version parallel computing framework, CUDNN8.0.5 version of deep neural network acceleration library, the compilation environment is Pycharm, and the programming language is Python3.7.

[0122] Since the amount of data in the self-made dataset is small, in order to reduce overfitting and improve the generalization ability of the model, the model uses the basic pre-trained model provided by the Baidu AI Studio platform. To avoid large updates to the parameters of the pre-trained model due to new sample data, a learning rate warmup learning strategy is used in the first 5 epochs, and the training batch size is defined as batch size = 16, the initial learning rate lr = 0.01, and the learning rate is decayed by cosine annealing. The Adam optimizer is used, and the momentum is set to momentum = 0.937. The Obj branch and the Cls branch use the BCE loss loss function, and the Reg branch uses the IOU loss loss function.

[0123] The tomato maturity detection model needs to compromise between the accuracy and speed of the model. Under the premise of meeting the detection speed requirements, the detection accuracy should be improved as much as possible. In order to comprehensively evaluate the performance of the model, the following parameters are selected as model evaluation indicators: precision (Precision, P), recall (Recall, R), F1 value (F1 Score), mean average precision (mean average precision, mAP), detection speed frame rate (frame per second, FPS), and model parameters. In order to adapt to the deployment requirements of the model in lightweight edge devices, the model performance indicator mAP, the model reasoning real-time indicator FPS, and the parameter quantity and memory usage indicators that reflect the model size are mainly considered.

[0124] Through experiments, the model detection effect, model accuracy and detection speed are comprehensively compared, and finally the improved model of this paper is proposed. The experimental results show that the average accuracy of the model YOLOv5s BCD reaches 95.6%, the model inner layer occupies 25.6MB, and the model detection speed is 28.5f·s-1. The model is suitable for deployment at the edge and meets the real-time requirements of detection.

[0125] The model detection module is used to input the tomato image to be identified into the trained YOLOv5s BCD model, output the tomato position information and the proportion of the red surface of the fruit, judge the tomato maturity level based on the appearance characteristics of the fruit, and collect tomatoes of the required maturity level.

[0126] The national standard GH / T 1193-2021 divides the maturity of tomatoes into six levels, namely, the unripe stage, the fruit is small in size, has not yet fully grown and developed, and the skin is green and dull; the green ripe stage, the fruit surface is shiny, and the skin changes from green to white-green; the color change stage, the transition period from white-green to red, yellow or light red spots begin to appear around the navel, and the fruit is less than 10% red; the early red ripe stage, red 10% to 30% red ripe, the fruit is 10%-30% red; the middle ripe stage, red 40% to 60% red ripe, the fruit is 40%-60% red; the late red ripe stage, red 70% to 100% red ripe, the fruit is 70%-100% red; the maturity level is mainly classified according to the morphological characteristics of tomatoes. When harvesting tomatoes, it is necessary to determine the harvesting time according to the maturity of the tomatoes, combined with the variety, use, storage and transportation conditions. Tomatoes harvested too early may lack flavor and sweetness, resulting in poor taste and quality of the fruit; harvesting too late may cause the tomatoes to be overripe and have poor storage resistance.

[0127] The logic for judging the maturity level of tomatoes based on the red surface ratio of the fruit and the appearance characteristics of the fruit is as follows: extracting the appearance characteristic data of the fruit, wherein the appearance characteristic data of the fruit includes the roundness of the fruit and the roughness of the fruit texture, wherein the formula for calculating the roundness and the roughness is as follows:

[0128]

[0129] Where YD is the roundness of the fruit, ZP is the circumference of the fruit, AAR is the fruit texture roughness, M is the total number of pixels in the single tomato image, and Z β is the gray value of the βth pixel, Z mean is the average gray value of all pixels, and β is the index of the pixel in the single-fruit tomato image;

[0130] The perimeter of the fruit is obtained through edge detection algorithm.

[0131] The maturity coefficient is generated based on the red surface ratio of the fruit, the roundness of the fruit and the roughness of the fruit texture. The formula for calculating the maturity coefficient is:

[0132]

[0133] In the formula, SD is the maturity coefficient, and k1, k2, and k3 are the weight coefficients of the red surface ratio of the fruit, the roundness of the fruit, and the roughness of the fruit texture respectively, where k1 > k2 > k3 and k1, k2, and k3 are all greater than 0; among them, mature tomatoes are usually more plump and have a uniform shape, while immature tomatoes may be flatter or irregular and mature tomatoes usually have a smoother surface. Therefore, the roughness of the fruit texture is inversely proportional to the maturity coefficient, R rad and YD are directly proportional to the maturity coefficient.

[0134] The logic for judging the tomato maturity level based on the generated maturity coefficient is as follows:

[0135] When 0 < SD ≤ 0.1*yz, the tomato maturity level is judged to be the immature stage;

[0136] When 0.1*yz < SD ≤ 0.3*yz, the tomato maturity level is judged to be the green-ripe stage;

[0137] When 0.3*yz < SD ≤ 0.5*yz, the tomato maturity level is judged to be the color-changing stage;

[0138] When 0.5*yz < SD ≤ 0.6*yz, the tomato maturity level is judged to be the early red-ripe stage;

[0139] When 0.6*yz < SD ≤ 0.8*yz, the tomato maturity level is judged to be the mid red-ripe stage;

[0140] When 0.8*yz < SD ≤ 1.0*yz, the tomato maturity level is judged to be the late red-ripe stage.

[0141] Collect tomatoes of the required maturity level according to the location information, so as to achieve maturity grading and differential harvesting according to market demand and storage and transportation plans, which can improve the quality and taste of tomatoes and increase farmers' income.

[0142] Please refer to Figure 2 , the present invention also provides a method for detecting tomato maturity based on deep learning. The method for detecting tomato maturity based on deep learning is used to control the above-mentioned system for detecting tomato maturity based on deep learning. The specific steps include:

[0143] Step 1: Collect a number of sample images of the natural scene in the tomato orchard at different daytime periods under different weather conditions, and mark the position of each single tomato fruit and the red surface ratio of the fruit in each sample image. Each sample image contains at least one tomato. The red surface ratio of the fruit is the ratio of the area where the fruit changes from green to red to the area of the entire single tomato fruit;

[0144] Step 2: Build the YOLOv5s BCD model. The YOLOv5s BCD model consists of four parts: input end, backbone network, neck network and head network. A bidirectional pyramid network is added to the neck network, a coordinate attention mechanism is introduced into the head network of the YOLOv5s model, and a decoupled head design is adopted in the head network of the YOLOv5s model.

[0145] Step 3: Use the labeled sample image as the input of the improved YOLOv5s BCD model, use the position of the tomato fruit and the proportion of the red surface of the fruit in the image as labels, and train the YOLOv5s BCD model;

[0146] Step 4: Input the tomato image to be identified into the trained YOLOv5s BCD model, output the tomato location information and the proportion of the red surface of the fruit, judge the tomato maturity level based on the appearance characteristics of the fruit, and collect tomatoes of the required maturity level.

[0147] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0148] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product. Those skilled in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software methods depends on the specific application and design constraints of the technical solution.

[0149] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0150] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A tomato maturity detection system based on deep learning, characterized in that: Specifically include: The sample image module is used to collect a number of sample images of natural scenes of a tomato garden under different weather conditions and at different time periods during the day, and mark the position of each single tomato fruit and the proportion of the fruit with red surface in each sample image, wherein each sample image contains at least one tomato, and the proportion of the fruit with red surface is the proportion of the area of ​​the fruit that changes from cyan to red to the entire area of ​​the single tomato fruit; Model building module, used to build the YOLOv5s BCD model. The YOLOv5s BCD model consists of four parts: input end, backbone network, neck network and head network. A bidirectional pyramid network is added to the neck network, a coordinate attention mechanism is introduced into the head network of the YOLOv5s model, and a decoupled head design is adopted in the head network of the YOLOv5s model. The model training module is used to take the labeled sample images as the input of the improved YOLOv5s BCD model, and take the position of the tomato fruit and the proportion of the red surface of the fruit in the image as labels to train the YOLOv5s BCD model; The model detection module is used to input the tomato image to be identified into the trained YOLOv5s BCD model, output the tomato position information and the proportion of the red surface of the fruit, judge the tomato maturity level based on the appearance characteristics of the fruit, and collect tomatoes of the required maturity level.

2. The tomato maturity detection system based on deep learning according to claim 1, characterized in that: The collected images are used as sample images to form a sample image set; the position and red area of ​​each tomato fruit in each sample image are marked, wherein the Labelme annotation software is used for image annotation, and the specific annotation rules are as follows: the position of each single fruit is marked and the proportion of the red surface of the fruit is marked; the fruit that is blocked by stems, leaves or other fruits and whose maturity cannot be identified is not marked; the fruit that is blocked by stems, leaves or other fruits and whose blocked area exceeds two-thirds is not marked; the fruit that is too far away to identify its maturity is not marked. If there is no single tomato fruit that meets the annotation rules in a sample image, the image is removed from the sample image set; The logic for obtaining the red surface ratio of the fruit is: converting the sample image from the RGB color space to the HSV color space, specifically: calculating the image feature data according to the intensity values ​​of the red, green and blue channels of each pixel. The specific calculation formula is as follows: In the formula, atan2 is the inverse tangent function, which is used to calculate the inverse tangent value of two variables, H p , S p and V p are the hue, saturation and brightness of the pth pixel in the visible light image, R p Represents the intensity value of the red channel of the pth pixel, G p Indicates the intensity value of the green channel of the pth pixel, B p Represents the intensity value of the blue channel of the p-th pixel. After converting to the HSV color space, set the red color threshold range, that is: H min ≤H p ≤H max S min ≤S p ≤S max In min ≤V p ≤V max In the formula, H min and H max are the minimum and maximum values ​​of the red color hue threshold range, S min and S max are the minimum and maximum values ​​of the red color saturation threshold range, V min and V max are the minimum and maximum values ​​of the red color brightness threshold range respectively; the pixels in the image that meet the set red color threshold range are recorded, and the area formed by all recorded pixels is calculated. The specific formula is: A rad =N rad *A PRIX In the formula, A rad is the red fruit area, N rad To record the total number of pixels, A PRIX is the actual area represented by each pixel; The red surface ratio of the fruit is calculated based on the red surface area of ​​the fruit. The specific formula is: In the formula, R rad is the red surface ratio of the fruit, A total is the total area of ​​a single tomato fruit in the image plane.

3. The tomato maturity detection system based on deep learning according to claim 2, characterized in that: A bidirectional pyramid network is added to the neck network for multi-scale feature fusion. The specific logic is: since automatic tomato picking focuses on the near-field targets and ignores the distant targets whose maturity cannot be identified, the neck network only uses three feature layers, namely P3, P4 and P5 layers of the Backbone network, to achieve cross-layer connection and weighted feature fusion of three-layer features. By removing nodes with only one input edge to simplify the network, an edge is added between feature layers at different layers but with the same resolution to achieve cross-layer connection and finally form a BiFPN layer, and the BiFPN layer is stacked three times. The BiFPN is a weighted bidirectional feature pyramid network. BiFPN adopts a weighted feature fusion method to obtain the weights of different layers through learning. The formula for limiting the weights of different layers is as follows: Where O is the limiting threshold of the weights of different layers, ω i ∈(ω1,ω2,ω3),ω j ∈(ω1,ω2,ω3), where ω1 is the weight of the P4 feature layer in the Backbone network, ω2 is the weight of the P5 feature layer in the Backbone network, and ω3 is the weight of the P3 feature layer in the Backbone network. i ∈(I1,I2,I3), I1 is the learnable weight of the P4 feature layer in the Backbone network, I2 is the learnable weight of the P5 feature layer in the Backbone network, I3 is the learnable weight of the P3 feature layer in the Backbone network, ε is the initial learning rate, where i and j are the indexes of the feature layers in the Backbone network, and i∈(1,2,3), and j∈(1,2,3).

4. The tomato maturity detection system based on deep learning according to claim 3, characterized in that: Feature fusion is performed in the BiFPN layer, where the formula for feature fusion is: In the formula, represents the output of node m4 in the BiFPN layer, is the output of node o4 in the BiFPN layer, Conv represents the convolution operation, ω1 is the weight of the P4 feature layer in the Backbone network, ω2 is the weight of the P5 feature layer in the Backbone network, is the output of node i5 in the BiFPN layer, resize represents upsampling and downsampling operations, is the output of node o3 in the BiFPN layer, ω′1, ω′2 and ω′3 represent the weights of node i4, node m4 and node o3 on the output of node o4 respectively; after the feature fusion of three stacked BiFPN layers, a multi-scale fusion feature map is output; Among them, the nodes i3, i4 and i5 are the feature input nodes of the feature layers P3, P4 and P5 respectively, the node m4 is the feature fusion node of nodes i4 and i5, the node o3 is the output node after the features of the feature input node i3 and the feature fusion node m4 are fused, and the node o4 represents the output node after the features of the feature fusion node m4, the feature input node i4 and the output node o3 are fused.

5. The tomato maturity detection system based on deep learning according to claim 4, characterized in that: The logic of generating attention weights through the coordinate attention mechanism and obtaining weighted feature maps is as follows: the input multi-scale fusion feature map first encodes each channel in the width and height directions through H×1 and 1×W pooling kernels respectively, obtains the global receptive field and encodes the channel and position information. The formula for encoding output along the height and width directions is: In the formula, is the output after the pooling operation in the width direction, representing the encoding of the Cth channel in the height direction, is the output after the pooling operation in the height direction, representing the encoding of the Cth channel in the width direction. W and H represent the width and height of the multi-scale fusion feature map, respectively. c (H, p) represents the element value at the height H and width p position on the Cth channel of the feature map, X c (W,q) represents the element value at the width W and height q position on the Cth channel of the feature map, where C is the index of the multi-scale fusion feature map channel, and p and q are the indexes of the width and height of the multi-scale fusion feature map respectively; The two encoded outputs are then concatenated and convolved according to the following formula: In the formula, f represents the tensor obtained by cascading and convolution operations on the encoded output, δ(·) represents the nonlinear Relu operation, F1 is the 1×1 convolution Conv operation, Represents the cascade Concatenate operation along the channel dimension, where the channel dimension is controlled to N / r, N is the number of channel dimensions, and r is the scale factor; Then perform a Split operation on the output result to get f H and f W Two tensors, where f H and f W They represent the tensor containing the feature information in the height direction and the tensor containing the feature information in the width direction respectively. H and f W The two tensors are respectively subjected to 1×1 convolution and sigmoid function activation operations, and finally two weighted tensors g with the same latitude as the multi-scale fusion feature map are obtained. H and g W , as the attention weight, each channel of the multi-scale fusion feature map is element-wise multiplied based on the attention weight of each channel. The final output of the coordinate attention mechanism module is expressed as: In the formula, y C (p,q) represents the value of the Cth channel at position (p,q) in the final output feature map. and They represent the weight of the Cth channel at the height position q and the weight at the width position p respectively.

6. The tomato maturity detection system based on deep learning according to claim 1, characterized in that: A decoupled head design is adopted, in which the network output of the decoupled model design of the head includes three parts: Classification, Regression and Objectness. After fusion, the P3-5 layers of the feature map enter the coordinate attention mechanism module of the head respectively, and are output to the head decoupled model in three different granularities. The specific workflow of the head decoupled model is as follows: first, the number of feature channels is reduced to 256 using the CBS1×1 network, and then it is divided into two parallel branches to learn the classification and regression tasks respectively through the CBS3×3 network, and the Objectness branch is added to the regression task. Finally, the Conv1×1 convolution layer is used to achieve dimensional matching of the output parameters. The model splices the output tensor into a tensor of H×W×(N+1+4) dimensions, and finally splices the output tensors of the three decoupled heads of different granularities, where CBS represents the combination of three operations: convolutional layer, batch normalization and activation function.

7. The tomato maturity detection system based on deep learning according to claim 2, characterized in that: The logic for judging the maturity level of tomatoes based on the red surface ratio of the fruit and the appearance characteristics of the fruit is as follows: extracting the appearance characteristic data of the fruit, wherein the appearance characteristic data of the fruit includes the roundness of the fruit and the roughness of the fruit texture, wherein the formula for calculating the roundness and the roughness is as follows: Where YD is the roundness of the fruit, ZP is the circumference of the fruit, AAR is the fruit texture roughness, M is the total number of pixels in the single tomato image, and Z β is the gray value of the βth pixel, Z mean is the average gray value of all pixels, and β is the index of the pixel in the single-fruit tomato image; Generate a maturity coefficient based on the proportion of the red surface of the fruit, the roundness of the fruit, and the roughness of the fruit texture. The formula for calculating the maturity coefficient is as follows: In the formula, SD is the maturity coefficient, and k1, k2, and k3 are the weight coefficients of the proportion of the red surface of the fruit, the roundness of the fruit, and the roughness of the fruit texture, respectively, where k1 > k2 > k3 and k1, k2, and k3 are all greater than 0; The logic for judging the tomato maturity level based on the generated maturity coefficient is as follows: When 0 < SD ≤ 0.1 * yz, judge that the tomato maturity level is the immature stage; When 0.1 * yz < SD ≤ 0.3 * yz, judge that the tomato maturity level is the green mature stage; When 0.3 * yz < SD ≤ 0.5 * yz, judge that the tomato maturity level is the color-changing stage; When 0.5 * yz < SD ≤ 0.6 * yz, judge that the tomato maturity level is the early red-ripe stage; When 0.6 * yz < SD ≤ 0.8 * yz, judge that the tomato maturity level is the mid red-ripe stage; When 0.8 * yz < SD ≤ 1.0 * yz, judge that the tomato maturity level is the late red-ripe stage.

8. A tomato maturity detection method based on deep learning, characterized in that: The described method for detecting tomato maturity based on deep learning is used to control the system for detecting tomato maturity based on deep learning according to any one of claims 1-7. The specific steps include: Collect several sample images of the natural scene in the tomato orchard at different times of the day under different weather conditions, and mark the position of each single tomato fruit and the proportion of the red surface of the fruit in each sample image. Each sample image contains at least one tomato, and the proportion of the red surface of the fruit is the proportion of the area where the fruit changes from green to red in the area of the entire single tomato fruit; Construct a YOLOv5s BCD model. The YOLOv5s BCD model includes four parts: an input end, a backbone network, a neck network, and a head network. A bidirectional pyramid network is added to the neck network, a coordinate attention mechanism is introduced into the head network of the YOLOv5s model, and a decoupled head design is adopted in the head network of the YOLOv5s model; Use the sample images with marks as the input of the improved YOLOv5s BCD model, and use the position of the single tomato fruit in the image and the identification of the proportion of the red surface of the fruit as labels to train the YOLOv5s BCD model; Input the tomato image to be recognized into the trained YOLOv5s BCD model, output the tomato position information and the proportion of the red surface of the fruit, judge the tomato maturity level in combination with the appearance characteristics of the fruit, and collect tomatoes with the required maturity level.

Images