Cherry tomato maturity detection method and system applied to machine picking

Abstract

The invention is suitable for the technical field of smart agriculture, and provides a cherry tomato maturity detection method and system applied to machine picking, and the method comprises the steps: firstly obtaining to-be-detected actual image information, and then inputting the to-be-detected actual image information to a trained target detection model, and maturity detection result information is efficiently and accurately generated. According to the method, the target detection model with high precision and high speed can be provided, the reliability is remarkably improved, the method is particularly suitable for accurate positioning and maturity grading work of the target in the mechanical automatic picking process of the cherry tomatoes, valuable reference is provided for the field of agricultural automation, and the method has wide application prospects.

Description

Technical Field

[0001] The present application relates to the technical field of smart agriculture, and more specifically, to a method and system for detecting the maturity of cherry tomatoes used in machine picking. Background Art

[0002] Cherry tomatoes are rich in lycopene, organic acids, vitamins and other nutrients, and are very popular among consumers in daily life. Because manual picking is labor-intensive and costly, machine picking is more efficient and economical than manual picking.

[0003] At present, due to the complex growth characteristics of cherry tomatoes, the color characteristics of cherry tomatoes are easily affected by environmental factors, which makes it difficult to accurately judge their degree of maturity, resulting in less than ideal picking effects and low reliability, which needs further improvement. Summary of the invention

[0004] Based on this, the embodiment of the present application provides a method and system for detecting the maturity of cherry tomatoes for machine picking to solve the problem of low reliability in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a method for detecting the maturity of cherry tomatoes for machine picking, the method comprising:

[0006] Obtaining actual image information to be detected;

[0007] The actual image information to be detected is input into a trained target detection model to generate maturity detection result information.

[0008] Compared with the prior art, the beneficial effects are: in the cherry tomato maturity detection method for machine picking provided in the embodiment of the present application, the terminal device can first obtain the actual image information to be detected, and then input the actual image information to be detected into the trained target detection model, and accurately generate maturity detection result information, thereby accurately judging the maturity of cherry tomatoes, meeting the needs of cherry tomato picking scenarios with high real-time requirements, greatly improving the picking effect and effectively improving the reliability, and to a certain extent solving the current problem of low reliability.

[0009] In a second aspect, an embodiment of the present application provides a cherry tomato maturity detection system for machine picking, the system comprising:

[0010] The actual image information acquisition module to be detected is used to obtain the actual image information to be detected;

[0011] Maturity detection result information generation module: used to input the actual image information to be detected into the trained target detection model to generate maturity detection result information.

[0012] In a third aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of the first aspect described above when executing the computer program.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method of the first aspect described above are implemented.

[0014] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0016] Figure 1 It is a flow chart of a method for detecting the maturity of cherry tomatoes provided in one embodiment of the present application;

[0017] Figure 2 It is a schematic diagram of the process before step S201 in the cherry tomato maturity detection method provided in one embodiment of the present application;

[0018] Figure 3 It is a flow chart after step S104 in the method for detecting the maturity of cherry tomatoes provided in one embodiment of the present application;

[0019] Figure 4 : is a flow chart of step S201 in the method for detecting the maturity of cherry tomatoes provided in one embodiment of the present application;

[0020] Figure 5 Schematic diagram of cherry tomatoes in the method for detecting the maturity of cherry tomatoes provided in one embodiment of the present application;

[0021] Figure 6 Schematic diagram of cherry tomato maturity recognition results in a cherry tomato maturity detection method provided in an embodiment of the present application;

[0022] Figure 7 This is a module block diagram of a cherry tomato maturity detection system provided in an embodiment of the present application;

[0023] Figure 8 It is a schematic diagram of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0024] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0025] In the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0026] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0027] In order to illustrate the technical solution described in this application, a specific embodiment is provided below for illustration.

[0028] See also Figure 1 , Figure 1 It is a schematic flow diagram of a cherry tomato maturity detection method for machine picking provided by an embodiment of the present application. In the present embodiment, the execution subject of the cherry tomato maturity detection method is a terminal device. It is understandable that the types of terminal devices include but are not limited to mobile phones, tablet computers, notebook computers, ultra-mobile personal computers (Ultra-Mobile Personal Computer, UMPC), netbooks, personal digital assistants (Personal Digital Assistant, PDA), etc., and the present embodiment of the application does not impose any restrictions on the specific types of terminal devices.

[0029] See also Figure 1 The method for detecting the maturity of cherry tomatoes provided in the present application example includes but is not limited to the following steps:

[0030] In S100, actual image information to be detected is obtained.

[0031] Specifically, the terminal device may first obtain actual image information to be detected, wherein the actual image information to be detected is used to describe an actual image obtained by photographing cherry tomatoes.

[0032] In S200, the actual image information to be detected is input into the trained object detection model to generate maturity detection result information.

[0033] Specifically, after the terminal device obtains the actual image information to be detected, the terminal device can input the actual image information to be detected into the trained target detection model to accurately generate maturity detection result information, which is conducive to the subsequent set of accurate evaluations of the maturity of cherry tomatoes and determines whether to mechanically pick the cherry tomatoes, effectively improving the picking efficiency and picking quality, and greatly improving reliability. Among them, the target detection model is used to detect the maturity of cherry tomatoes; the maturity detection result information is used to describe the detection results of the maturity of cherry tomatoes.

[0034] In some possible implementations, in order to improve reliability, before step S200, the method further includes but is not limited to the following steps:

[0035] In S201, a target detection model is constructed.

[0036] Specifically, before the terminal device generates maturity detection result information, the terminal device can first build a target detection model

[0037] In some possible implementations, to further help improve reliability, see Figure 2 Before step S201, the method further includes but is not limited to the following steps:

[0038] In S101, the original image set information of the growth posture of cherry tomatoes is obtained.

[0039] Specifically, before the terminal device builds a target detection model, the terminal device can first use a preset image acquisition device to photograph cherry tomatoes in a natural environment to obtain the original image set information of the growth posture of the cherry tomatoes, wherein the image acquisition device can be a camera built into the mobile phone, the shooting distance can be ten centimeters to sixty centimeters, and the shooting scene can be a scene under natural lighting conditions during the day; the original image set information of the growth posture of cherry tomatoes includes multiple original image information.

[0040] In a possible implementation, in order to reduce the overfitting of the network model due to insufficient diversity of training samples, the shooting angles are the left side, the right side and the front side, and the shooting lighting conditions can include two light intensities: front light and back light. The shooting occlusion conditions can include various actual cherry tomato growth postures such as overlap and adhesion. The shot image can be in the ".jpeg" file format, and the shooting resolution can be 4000×3000 pixels.

[0041] In S102, data preprocessing is performed on the original image set information of the growth posture of cherry tomatoes to generate first optimized image set information of the growth posture of cherry tomatoes.

[0042] Specifically, after the terminal device obtains the original image set information of the growth posture of cherry tomatoes, the terminal device can perform data preprocessing on the original image set information of the growth posture of cherry tomatoes, that is, perform data cleaning and screening processing to generate the first cherry tomato growth posture optimized image set information, thereby excluding low-quality images and retaining images that meet the actual application requirements, wherein the first cherry tomato growth posture optimized image set information includes multiple first optimized image information.

[0043] Exemplarily, the terminal device can eliminate low-quality images in the original image set information of the growth posture of cherry tomatoes. The low-quality images may be images that are too blurred, severely overexposed, heavily occluded and / or irrelevant, so as to retain images with a clear background that meet the requirements, thereby generating the first cherry tomato growth posture optimized image set information.

[0044] In S103, data compression processing is performed on the first cherry tomato growth posture optimized image set information to generate a second cherry tomato growth posture optimized image set information.

[0045] Specifically, after the terminal device generates the first cherry tomato growth posture optimized image set information, the terminal device can perform data compression processing on the first cherry tomato growth posture optimized image set information to generate the second cherry tomato growth posture optimized image set information, thereby realizing data compression of the image and reducing the subsequent calculation amount, wherein the second cherry tomato growth posture optimized image set information includes multiple second optimized image information, and the image resolution of the second optimized image information is smaller than the image resolution of the first optimized image information.

[0046] Exemplarily, the terminal device may perform the processing for each piece of first optimized image information: using the resize function in the OpenCV software to compress the resolution of the first optimized image information to 1200×900 pixels, and generate second optimized image information.

[0047] In some possible implementations, in order to further improve reliability, after step S103, the method further includes but is not limited to the following steps:

[0048] In S104, the second cherry tomato growth posture optimized image set information is subjected to maturity type labeling processing to generate ripe fruit type information, semi-ripe fruit type information and immature fruit type information.

[0049] Specifically, after the terminal device generates the second cherry tomato growth posture optimized image set information, the terminal device can use the preset data labeling tool to perform maturity type labeling processing on the second cherry tomato growth posture optimized image set information, and generate ripe fruit type information, semi-ripe fruit type information and immature fruit type information, thereby realizing the classification and labeling of cherry tomatoes of different maturity levels, and accurately determining the data labels corresponding to cherry tomatoes of various maturity levels.

[0050] Exemplarily, the terminal device can use a labeling tool specifically named LabelImg to manually label the cherry tomatoes in the second cherry tomato growth posture optimization image set information according to the labeling format of the VOC data set, and label the rectangular frame of the cherry tomatoes; wherein the labeling rule can be: not only the completely exposed cherry tomatoes cut inside the rectangular frame are labeled, but also the part of the cherry tomatoes exposed to the image that is blocked or adhered is labeled with a rectangular frame, and the part that appears at the image boundary and the display degree of the cherry tomatoes due to occlusion is less than 10% are all unlabeled; it should be noted that the categories of labeled maturity can include ripe fruit type information, semi-ripe fruit type information and immature fruit type information, and the ripe fruit type information can be named ripe, the semi-ripe fruit type information can be named turning, and the immature fruit can be named green. In a possible implementation, the labeling of cherry tomatoes can be done manually.

[0051] In some possible implementations, in order to improve the detection accuracy of cherry tomatoes ripeness, please refer to Figure 3 After step S104, the method further includes but is not limited to the following steps:

[0052] In S105, based on a preset filtering algorithm, noise removal is performed on the second cherry tomato growth posture optimized image set information to generate a third cherry tomato growth posture optimized image set information.

[0053] Without loss of generality, the terminal device may first use an adaptive histogram equalization method on the second cherry tomato growth posture optimization image set information to reduce the impact of light on the model recognition accuracy.

[0054] Specifically, after the terminal device generates ripe fruit type information, semi-ripe fruit type information and immature fruit type information, the terminal device can remove noise from the second cherry tomato growth posture optimized image set information based on a preset filtering algorithm to generate a third cherry tomato growth posture optimized image set information, thereby fully considering the presence of noise in the image, reducing the image quality degradation caused by enhanced noise caused by histogram equalization, and further improving the image quality and the recognition accuracy of the model. Exemplarily, the terminal device can simultaneously combine the mean filtering algorithm and the Gaussian filtering algorithm to remove noise from the second cherry tomato growth posture optimized image set information.

[0055] In S106, data enhancement processing is performed on the third cherry tomato growth posture optimized image set information to generate a fourth cherry tomato growth posture optimized image set information.

[0056] Specifically, after the terminal device generates the third cherry tomato growth posture optimized image set information, the terminal device can perform data enhancement processing on the third cherry tomato growth posture optimized image set information to generate the fourth cherry tomato growth posture optimized image set information, thereby fully considering the impact of different cherry tomato growth postures on model recognition and improving the robustness of the model, wherein the data enhancement processing can be random rotation of the image, color enhancement, brightness change and / or adding Gaussian noise.

[0057] In order to build an effective and accurate target detection model, see some possible implementations. Figure 4 Step S201 includes but is not limited to the following steps:

[0058] In S2011, an untrained object detection model is obtained.

[0059] Specifically, the terminal device may first obtain an untrained target detection model, wherein the target detection model may be a YOLO v5 model.

[0060] In S2012, the preset CA attention mechanism module and loss function are added to the untrained target detection model to generate a first optimized detection model.

[0061] Specifically, the attention mechanism is used to guide the target detection model to focus on cherry tomatoes and their locations; after the terminal device obtains the untrained target detection model, the terminal device can add the preset CA attention mechanism module and loss function to the untrained target detection model to generate a first optimized detection model. By introducing the attention mechanism, the target detection model can adaptively learn the importance of each channel, enhance attention to specific channels according to task requirements, filter out irrelevant information, and effectively improve the representation ability and performance of the model.

[0062] Without loss of generality, the CA attention mechanism module not only considers the attention in the channel dimension and the spatial dimension at the same time, but also can make the model pay more attention to useful channel information by learning adaptive channel weights, which is conducive to more accurate positioning and identification of the position of cherry tomatoes.

[0063] In one possible implementation, the terminal device can first give an input X and use a pooling kernel of size (H, 1) or (1, W) to encode each channel along the horizontal coordinate and vertical coordinate respectively. Therefore, the output of the cth channel with height h can be expressed as:

[0064]

[0065] The output of the cth channel with width w can be expressed as:

[0066]

[0067] The above two transformations aggregate features along two spatial directions respectively, and then encode the obtained feature maps into a pair of direction-aware and position-sensitive attention maps respectively, so as to capture long-range dependencies in one spatial direction while maintaining precise position information in another spatial direction. The above two transformations can be applied complementarily to the input feature map to enhance the representation of objects of interest and more accurately locate and identify target areas.

[0068] Furthermore, by embedding coordinate information, the terminal device can better obtain the global receptive field and encode accurate location information. For example, the terminal device can concatenate the above transformation and then transform it using a 1×1 convolution transformation function F1, that is:

[0069] hw

[0070] f=δ(F1([z,z])),

[0071] Where δ is a nonlinear activation function, [,] is the concatenate operation along the spatial dimension, and f is an intermediate feature map that encodes spatial information in the horizontal and vertical directions.

[0072] The terminal device can then slice f along the spatial dimension and use another two 1×1 convolution transformation functions F h and F w f h and f w Transformed into a tensor with the same channel number as the input X, and then processed using the sigmoid activation function to obtain the attention vector g h and g w, where the attention vector g h The expression can be:

[0073] hh

[0074] g=σ(F h (f)),

[0075] Attention vector g w The expression can be:

[0076] ww

[0077] g=σ(F w (f)),

[0078] Finally, the terminal device can w and g h To expand, the output formula of the CA attention mechanism module can be:

[0079]

[0080] Without loss of generality, the loss function is used to measure the gap between the predicted information and the real information; if the predicted information is closer to the real information, the loss function value is smaller. The loss function of the YOLO v5 model can be composed of three parts, namely positioning loss, classification loss and confidence loss. In order to obtain accurate information on the location and maturity of cherry tomatoes in a natural environment, the terminal device can optimize the loss function to balance the training errors of the prediction box, confidence and classification.

[0081] It should be noted that the loss function can adopt a loss function specifically named Wise loU (WIoU), wherein WIoU has three versions. WIoU v1 constructs an attention-based bounding box loss, and WIoU v2 and WIoU v3 construct a gradient gain calculation method on this basis to attach a focusing mechanism. The cherry tomato maturity detection method can adopt a specific WIoU v3 loss function. Compared with the traditional loss function, the WIoU v3 loss function can evaluate the quality of the anchor frame by using a dynamic non-monotonic focusing mechanism and outliers, thereby solving the bounding box regression balance problem between high-quality and low-quality data, thereby reducing the competitiveness of high-quality data while also reducing the harmful gradients generated by low-quality data. Moreover, through the dynamic non-monotonic focusing mechanism of the WIoU loss function, it is possible to allocate a smaller gradient gain to the low-quality anchor frame, so that the BBR is more focused on processing the anchor frame of ordinary quality, weakening the influence of low-quality data on the BBR, thereby improving the overall performance of the detector.

[0082] Without loss of generality, the terminal device can first define the outlier degree β based on WIoU v3 to describe the quality of the anchor frame, which can be defined as:

[0083]

[0084] The non-monotonic focusing coefficient formula constructed by outlier degree β can be:

[0085]

[0086] Where W g Indicates the width of the minimum bounding box, H g Indicates the height of the minimum bounding box, R WIoU ∈[1,e),L IoU ∈[0,1].

[0087] Among them, the smaller the outlier degree β, the higher the quality of the anchor frame, and the larger the outlier degree β, the lower the quality of the anchor frame; the terminal device can assign a small gradient gain to the anchor frame with a smaller outlier degree, thereby focusing the bounding box regression on the anchor frame of normal quality, and assign a small gradient gain to the anchor frame with a larger outlier degree, thereby effectively preventing the low-quality anchor frame from generating large harmful gradients.

[0088] Specifically, when β = δ, δ can make r = 1; when β = C (C is a constant), the anchor box can obtain the highest gradient gain; It is dynamic, so the quality division standard of the anchor box can also be dynamic, so that the WIoU v3 loss function can make the gradient gain allocation strategy that best suits the current situation at any time.

[0089] Furthermore, in order to prevent low-quality anchor frames from falling behind in the early stages of training, the terminal device can initialize is 1, so that L IoU = 1 has the highest gradient gain. At the same time, in order to maintain this strategy in the early stage of training, the terminal device can pre-set a momentum m to delay near time, where the momentum m can be expressed as:

[0090]

[0091] It should be noted that this setting can make it possible to In the middle and late stages of training, the WIoU v3 loss function can assign small gradient gains to low-quality anchor boxes to reduce harmful gradients. At the same time, the WIoU v3 loss function can focus on anchor boxes of ordinary quality, thereby improving the positioning performance of the model.

[0092] In S2013, based on a preset division ratio, the fourth cherry tomato growth posture optimization image set information is divided and processed to generate training set information, verification set information and test set information.

[0093] Specifically, after the terminal device generates the first optimized detection model, the terminal device can divide and process the fourth cherry tomato growth posture optimized image set information based on a preset division ratio to generate training set information, verification set information and test set information.

[0094] Exemplarily, the terminal device can randomly divide the data set after data amplification (i.e., the fourth cherry tomato growth posture optimized image set information) into training set information, validation set information and test set information in a ratio of 7:1:2, so as to facilitate the subsequent evaluation of the performance and generalization effect of the model, wherein the training set information is used for model training, the validation set information is used for model verification, and the test set information is used for model testing.

[0095] In S2014, based on a preset clustering algorithm, the anchor frames of the training set information are clustered to generate cluster center position information.

[0096] Specifically, the terminal device may perform clustering processing on the anchor frames of the training set information based on a preset clustering algorithm to generate cluster center position information, wherein the clustering algorithm may be a K-means++ clustering algorithm.

[0097] It should be noted that the predefined anchor box size can be obtained by clustering the coco dataset and the voc dataset. Both the coco dataset and the voc dataset can be training set information. Among them, the coco dataset can have 80 types of targets, and the voc dataset can have 20 types of targets. The sizes and category sizes of these targets are different. The clustered anchor boxes may not be completely suitable for the current data. Therefore, the terminal device can add an adaptive anchor box function to the YOLOv5 model, so that the target detection model can use the K-means clustering algorithm to automatically learn and adjust the anchor box suitable for the current dataset; during the training process, the target detection model can automatically learn and update the scale and aspect ratio of the anchor box to better adapt to the target features in the current dataset, thereby improving the detection performance and generalization ability of the model.

[0098] Without loss of generality, since the first step of the K-means clustering algorithm is to randomly select k cluster centers, the distribution of cluster centers may be uneven, which may cause the algorithm to fall into a local optimal solution and fail to converge to the global minimum, resulting in errors in the clustering results. The K-means++ clustering algorithm tends to select data points that are farther away when selecting the initial cluster center points, so as to better capture the overall structure of the data. Therefore, in order to avoid the disadvantages of initializing the cluster center, the clustering algorithm of the cherry tomato maturity detection method can be the K-means++ clustering algorithm.

[0099] For example, the terminal device may first randomly select a data point from the data set x as the first cluster center C1, then calculate the shortest distance d(x) between each data point and the selected cluster center C1, and select the next cluster center C2 according to the roulette method, so that its probability p(x) is proportional to the square of the distance, and the data point corresponding to the maximum value of p(x) is the next cluster center C3, where the probability p(x) can be expressed as:

[0100]

[0101] Then, the terminal device can repeat the above steps of "calculating the shortest distance d(x) between each data point and the selected cluster center C1, selecting the next cluster center C2 according to the roulette method, so that its probability p(x) is proportional to the square of the distance, and the data point corresponding to the maximum value of p(x) is the next cluster center C3" until k cluster centers are selected, and then assigning the data points to the nearest cluster center, and then updating the position of the cluster center according to the assigned data points, and finally repeating the above steps until the cluster center no longer changes or reaches the predetermined number of iterations. Through continuous iterative experiments, the terminal device can finally select 9 anchor boxes [[10,13,16,30,33,23],[30,61,62,45,59,119],[116,90,156,198,373,326]].

[0102] In S2015, based on the training set information and the cluster center position information, the first optimized detection model is trained until the performance evaluation values ​​corresponding to the validation set information and the test set information meet the requirements, thereby generating a trained target detection model.

[0103] Specifically, after the terminal device implements clustering of anchor frames using the improved K-means++ algorithm, the terminal device can use the training set information and the cluster center location information to train the first optimized detection model until the performance evaluation values ​​corresponding to the validation set information and the test set information meet the requirements, thereby generating a trained target detection model. In particular, due to the recognition of the maturity of cherry tomatoes in a complex natural environment, it is necessary to consider the accuracy and real-time performance of the detection network. In order to more objectively measure the detection effect of the model on cherry tomatoes of different maturity, the performance evaluation value may include precision information, recall information, average precision information, average precision mean information and / or harmonic value information, wherein the precision information is used to describe the precision (Precision, P), the recall information is used to describe the recall rate (Recal l, R), the average precision mean information is used to describe the average precision (Average Precision, AP), and the harmonic value information is used to describe the harmonic value (Mean Average Precision, mAP), that is, the F1 value (F1-score); the performance evaluation value is used as a performance evaluation standard for evaluating the target detection method; the detection time of the target detection model can use the average time consumed by the target detection network to detect an image as the standard, and its unit is milliseconds.

[0104] In one possible implementation, the terminal device can use Pycharm to build and improve the YOLOv5n network model. The operating system used for experiments and training can be Windows 10, the processor can be 12th Gen Intel(R) Core(TM) i5-12400F 2.50GHz, the memory can be 16G, the graphics card can be NVIDIA GeForce RTX 3060, 12288MiB, the video memory can be 8G, and the Cuda 11.7, Cudnn8.5.0 and OpenCV4.6.0 libraries are used to accelerate the operation, and Python 3.9.12 is used as the programming language on the Keras deep learning framework.

[0105] In a possible implementation, the input size of the image can be 640×640×3, the batch size can be 32, and the training cycle can be 1000 times. And the stochastic gradient descent method SGD is used as the optimizer to speed up the training process of the neural network. The initial learning rate can be set to 0.01, the momentum factor can be set to 0.937, and the weight decay can be set to 0.0005. In order to improve the performance and training efficiency of the target detection model, the terminal device can adopt an early stopping strategy. During the training process, the performance indicators of the model are monitored in real time, and the training is stopped early when no improvement is observed in the last 100 training cycles to save computing resources.

[0106] Without loss of generality, the terminal device can use the pre-trained weight files and the enhancements of the Mosaic data to further improve the training effect of the model, thereby helping to improve the accuracy and generalization ability of the model and speed up the training process.

[0107] For example, see Figure 5 and Figure 6 The inventors conducted a large number of experiments on the cherry tomato maturity detection method; the comparison results with the original YOLOv5n model are shown in Table 1 below. As can be seen from Table 1, compared with YOLOv5n, the improved YOLOv5n target detection network can achieve an average accuracy of 95% under the validation set. The average detection time is 5.3 / ms, which can effectively identify cherry tomatoes and detect fruit maturity in a natural environment, and the improved YOLOv5n network structure occupies only 4.4M memory, providing a reference for accurately locking targets and graded picking in the automatic mechanical picking of cherry tomatoes.

[0108] Table 1 Model comparison results

[0109]

[0110]

[0111] The implementation principle of the method for detecting the maturity of cherry tomatoes picked by machine in the embodiment of the present application is as follows: the terminal device can first obtain the actual image information to be detected, and then input the actual image information to be detected into the trained target detection model to accurately generate maturity detection result information, thereby accurately judging the maturity of cherry tomatoes. The method is suitable for cherry tomato picking scenarios with high real-time requirements, improves the picking effect, and effectively improves reliability.

[0112] It should be noted that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0113] The embodiment of the present application also provides a cherry tomato maturity detection system for machine picking. For ease of description, only the parts related to the present application are shown, such as Figure 7 As shown, the system 70 includes:

[0114] The actual image information acquisition module 71 to be detected is used to acquire the actual image information to be detected;

[0115] The maturity detection result information generating module 72 is used to input the actual image information to be detected into the trained target detection model to generate maturity detection result information.

[0116] Optionally, the system 70 further includes:

[0117] Target detection model building module: used to build target detection models;

[0118] Cherry tomato growth posture original image set information acquisition module: used to acquire cherry tomato growth posture original image set information, wherein the cherry tomato growth posture original image set information includes multiple original image information;

[0119] A first cherry tomato growth posture optimized image set information generation module is used to perform data preprocessing on the original image set information of the cherry tomato growth posture to generate the first cherry tomato growth posture optimized image set information, wherein the first cherry tomato growth posture optimized image set information includes a plurality of first optimized image information;

[0120] The second cherry tomato growth posture optimized image set information generation module is used to perform data compression processing on the first cherry tomato growth posture optimized image set information to generate the second cherry tomato growth posture optimized image set information, wherein the second cherry tomato growth posture optimized image set information includes multiple second optimized image information, and the image resolution of the second optimized image information is smaller than the image resolution of the first optimized image information.

[0121] Optionally, the system 70 further includes:

[0122] Fruit type information generation module: used to perform maturity type labeling processing on the second cherry tomato growth posture optimized image set information, and generate ripe fruit type information, semi-ripe fruit type information and immature fruit type information.

[0123] Optionally, the system 70 further includes:

[0124] A third cherry tomato growth posture optimized image set information generation module is used to remove noise from the second cherry tomato growth posture optimized image set information based on a preset filtering algorithm to generate a third cherry tomato growth posture optimized image set information;

[0125] The fourth cherry tomato growth posture optimized image set information generation module is used to perform data enhancement processing on the third cherry tomato growth posture optimized image set information to generate the fourth cherry tomato growth posture optimized image set information.

[0126] Optionally, the target detection model building module includes:

[0127] Target detection model acquisition submodule: used to obtain an untrained target detection model;

[0128] The first optimized detection model generation submodule is used to add the preset CA attention mechanism module and loss function to the untrained target detection model to generate the first optimized detection model;

[0129] The training set information generation submodule is used to divide the fourth cherry tomato growth posture optimization image set information based on a preset division ratio to generate training set information, verification set information and test set information;

[0130] Cluster center location information generation submodule: used to cluster the anchor frames of the training set information based on a preset clustering algorithm to generate cluster center location information;

[0131] Target detection model generation submodule: used to train the first optimized detection model based on the training set information and cluster center location information until the performance evaluation values ​​corresponding to the validation set information and the test set information meet the requirements, and generate a trained target detection model, wherein the performance evaluation values ​​include precision information, recall rate information, average precision information, average precision mean information and / or harmonic value information.

[0132] It should be noted that the information interaction, execution process and other contents between the above-mentioned modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0133] The present application also provides a terminal device, such as Figure 8 As shown, the terminal device 80 of this embodiment includes: a processor 81, a memory 82, and a computer program 83 stored in the memory 82 and executable on the processor 81. When the processor 81 executes the computer program 83, the steps in the above-mentioned cherry tomato maturity detection method embodiment are implemented, for example Figure 1 Steps S100 to S200 shown; or, when the processor 81 executes the computer program 83, the functions of each module in the above device are realized, for example Figure 7 The functions of modules 71 to 72 are shown.

[0134] The terminal device 80 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server, and the terminal device 80 includes but is not limited to a processor 81 and a memory 82. Those skilled in the art will appreciate that Figure 8 It is only an example of the terminal device 80 and does not constitute a limitation of the terminal device 80. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 80 may also include input and output devices, network access devices, buses, etc.

[0135] Among them, the processor 81 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.; the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0136] The memory 82 may be an internal storage unit of the terminal device 80, such as a hard disk or memory of the terminal device 80, or the memory 82 may be an external storage device of the terminal device 80, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device 80; further, the memory 82 may also include both an internal storage unit and an external storage device of the terminal device 80, and the memory 82 may also store a computer program 83 and other programs and data required by the terminal device 80, and the memory 82 may also be used to temporarily store data that has been output or is to be output.

[0137] One embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the steps of each of the above method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc.; the computer-readable medium may include: any entity or device that can carry computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0138] The above are all preferred embodiments of the present application, and the protection scope of the present application is not limited thereto. Therefore, all equivalent changes made according to the methods, principles, and structures of the present application should be included in the protection scope of the present application.

Claims

1. A method for detecting the maturity of cherry tomatoes picked by machines, characterized in that: The method comprises: Obtaining actual image information to be detected; The actual image information to be detected is input into a trained target detection model to generate maturity detection result information.

2. The method according to claim 1, characterized in that Before inputting the actual image information to be detected into the trained target detection model to generate maturity detection result information, the method further includes: Constructing the target detection model; Wherein, before constructing the target detection model, the method further includes: Acquire the original image set information of the growth posture of cherry tomatoes, wherein the original image set information of the growth posture of cherry tomatoes includes a plurality of original image information; Performing data preprocessing on the original image set information of the growth posture of cherry tomatoes to generate first optimized image set information of the growth posture of cherry tomatoes, wherein the first optimized image set information of the growth posture of cherry tomatoes includes a plurality of first optimized image information; The first cherry tomato growth posture optimized image set information is subjected to data compression processing to generate a second cherry tomato growth posture optimized image set information, wherein the second cherry tomato growth posture optimized image set information includes a plurality of second optimized image information, and the image resolution of the second optimized image information is smaller than the image resolution of the first optimized image information.

3. The method according to claim 2, characterized in that After performing data compression processing on the first cherry tomato growth posture optimized image set information to generate the second cherry tomato growth posture optimized image set information, the method further includes: The second cherry tomato growth posture optimized image set information is subjected to maturity type labeling processing to generate ripe fruit type information, semi-ripe fruit type information and immature fruit type information.

4. The method according to claim 3, characterized in that After the maturity type labeling process is performed on the second cherry tomato growth posture optimized image set information to generate ripe fruit type information, semi-ripe fruit type information and immature fruit type information, the method further includes: Based on a preset filtering algorithm, the second cherry tomato growth posture optimized image set information is subjected to noise removal processing to generate a third cherry tomato growth posture optimized image set information; Data enhancement processing is performed on the third cherry tomato growth posture optimized image set information to generate a fourth cherry tomato growth posture optimized image set information.

5. The method according to claim 4, characterized in that The constructing of the target detection model comprises: Get an untrained object detection model; Adding a preset CA attention mechanism module and a loss function to the untrained target detection model to generate a first optimized detection model; Based on a preset division ratio, the fourth cherry tomato growth posture optimization image set information is divided and processed to generate training set information, verification set information and test set information; Based on a preset clustering algorithm, clustering is performed on the anchor frames of the training set information to generate cluster center position information; Based on the training set information and the cluster center location information, the first optimized detection model is trained until the performance evaluation values ​​corresponding to the validation set information and the test set information meet the requirements, thereby generating a trained target detection model, wherein the performance evaluation values ​​include precision information, recall rate information, average precision information, average precision mean information and / or harmonic value information.

6. A cherry tomato maturity detection system for machine picking, characterized in that: The system comprises: The actual image information acquisition module to be detected is used to obtain the actual image information to be detected; Maturity detection result information generation module: used to input the actual image information to be detected into the trained target detection model to generate maturity detection result information.

7. The system according to claim 6, characterized in that The system further comprises: Target detection model construction module: used to construct the target detection model; A cherry tomato growth posture original image set information acquisition module is used to acquire the cherry tomato growth posture original image set information, wherein the cherry tomato growth posture original image set information includes a plurality of original image information; A first cherry tomato growth posture optimized image set information generation module: used to perform data preprocessing on the cherry tomato growth posture original image set information to generate the first cherry tomato growth posture optimized image set information, wherein the first cherry tomato growth posture optimized image set information includes a plurality of first optimized image information; The second cherry tomato growth posture optimized image set information generation module is used to perform data compression processing on the first cherry tomato growth posture optimized image set information to generate the second cherry tomato growth posture optimized image set information, wherein the second cherry tomato growth posture optimized image set information includes multiple second optimized image information, and the image resolution of the second optimized image information is smaller than the image resolution of the first optimized image information.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images