Tomato maturity estimation and grading counting method, system and equipment

Abstract

The invention provides a tomato maturity estimation and hierarchical counting method, system and device based on occlusion state decision, and the method comprises the steps: processing a tomato image through a multi-task deep learning network, carrying out the maturity grade estimation through combining a regression task, and carrying out the occlusion analysis through a classification task; depth feature extraction is carried out for various factors such as color features, texture information and spatial relations of tomatoes so as to improve maturity estimation precision and counting accuracy. In order to solve the maturity estimation problem under the shielding condition, shielding state prediction is introduced, an exponential weighted average (EMA) mechanism is adopted, and the influence of shielding on maturity estimation is minimized by dynamically adjusting the weight of maturity prediction of each frame. Through combination of multi-frame image detection information, the occlusion perception updating method provided by the invention effectively improves the precision of tomato maturity estimation, and realizes automatic counting of tomatoes with different maturity grades in a greenhouse environment.

Description

Technical Field

[0001] The present invention relates to the field of agricultural automation, and specifically, to a method, system and device for estimating and grading and counting tomato maturity based on occlusion state decision-making. Background Art

[0002] With the continuous growth of the global population and the continuous improvement of people's living standards, higher requirements are put forward for agricultural products in terms of quantity and quality. Agricultural automation technology has emerged and developed rapidly, and its application in the field of greenhouse agriculture is particularly remarkable. Automated inspection robots have gradually become an important force in crop monitoring and management. As a high-economic-value crop widely planted globally, the growth process of tomatoes is extremely complex and is easily affected by environmental factors. In vertical agriculture and greenhouse environments, the growth heights and postures of different tomato fruits are different, and the occlusion problem between them is also relatively prominent, which brings many technical problems to the automated inspection system.

[0003] At present, tomato maturity estimation is a key link in tomato planting management, which is crucial for determining the optimal harvesting time, ensuring fruit quality and realizing efficient supply chain management. Existing tomato maturity estimation methods mainly include traditional visual processing techniques and deep learning-based algorithms. However, in actual application scenarios, these methods have obvious limitations. Traditional visual processing techniques are difficult to accurately extract fruit features in the face of changing lighting conditions; while deep learning-based algorithms, although improving accuracy to a certain extent, still cannot achieve precise assessment of maturity when dealing with fruit occlusion problems and coping with fruit surface texture differences. Especially when tomato fruits are partially occluded by stems, leaves or other fruits, traditional methods cannot obtain complete fruit feature information, resulting in a significant reduction in the accuracy of maturity estimation.

[0004] In order to improve the accuracy of tomato maturity estimation, many existing technologies choose to rely on deep learning-based classification networks for maturity level determination. However, this method has a series of inherent defects. On the one hand, the growth of tomatoes is a continuous dynamic process, and their maturity does not show a clear phased "level" division. Due to the influence of environmental factors on the fruit growth rate, there are often transitional fruits in the transition period between two maturity states, which makes it difficult for traditional classification methods to accurately define. When manually annotating maturity categories, due to the interference of various factors such as environmental lighting, occluders and fruit surface texture, annotators are prone to ignoring subtle changes, resulting in subjective and error-prone annotation results, which in turn affect the label consistency of network training.

[0005] On the other hand, the fixed category division relied on by the maturity classification task cannot sensitively capture the fine-grained changes in tomato maturity. As tomatoes change their states continuously during the growth process, the fixed maturity thresholds cannot accurately describe their complex growth trajectories. For example, if tomatoes are simply divided into three grades: immature, mature, and overripe, in actual situations, it is difficult to accurately classify the states of tomatoes between two grades. The classification network can only select approximate categories and cannot precisely quantify the actual maturity values, resulting in a decrease in overall accuracy.

[0006] Through the retrieval of patent documents, it is found that the invention patent with the publication number CN107609111A discloses a retrieval method for loquat fruit variety identification, quality grading, and maturity determination, including: collection of loquat fruit standard samples covering different varieties, different quality levels, and different maturities; investigation and analysis of the external appearance characteristics and internal physical and chemical characteristics of loquat fruits; establishment of models related to external appearance characteristics and internal physical and chemical characteristics; collection, mining, and analysis of sample images and videos; definition of user preferences and retrieval query process; output of results and feedback optimization. This patent does not solve the occlusion problem.

[0007] To sum up, traditional tomato maturity estimation methods, especially those based on classification networks, have serious deficiencies in maturity category division, annotation accuracy, and description of fruit growth continuity, greatly restricting the performance of tomato maturity estimation models based on deep learning and unable to meet the development needs of modern agriculture for precision and intelligence. Therefore, it is of great practical significance and urgency to develop a brand-new method, system, and equipment for tomato maturity estimation, grading, and counting. Summary of the Invention

[0008] Aiming at the defects in the prior art, the purpose of the present invention is to provide a method, system, and equipment for tomato maturity estimation, grading, and counting based on occlusion state decision-making.

[0009] According to a method for tomato maturity estimation, grading, and counting based on occlusion state decision-making provided by the present invention, the method includes the following steps:

[0010] Step S1, obtaining an image of a tomato fruit through an image acquisition device;

[0011] Step S2, performing maturity annotation on the tomato fruit in the image through a tomato maturity annotation tool to generate an annotation file;

[0012] Step S3, based on the image and the annotation file, using a cherry tomato multi-task detection model to estimate the maturity of the tomato fruit and output the maturity estimation result and the occlusion state;

[0013] Step S4: Classify and count the tomato fruits according to the maturity estimation result and occlusion status, and calculate the number and total weight of fruits with different maturities.

[0014] Preferably, step S2 includes the following sub-steps:

[0015] Step S2.1: Process the tomato area selected by the user through the automatic hue value calculation module, calculate the average hue value within the tomato area, and determine the preliminary maturity level according to the preset maturity mapping framework;

[0016] Step S2.2: When there is a deviation between the preliminary maturity level and the actual situation, manually adjust the preliminary maturity level of the tomato area through the manual adjustment module;

[0017] Step S2.3: Integrate the hue value and its weight of each tomato in the image through the data aggregation and maturity evaluation module, calculate the maturity percentage of the overall tomato cluster through weighted aggregation, and assign the final maturity level to each tomato;

[0018] Step S2.4: Prompt the user to make annotations according to the occlusion status, and generate an annotation file containing maturity labels and occlusion labels.

[0019] Preferably, in step S2.3, the calculation formula for the overall maturity percentage is as follows:

[0020]

[0021] where, m c represents the overall maturity percentage of a cluster of tomato clusters, n is the number of single tomato fruits in the tomato cluster, h f,i is the average hue value of the selected area of the i-th tomato, and are respectively the minimum and maximum hue values of the hue range corresponding to the j-th class of maturity to which the single tomato fruit belongs, ω i is the maturity weight value corresponding to the maturity level of the i-th tomato fruit.

[0022] Preferably, the cherry tomato multi-task detection model in step S3 is based on deep learning, and by decoupling the regression task and the classification task, it outputs the maturity estimation result and occlusion status information.

[0023] Preferably, step S3 includes the following sub-steps:

[0024] Step S3.1: Use the MobileViT backbone network to extract features from the input image;

[0025] Step S3.2: In the maturity estimation task, generate a continuous maturity score through the fusion module and the sigmoid activation function;

[0026] Step S3.3, in the occlusion judgment task, output the occlusion state through the fusion module and the softmax activation function;

[0027] Step S3.4, use the combined loss function to train the model. The combined loss function includes mean squared error loss and cross-entropy loss, and dynamically adjusts the loss contribution of each task through learnable uncertainty weights.

[0028] Step S3.5, combine the exponential moving average mechanism to dynamically adjust the estimation result of the maturity of tomato fruits.

[0029] Preferably, in step S3.4, the calculation formula of the total loss is as follows:

[0030] L = exp(-logσ reg )·L reg +logσ reg +exp(-logσ cls )·L cls +logσ cls (2)

[0031] Where L reg and L cls respectively represent the regression loss and the classification loss, while σ reg and σ cls are learnable weight parameters that respectively control the contribution of the losses of the regression task and the classification task to the total loss.

[0032] Preferably, in step S3.5, the formula for dynamic adjustment is:

[0033] M t = α·M new +(1-α)·M t-1 (3)

[0034] Where M t represents the maturity estimation result after the fusion of the current frame result, M new represents the maturity estimation result of the current frame, M t-1 represents the fused maturity estimation result of the previous frame, α is the smoothing factor, and the maturity estimation result is updated according to the occlusion state using different smoothing factors.

[0035] Preferably, step S4 uses a multi-object tracking model to perform real-time tracking on each tomato fruit, including the following sub-steps:

[0036] Step S4.1, assign a unique identifier to each tomato fruit and keep the ID consistent in the image sequence;

[0037] Step S4.2: Allocate tomato fruits to different maturity categories according to the maturity estimation results.

[0038] Step S4.3: Count the number of tomatoes in each maturity category, calculate the total weight by combining the average weight of each category, and output the number of fruits and the total weight of different maturities for automated harvesting and yield estimation.

[0039] The present invention also provides a tomato maturity estimation and grading counting system based on occlusion state decision-making, including:

[0040] Module M1: Obtain images of tomato fruits through an image acquisition device.

[0041] Module M2: Label the maturity of tomato fruits in the image through a tomato maturity annotation tool to generate an annotated image.

[0042] Module M3: Based on the annotated image, use a cherry tomato multi-task detection model to estimate the maturity of tomato fruits and output the maturity estimation result and occlusion state.

[0043] Module M4: Classify and count tomato fruits according to the maturity estimation result and occlusion state, and calculate the number of fruits and the total weight of different maturities.

[0044] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the above-mentioned tomato maturity estimation and grading counting method based on occlusion state decision-making.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] 1. The present invention processes tomato images through a multi-task deep learning network, uses the regression task for maturity level estimation, and at the same time uses the classification task for occlusion analysis. For various factors such as the color characteristics, texture information, and spatial relationship of tomatoes, deep feature extraction is performed to improve the accuracy of maturity estimation and the accuracy of counting.

[0047] 2. To solve the problem of maturity estimation in the case of occlusion, the present invention introduces occlusion state prediction and adopts an exponential weighted average (EMA) mechanism, which can dynamically adjust the weight of each frame of maturity prediction, thereby minimizing the impact of occlusion on maturity estimation. When a tomato is in an occluded state, a smaller smoothing factor is used to update the maturity estimation to reduce the influence of unreliable estimation; when a tomato is not occluded, a larger smoothing factor is used for updating to ensure the stability and accuracy of maturity estimation.

[0048] 3. By integrating the detection information of multiple frames of images, the proposed occlusion-aware update method of the present invention effectively improves the accuracy of tomato maturity estimation and realizes the automatic counting of tomatoes with different maturity levels in the greenhouse environment.

[0049] 4. The present invention integrates maturity estimation, occlusion detection and fruit tracking technologies, improving the automatic inspection ability and yield prediction ability of tomato picking robots in complex greenhouse environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] By reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings, other features, objects and advantages of the present invention will become more apparent:

[0051] Figure 1 It is a flowchart of a method for estimating and grading the maturity of tomatoes based on occlusion state decision in this embodiment;

[0052] Figure 2 It is a schematic diagram of the tomato inspection robot in this embodiment;

[0053] Figure 3 It is a schematic diagram of a single tomato with different maturity levels in this embodiment;

[0054] Figure 4 It is a schematic diagram of the calibration process of the custom tomato maturity calibration tool in this embodiment;

[0055] Figure 5 It is a structural diagram of a multi-task network model improved based on MobileViT in the embodiment of the present invention.

[0056] Figure 6 It is the maturity estimation results from different perspectives during the movement of the inspection robot in the embodiment of the present invention;

[0057] Figure 7 It is the real-time maturity tracking and counting results in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0058] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several changes and improvements can still be made. These all belong to the protection scope of the present invention.

[0059] The present invention discloses a method, system and device for estimating and grading and counting the maturity of tomatoes based on occlusion state decision, aiming to solve the problems of large deviation in estimating the maturity of tomatoes and low accuracy in counting in the prior art. The method processes tomato images through a multi-task deep learning network, uses the regression task for estimating the maturity level, and at the same time uses the classification task for occlusion analysis. For various factors such as the color characteristics, texture information and spatial relationship of tomatoes, deep feature extraction is performed to improve the accuracy of maturity estimation and the accuracy of counting. In order to solve the problem of maturity estimation in the case of occlusion, the present invention introduces occlusion state prediction and adopts an exponential weighted average (EMA) mechanism, which can dynamically adjust the weight of each frame of maturity prediction, thereby minimizing the impact of occlusion on maturity estimation. When the tomato is in an occluded state, a smaller smoothing factor is used to update the maturity estimation to reduce the impact of unreliable estimation; when the tomato is not occluded, a larger smoothing factor is used for updating to ensure the stability and accuracy of maturity estimation. By integrating the detection information of multiple frames of images, the proposed occlusion-aware update method effectively improves the accuracy of tomato maturity estimation and realizes the automatic counting of tomatoes with different maturity levels in a greenhouse environment.

[0060] Embodiment 1:

[0061] Figure 1 It is a flowchart of a method for estimating and grading and counting the maturity of tomatoes based on occlusion state decision in this embodiment.

[0062] As Figure 1 shown, this embodiment provides a method for estimating and grading and counting the maturity of tomatoes based on occlusion state decision, including the following steps:

[0063] Step S1, obtaining an image of a tomato fruit through an image acquisition device.

[0064] Figure 2 It is a schematic diagram of a tomato inspection robot in this embodiment.

[0065] In this embodiment, an image of a tomato fruit is obtained through a camera on the tomato inspection robot.

[0066] Step S2, performing maturity annotation on the tomato fruits in the image through a tomato maturity annotation tool to generate an annotation file.

[0067] In this embodiment, the tomato maturity annotation tool aims to provide standardized annotation data support for a multi-task deep learning model.

[0068] Figure 3 It is a schematic diagram of a single tomato with different maturity levels in this embodiment.

[0069] As Figure 3As shown, the tomato maturity annotation tool automatically calculates and annotates the hue values of single tomato regions according to the six maturity stages of single tomato fruits defined by horticultural experts, so as to infer their maturity levels. In the specific implementation process, the tomato maturity annotation tool initially determines the maturity classification of tomatoes by analyzing the hue values of tomatoes in the image and combining the preset hue range mapping framework. If there is a deviation between the automatically calculated result and the actual situation, the user can make manual adjustments. The design of this tool effectively improves the consistency and accuracy of tomato maturity annotation, providing reliable data support for subsequent maturity estimation and yield counting.

[0070] Figure 4 This is a schematic diagram of the calibration process of the custom tomato maturity calibration tool in this embodiment.

[0071] As Figure 4 shown, step S2 includes the following sub-steps:

[0072] In step S2.1, the automatic hue value calculation module processes the tomato region selected by the user, calculates the average hue value within the tomato region, and determines the preliminary maturity level according to the preset maturity mapping framework, so as to efficiently and accurately provide preliminary maturity annotations for a large number of tomato images.

[0073] In step S2.2, when there is a deviation between the preliminary maturity level and the actual situation, the manual adjustment module manually adjusts the preliminary maturity level of the tomato region; by directly specifying the maturity stage and applying the corresponding weight. This provides flexibility, enabling the annotation process to adapt to various actual situations and avoiding errors in automated calculations.

[0074] In step S2.3, the data aggregation and maturity evaluation module integrates the hue values and their weights of each tomato in the image, calculates the maturity percentage of the overall tomato cluster through weighted aggregation, and assigns a final maturity level to each tomato. Specifically, the calculation formula for the overall maturity percentage is as follows:

[0075]

[0076] where, m c represents the overall maturity percentage of a cluster of tomatoes, n is the number of single tomato fruits in the tomato cluster, h f,i is the average hue value of the selected region of the i-th tomato, and are respectively the minimum and maximum hue values of the hue range corresponding to the j-th type of maturity to which the single tomato fruit belongs, ω i is the maturity weight value corresponding to the maturity level of the i-th tomato fruit.

[0077] With the above design and functions, the tomato maturity annotation tool can not only automatically generate hue values and maturity levels based on expert knowledge, but also allow users to make manual adjustments according to actual situations, ensuring flexibility and accuracy. Finally, by integrating hue value calculation, weighted aggregation, and occlusion status annotation, the tool realizes a systematic and reliable evaluation process for tomato maturity.

[0078] Step S2.4, prompt the user to make an annotation ("yes" or "no") according to the occlusion status, and generate an annotation file containing the maturity label and the occlusion label, further improving the accuracy of maturity evaluation.

[0079] Specifically, the annotation file is saved in text format (such as a txt document) and used as the true label for deep learning supervised training for subsequent model training and evaluation.

[0080] Step S3, based on the image and the annotation file, use the Cherry Tomato Multi-Task Detection (CT-MTD) model to estimate the maturity of tomato fruits, and output the maturity estimation result and the occlusion status.

[0081] Specifically, the Cherry Tomato Multi-Task Detection (CT-MTD) model is based on deep learning. By decoupling the regression task and the classification task, it outputs the maturity estimation result and the occlusion status information. The CT-MTD model uses a lightweight MobileViT backbone for efficient feature extraction. At the same time, this multi-task framework is specifically customized for maturity estimation and occlusion status classification, with strong task collaboration ability and high efficiency.

[0082] Figure 5 It is the structural diagram of the multi-task network model improved based on MobileViT in the embodiments of the present invention.

[0083] As Figure 5 shown, step S3 includes the following sub-steps:

[0084] Step S3.1, use the MobileViT backbone network to extract features from the input image.

[0085] In this embodiment, the image of the tomato fruit obtained in step S1 is used as the input. The input image is first processed through a series of convolutional layers, including an initial 3×3 convolution and downsampling operations, followed by multiple MobileViT blocks and Mobile Inverted Bottleneck (MV2) layers. Through these operations, the spatial dimension is gradually reduced while enhancing the feature extraction ability. These layers constitute the shared backbone of the model, enabling the model to efficiently extract general low-level features, thereby improving the efficiency of multi-task detection. To improve the performance of the model in complex environments, the CT-MTD model introduces a Coordinate Attention (CA) mechanism before the branch of each task. For details, see Figure 2. The CA mechanism enhances important spatial features by encoding spatial position information, especially in cases where tomato fruits may be blocked by leaves or other fruits. After the CA mechanism, the model applies a Fusion Block to integrate features and further optimize the information flow to improve the performance of each task.

[0086] Step S3.2, in the maturity estimation task, generate a continuous maturity score through the fusion module and the sigmoid activation function.

[0087] In this embodiment, the maturity estimation problem is modeled as a non - linear regression problem. After the fusion module, a linear layer and a sigmoid activation function are connected to generate a continuous maturity score between 0 and 1, representing the maturity level of the tomato. This score depends on the hue and other visual features and can finely capture the gradual changes from immature to mature. This regression - based method can effectively evaluate maturity and adapt to the phased transitions that are difficult to simply classify into discrete categories.

[0088] Step S3.3, in the occlusion judgment task, output the occlusion state through the fusion module and the softmax activation function.

[0089] In this embodiment, the occlusion judgment task is regarded as a binary classification problem. After the fusion module, a Batch Normalization layer, a linear layer and a softmax activation function are connected, and finally the occlusion state (occluded or unoccluded) is output. This branch can support downstream applications that require a clear view of each tomato, such as precise picking and counting tasks.

[0090] Specifically, the calculation formula of the total loss is as follows:

[0091] L = exp(-logσ reg )·L reg +logσ reg +exp(-logσ cls )·L cls +logσ cls (5)

[0092] Where L reg and L cls represent the regression loss and the classification loss respectively, while σ reg and σ cls are learnable weight parameters that respectively control the contribution of the losses of the regression task and the classification task to the total loss. Through this uncertainty - weighted method, the model can adaptively adjust the learning process of different tasks and optimize the overall performance.

[0093] Step S3.4: Compare the predicted results of the model with the true labels, and use a combined loss function to train the model. The combined loss function includes mean squared error (MSE) loss and cross-entropy loss, and dynamically adjusts the loss contributions of each task through learnable uncertainty weights.

[0094] In this embodiment, since the CT-MTD model simultaneously processes two tasks: maturity estimation and occlusion state classification, a combined loss function is used for these two tasks:

[0095] (1) For the maturity estimation task (regression loss), mean squared error (MSE) is adopted as the loss function to penalize the difference between the predicted maturity score and the true value;

[0096] (2) For the occlusion detection task (classification loss), the cross-entropy loss function is used, which is commonly used in binary classification problems to calculate the negative log-likelihood between the predicted occlusion probability and the true label, thereby enhancing the model's ability to distinguish between occluded and unoccluded tomatoes.

[0097] To optimize the training of the multi-task model, we introduce learnable uncertainty weights for each task. This can dynamically adjust the contribution of the total loss according to the relative importance of each task, further improving the performance of the model.

[0098] In step S3.5, combined with the exponential moving average (EMA) mechanism, the maturity estimation results of tomato fruits are dynamically adjusted.

[0099] Figure 6 This is the maturity estimation results of the inspection robot at different perspectives during the movement in the embodiment of the present invention;

[0100] Specifically, when performing maturity estimation, accurate estimation requires sufficient visual feature support. Experimental results show that, as Figure 6 shown, for unoccluded tomatoes, the maturity level predicted by the model is about 40.9%, which is relatively close to the true maturity of 46.3% manually annotated; however, in the presence of occlusion, the predicted maturity level will increase to 51.6% and 52.1%. This deviation is mainly due to the loss of key visual features caused by occlusion, which affects the accuracy of maturity estimation. To effectively address this problem, the present invention introduces the occlusion state as an output of the model and combines the exponential moving average (EMA) mechanism to dynamically adjust the maturity prediction results in each frame of image.

[0101] The EMA mechanism plays a key role in the maturity estimation of multi-frame images. Specifically, for each tomato fruit t being tracked, the system updates the maturity estimation value with different smoothing factors according to its occlusion state. When the fruit is not occluded (O = No), a higher smoothing factor (α = 0.5) is used, while when the fruit is occluded (O = Yes), a smaller smoothing factor (α = 0.1) is used. This processing method effectively reduces the error caused by unreliable estimations and improves the robustness of the system through the accumulation of time information, ensuring more stable and accurate maturity tracking among multi-frame images.

[0102] M t = α·M new + (1 - α)·M t-1 (6)

[0103] Where M t represents the maturity estimation result after the fusion of the current frame result, M new represents the maturity estimation result of the current frame, and M t-1 represents the fused maturity estimation result of the previous frame.

[0104] Step S4: According to the maturity estimation result and the occlusion state, classify and count the tomato fruits, and calculate the number and total weight of fruits with different maturities.

[0105] In this embodiment, the effective integration of the fruit tracking technology and the maturity estimation framework enables accurate estimation of the maturity of tomatoes and yield counting in complex environments such as greenhouses. In practical applications, tomatoes are often occluded, and some fruits are occluded by stems or other tomato fruits, which poses a great challenge to accurate maturity estimation.

[0106] In the process of tomato maturity estimation, in addition to considering the influence of occlusion, another key factor is the tracking and classification of fruits. To achieve comprehensive counting based on maturity levels, the present invention further combines the maturity estimation and occlusion state update mechanism with the fruit tracking and counting framework to ensure accurate tracking and classification of tomato fruits in complex environments.

[0107] Specifically, step S4 uses a multi-object tracking model (such as BYTETracker) to perform real-time tracking on each tomato fruit, including the following sub-steps:

[0108] Step S4.1: Assign a unique identifier (ID) to each tomato fruit and keep the ID consistent in the image sequence to ensure that the same fruit can be tracked across different frames. The tracking information of each fruit includes its position, maturity estimation, occlusion state, etc. in the current frame.

[0109] Step S4.2: Allocate tomato fruits to different maturity categories according to the maturity estimation results.

[0110] By tracking the tomato fruits for each ID, the system can obtain the status changes of each tomato fruit in the entire image sequence. The fused results of maturity estimation (the estimation results obtained from the current frame and historical frame information) will be incorporated into the framework of fruit tracking, updating the status of the fruits in each frame of the image, and performing classification and counting. That is, the system allocates each tomato to different maturity categories according to the maturity estimation results of each tomato. The number of tomatoes in each category will be updated in real time for subsequent yield prediction.

[0111] Step S4.3: Count the number of tomatoes in each maturity category, calculate the total weight by combining the average weight of each category, and output the number of fruits and the total weight of different maturities for automated harvesting and yield estimation.

[0112] Figure 7 This is the real-time maturity tracking and counting result in the embodiment of the present invention.

[0113] Finally, based on the number of tomatoes in each maturity category and the average weight of each category, the system can estimate the total weight of the fruits in this area (such as Figure 7 ). By combining maturity estimation with grading tracking, the system can not only accurately track tomato fruits, but also precisely calculate the number of fruits and their total weight at different maturity stages, providing reliable data support for automated harvesting and yield estimation.

[0114] In this embodiment, the method for grading and counting tomato maturity is as follows:

[0115]

[0116]

[0117] Embodiment 2:

[0118] The present invention also provides a tomato maturity estimation and grading and counting system based on occlusion state decision. The tomato maturity estimation and grading and counting system based on occlusion state decision can be implemented by executing the process steps of the tomato maturity estimation and grading and counting method based on occlusion state decision. That is, those skilled in the art can understand the tomato maturity estimation and grading and counting method based on occlusion state decision as the preferred implementation manner of the tomato maturity estimation and grading and counting system based on occlusion state decision.

[0119] Specifically, the tomato maturity estimation and grading and counting system based on occlusion state decision includes:

[0120] Module M1, which acquires images of tomato fruits through an image acquisition device;

[0121] Module M2, which performs maturity annotation on the tomato fruits in the image through a tomato maturity annotation tool to generate an annotated image;

[0122] Module M3, which estimates the maturity of tomato fruits based on the annotated image by using a cherry tomato multi-task detection model, and outputs the maturity estimation result and the occlusion state;

[0123] Module M4, which grades and counts the tomato fruits according to the maturity estimation result and the occlusion state, and calculates the number and total weight of fruits with different maturities.

[0124] Embodiment 3:

[0125] Figure 2 This is a schematic diagram of the tomato inspection robot in this embodiment.

[0126] Figure 2 As shown, this embodiment provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the method for estimating and grading the maturity of tomatoes based on the occlusion state decision in Embodiment 1 above.

[0127] Those skilled in the art know that in addition to implementing the system and its various devices, modules, and units provided by the present invention in the form of pure computer-readable program code, the method steps can be logically programmed to enable the system and its various devices, modules, and units provided by the present invention to be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers, etc., to achieve the same functions. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be regarded as the structure within the hardware component; the devices, modules, and units for implementing various functions can also be regarded as both software modules for implementing the method and the structure within the hardware component.

[0128] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. Without conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for estimating and grading tomato maturity based on occlusion state decision, characterized in that: The following steps are involved: Step S1, acquiring an image of a tomato fruit by an image acquisition device; Step S2, marking the maturity of the tomato fruits in the image using a tomato maturity marking tool to generate a marking file; Step S3, based on the image and the annotation file, using the cherry tomato multi-task detection model to estimate the maturity of the tomato fruit, and outputting the maturity estimation result and the occlusion status; Step S4, classifying and counting the tomato fruits according to the maturity estimation result and the shielding status, and calculating the number and total weight of fruits at different maturity levels.

2. The method for estimating and grading tomato maturity based on occlusion state decision according to claim 1, characterized in that: The step S2 includes the following sub-steps: Step S2.1, processing the tomato area selected by the user through the automatic tone value calculation module, calculating the average tone value in the tomato area, and determining the preliminary maturity level according to a preset maturity mapping framework; Step S2.2, when the preliminary maturity level deviates from the actual situation, manually adjusting the preliminary maturity level of the tomato area through a manual adjustment module; Step S2.3, integrating the hue value and weight of each tomato in the image through a data aggregation and maturity assessment module, calculating the maturity percentage of the entire tomato cluster through weighted aggregation, and assigning a final maturity level to each tomato; Step S2.4, prompting the user to label according to the occlusion status, and generating a labeling file including a maturity label and an occlusion label.

3. The method for estimating and grading tomato maturity based on occlusion state decision according to claim 2, characterized in that: In step S2.3, the calculation formula of the overall maturity percentage is as follows: Among them, m c represents the overall maturity percentage of a bunch of tomatoes, n is the number of single tomato fruits in the tomato cluster, h is f,i is the average hue value of the selected area of ​​the i-th tomato, and are the minimum and maximum hue values ​​of the hue range corresponding to the jth maturity category of a single tomato fruit, ω i is the maturity weight value of the i-th tomato fruit corresponding to the maturity level.

4. The method for estimating and grading tomato maturity based on occlusion state decision according to claim 1, characterized in that: The cherry tomato multi-task detection model in step S3 is based on deep learning, and outputs maturity estimation results and occlusion status information by decoupling the regression task and the classification task.

5. The method for estimating and grading tomato maturity based on occlusion state decision according to claim 4, characterized in that: The step S3 comprises the following sub-steps: Step S3.1, using the MobileViT backbone network to extract features from the input image; Step S3.2, in the maturity estimation task, a continuous maturity score is generated through the fusion module and the sigmoid activation function; Step S3.3, in the occlusion judgment task, output the occlusion state through the fusion module and the softmax activation function; Step S3.4, training the model using a combined loss function, wherein the combined loss function includes a mean square error loss and a cross entropy loss, and dynamically adjusting the loss contribution of each task through a learnable uncertainty weight; Step S3.5, combining the exponential moving average mechanism, dynamically adjusts the maturity estimation result of the tomato fruit.

6. The method for estimating and grading tomato maturity based on occlusion state decision according to claim 5, characterized in that: In step S3.4, the calculation formula of the total loss is as follows: L=exp(-logσ reg )·L reg +logσ reg +exp(-logσ cls )·L cls +logσ cls (8) Where L reg and L cls represent regression loss and classification loss respectively, and σ reg and σ cls are learnable weight parameters that control the contribution of the regression and classification task losses to the total loss.

7. The method for estimating and grading tomato maturity based on occlusion state decision according to claim 5, characterized in that: In step S3.5, the formula for dynamic adjustment is: M t =α·M new +(1-a)·M t-1 (9) Among them, M t represents the maturity estimation result after the fusion of the current frame results, M new Represents the maturity estimation result of the current frame, M t-1 It represents the fused maturity estimation result of the previous frame, α is the smoothing factor, and the maturity estimation result is updated with different smoothing factors according to the occlusion state.

8. The method for estimating and grading tomato maturity based on occlusion state decision according to claim 1, characterized in that: The step S4 uses a multi-target tracking model to track each tomato fruit in real time, and includes the following sub-steps: Step S4.1, assign a unique identifier to each tomato fruit and keep the ID consistent in the image sequence; Step S4.2, assigning tomato fruits to different maturity categories according to the maturity estimation results; Step S4.3, count the number of tomatoes in each maturity category, calculate the total weight based on the average weight of each category, and output the number and total weight of fruits at different maturity levels for automated harvesting and yield estimation.

9. A tomato maturity estimation and grading counting system based on occlusion state decision, characterized in that: include: Module M1, acquiring an image of a tomato fruit through an image acquisition device; Module M2, using a tomato maturity annotation tool to annotate the maturity of the tomato fruits in the image to generate an annotated image; Module M3, based on the annotated image, uses a cherry tomato multi-task detection model to estimate the maturity of the tomato fruit, and outputs a maturity estimation result and an occlusion status; Module M4, based on the maturity estimation result and the occlusion status, classifies and counts the tomato fruits, and calculates the number and total weight of fruits of different maturity levels.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by a processor, the steps of the tomato maturity estimation and grading counting method based on occlusion state decision according to any one of claims 1 to 8 are implemented.

Images