A machine vision fruit detection method and system based on an instance set

Abstract

This invention discloses a machine vision fruit detection method based on an instance set, comprising: acquiring fruit images under different lighting conditions; annotating the fruit images with bounding boxes, indicating the occlusion relationships between fruits in the annotation information, and classifying and labeling the fruit varieties and maturity; data preprocessing; inputting the preprocessed image data into a feature extraction model, outputting a predicted instance set; using the minimum matching loss function combining the classification loss function and the regression loss function as the loss function of the feature extraction model, solving for the predicted instance set to obtain the optimal solution; and further optimizing the optimal solution through a deduplication NMS post-processing module. This invention solves the problem that the detection accuracy and robustness of fruits in complex scenes are affected by factors such as changes in lighting, occlusion, and overlap, which leads to the need to improve detection accuracy and robustness.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a machine vision fruit detection method and system based on instance sets. Background Technology

[0002] In the process of agricultural modernization, the fruit industry, as an important component, plays a crucial role in ensuring food supply, increasing farmers' income, and promoting rural economic development. With rising labor costs and increasingly demanding fruit quality from consumers, traditional manual fruit inspection and harvesting methods are no longer sufficient to meet the needs of the modern fruit industry. Robotic vision-based fruit inspection technology has emerged to address this need, enabling rapid and accurate fruit inspection. This provides strong support for fruit harvesting, grading, and quality assessment, and is of great significance for improving production efficiency, reducing costs, and enhancing product quality in the fruit industry.

[0003] Currently, robotic vision-based fruit inspection technology still faces many challenges in practical applications. Orchard environments are complex and variable, with unstable lighting conditions. Fruits may be affected by factors such as shading from branches and leaves, and overlapping fruits, all of which increase the difficulty of fruit inspection. Existing inspection methods need improvement in accuracy and robustness under complex scenarios, making it difficult to meet the high precision and reliability requirements of actual production. Therefore, finding a more effective fruit inspection method has become a key research focus and hot topic.

[0004] In recent years, numerous researchers both domestically and internationally have dedicated themselves to the research of robot vision-based fruit detection technology, achieving a series of results. Early research mainly focused on methods based on traditional image processing and machine learning. These methods extract features such as color, shape, and texture of fruits and utilize classifiers such as Support Vector Machines (SVM) and decision trees for fruit recognition and detection. However, traditional methods have certain limitations in feature extraction and model generalization capabilities, making them difficult to adapt to complex and ever-changing orchard environments. With the rapid development of deep learning technology, fruit detection methods based on Convolutional Neural Networks (CNNs) have gradually become mainstream. CNNs can automatically learn the feature representation of images without the need for manual feature design, greatly improving the accuracy and efficiency of detection. For example, based on object detection frameworks such as the YOLO (You Only Look Once) series and Faster R-CNN, researchers have optimized and improved them for fruit detection tasks, achieving good detection results. However, in complex scenarios, such as drastic changes in lighting, severe fruit occlusion, and overlapping, the performance of these methods is still significantly affected. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the aforementioned issues, this invention provides a machine vision-based fruit detection method and system based on instance sets, which solves the problem that fruit detection accuracy and robustness need improvement due to factors such as changes in lighting, occlusion, and overlap in complex scenes.

[0007] (II) Technical Solution

[0008] To address the aforementioned technical problems, this invention provides a machine vision-based fruit detection method based on an instance set, comprising the following steps:

[0009] S1. Data Acquisition: Acquire images of fruits under different lighting conditions;

[0010] S2. Data annotation: The fruit image is annotated with bounding boxes. A rectangular annotation box is drawn for each visible fruit part. The annotation information indicates the occlusion relationship between fruits and classifies and marks the variety and maturity of the fruits.

[0011] S3, Data Preprocessing;

[0012] S4. Input the preprocessed image data into the feature extraction model and output the predicted set of instances: The feature extraction model uses FPN with RoIAlign as the baseline detector as the basic network architecture. After the RoI branch, connect K-1 parallel prediction heads. Each prediction head consists of two fully connected layers to generate a prediction set of less than or equal to K instances corresponding to a proposal.

[0013] S5. Model Training: Using the minimum matching loss function, which combines the classification loss function and the regression loss function, as the loss function of the feature extraction model, the optimal solution is obtained by solving the predicted instance set; and the optimal solution is further optimized by the deduplication NMS post-processing module; S6. Model Update: The parameters of the model are updated in reverse according to the calculated value of the loss function.

[0014] This invention also discloses a machine vision fruit detection system based on an instance set, comprising:

[0015] At least one processor; and at least one memory communicatively connected to said processor, wherein:

[0016] The memory stores program instructions that can be executed by the processor, and the processor can execute the method by calling the program instructions.

[0017] The present invention also discloses a non-transitory computer-readable storage medium storing computer instructions that cause the computer to perform the method described thereon.

[0018] (III) Beneficial Effects

[0019] The above-described technical solution of the present invention has the following advantages:

[0020] (1) This invention improves the original network structure by adding multiple new prediction heads at its tail to modify the definition of each proposal in the prediction process. It proposes a multi-instance prediction idea, which allows each proposal to predict a set of potentially highly overlapping instances. Instead of predicting a single target, it predicts the set of related basic ground truth instances. Compared with the traditional single-instance prediction framework, the new scheme can greatly simplify the learning process in dense scenes, reduce missed detections caused by occlusion, and improve detection accuracy and robustness.

[0021] (2) The present invention targets a set of potentially highly overlapping instances and removes duplicate instances by calculating the overlap between each instance and the retained instances, thereby better adapting to the situation of highly overlapping instances in fruit detection; and based on the principle that predictions from the same proposal must be different, it adds a judgment logic on whether the current instance and the instance retained in the previous stage come from the same proposal, thereby reducing the overlap calculation.

[0022] (3) In this invention, for a set of potentially highly overlapping instances, the minimum matching loss function of the classification loss function and the regression loss function is combined to transform the connection between the elements of different sets that meet the threshold value into the minimum weight matching of the bipartite graph. This avoids the problem that the number of elements in the real instance set may not reach K. It ensures that each fruit can be detected as much as possible in the case of overlap, thereby minimizing the classification error and reducing the false detection rate. The threshold value is set to eliminate obviously unreasonable false predictions and solve the problem of possible empty predictions, thereby further improving the detection accuracy and robustness.

[0023] (4) This invention expands the dataset and improves the generalization ability of the model by using image enhancement technology, so that the model can learn the features of fruit under different rotation angles, different sizes, different local regions, and noise interference, and improve the adaptability to changes in fruit angle, size, partial occlusion, and noise interference, thereby improving the detection accuracy and robustness of the model. Attached Figure Description

[0024] The features and advantages of the invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the invention in any way. In the drawings:

[0025] Figure 1 This is a flowchart of a machine vision fruit detection method based on an instance set according to an embodiment of the present invention. Detailed Implementation

[0026] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0027] This invention provides a machine vision-based fruit detection method based on an instance set, such as... Figure 1 As shown, it includes the following steps:

[0028] Step S1: Data Acquisition: Acquire fruit images under different lighting conditions;

[0029] The choice of camera is crucial during data acquisition. To acquire high-quality fruit images, we selected an industrial camera with high resolution, good low-light performance, and a wide dynamic range. High resolution captures subtle features of the fruit, such as surface texture and blemishes, providing richer information for subsequent inspection and analysis. Good low-light performance ensures clear images under various lighting conditions, especially in dimly lit environments such as early morning, evening, or cloudy days. A wide dynamic range allows the camera to accurately capture fruit details in scenes with both strong and weak light, avoiding overexposure or underexposure. For example, in an orchard, some fruit may be in direct sunlight while others are in the shade; a camera with a wide dynamic range can clearly record images of both groups of fruit simultaneously.

[0030] The selection of a sensor requires comprehensive consideration of multiple factors. We adopted a CMOS sensor due to its advantages such as low power consumption, relatively low cost, high sensitivity, and fast data readout speed. The low power consumption characteristic ensures that the camera does not consume excessive power during long-term data acquisition, making it suitable for use in environments with limited power supply, such as in the field. The relatively low cost facilitates large-scale data acquisition, reducing overall costs. The high sensitivity ensures sufficient image signals are obtained under different lighting conditions, while the fast data readout speed meets the needs of real-time acquisition, ensuring accurate image capture even when fruit is moving or the environment changes.

[0031] Step S2, Data Labeling: Use LabelImg to label the bounding boxes of the fruit images, draw a tight rectangular label box for each visible fruit part, indicate the occlusion relationship between fruits in the labeling information, and classify and mark the variety and maturity of the fruits.

[0032] When annotating fruit targets, we conducted in-depth research and comparison of various annotation tools. Common annotation tools include LabelImg, VGG Image Annotator, and VoTT. LabelImg is characterized by its simple operation and intuitive interface, enabling quick drawing of rectangular annotation boxes, making it suitable for bounding box annotation in object detection tasks. It supports multiple annotation formats, such as PASCAL VOC and YOLO, facilitating integration with different deep learning frameworks. VGGImage Annotator has richer features, supporting polygon annotation and keypoint annotation in addition to basic bounding box annotation, making it suitable for more complex target annotation tasks, but its operation is relatively more complex. VoTT is a cross-platform annotation tool that provides collaborative annotation capabilities, suitable for data annotation projects involving multiple participants, but it is not as convenient as LabelImg in terms of annotation ease and specificity for target detection tasks.

[0033] Considering the characteristics and requirements of fruit detection tasks, we ultimately chose LabelImg as the primary annotation tool. Fruit detection mainly focuses on the location and category of fruits, which can be met using rectangular bounding boxes. LabelImg's ease of use and excellent support for object detection tasks can improve annotation efficiency and accuracy. For some irregularly shaped fruits, although LabelImg's rectangular bounding boxes cannot perfectly fit the fruit's outline, in object detection tasks, the bounding boxes can provide sufficient positional information. Furthermore, in practical applications, the model focuses more on the overall position and approximate shape of the fruit than its precise outline.

[0034] To ensure accuracy and consistency in labeling, we have established detailed labeling specifications and procedures. Regarding the drawing of label boxes, labelers are required to draw rectangular boxes that closely follow the outer contour of the fruit, ensuring the fruit is completely contained within the box, while avoiding excessively large boxes that include too much background information. When multiple fruits overlap or obscure each other, a separate label box must be drawn for each visible portion of the fruit, and the obscuring relationship between the fruits must be noted in the labeling information. For example, if one apple is partially obscured by another apple, separate label boxes for both apples must be drawn, and the labeling file must record which apple is obscured and the degree of obscuring.

[0035] In terms of category labeling, fruits are classified and labeled according to their variety and ripeness. For different varieties of fruit, such as apples, bananas, and oranges, their corresponding English names or abbreviations are used as category labels. For fruits with different levels of ripeness, a description of ripeness is added to the variety label, such as "apple_ripe" for ripe apples and "apple_unripe" for unripe apples.

[0036] The annotation process consists of three main stages: initial annotation, review, and correction. In the initial annotation stage, annotators use the LabelImg tool to annotate the images, accurately drawing annotation boxes and filling in category labels according to annotation specifications. After initial annotation is completed, the review stage begins, where experienced reviewers meticulously check the annotation results, primarily verifying the accuracy of the annotation boxes, the correctness of the category labels, and the absence of any missing annotations. If any issues are found, the annotation results are returned to the annotators for correction. In the correction stage, annotators modify any errors or inaccuracies based on the review comments, ensuring the quality of the annotation results. This rigorous annotation specification and process guarantees the accuracy and consistency of the annotated data, providing reliable data support for subsequent model training.

[0037] Step S3, Data Preprocessing: The labeled data is processed using image enhancement techniques, followed by data normalization. The image enhancement techniques include rotation, scaling, cropping, and adding noise.

[0038] Using image enhancement techniques:

[0039] To expand the dataset and improve the model's generalization ability, we employed various image augmentation techniques. Rotation is a commonly used method, which simulates the visual effect of fruit at different angles by randomly rotating the image by a certain angle (e.g., between -15° and 15°). This allows the model to learn the characteristics of fruit at different rotation angles, improving its adaptability to changes in fruit angle. For example, in an orchard, fruit may exhibit different rotation angles due to wind, swaying branches, etc. Training the model with rotated and augmented images enables it to better recognize fruit at these different angles.

[0040] Scaling operations can change the size of fruit in an image, simulating changes in shooting distance. By randomly scaling the image (e.g., between 0.8x and 1.2x), the model learns the features of fruit of different sizes. In real-world applications, the distance between the robot and the fruit may vary, and the size of the fruit in the image will change accordingly. The scaled and enhanced image helps the model adapt to these size changes.

[0041] Cropping can extract fruit images from different regions of the original image, increasing data diversity. By randomly cropping a portion of the image (e.g., a cropping ratio between 0.7 and 0.9), images containing different parts of the fruit can be obtained, allowing the model to learn the features of the fruit in different local regions. For example, cropping might produce an image containing only half of the fruit, or an image containing both the fruit and some branches and leaves; these images allow the model to better handle situations where the fruit is partially occluded.

[0042] Adding noise is also an effective image enhancement method. By adding Gaussian noise or salt-and-pepper noise to an image, the interference encountered during image acquisition can be simulated, improving the model's robustness to noise. In real orchard environments, images may contain noise due to factors such as light interference and camera sensor noise. Adding noise to the image enhances the model, allowing it to learn how to handle this noise during training, thus enabling more accurate fruit detection in practical applications.

[0043] Data normalization processing:

[0044] Before inputting images into a model for training, the images need to be normalized. The purpose of normalization is to give the data a uniform scale and distribution, facilitating model training. A common normalization method is to map the pixel values ​​of the image from the range [0, 255] to the range [0, 1] or [-1, 1]. The specific normalization formula is:

[0045]

[0046] in These are the original pixel values. and These are the minimum and maximum values ​​of the original pixel values, respectively. These are normalized pixel values. This normalization process avoids model training instability caused by pixel values ​​that are too large or too small. For example, in neural networks, large pixel values ​​may cause some neurons to output excessively, leading to vanishing or exploding gradients. Normalization effectively alleviates these problems, allowing the model to converge faster.

[0047] Normalizing images can also improve the computational efficiency of the model. During model training, normalized data reduces computational load and speeds up training. Because normalized data has a uniform scale, the model can update parameters more stably and accurately, thereby improving training performance and generalization ability.

[0048] Step S4: Input the preprocessed image data into the feature extraction model: The feature extraction model uses FPN with RoIAlign as the baseline detector as the basic network architecture. After the RoI branch, K-1 parallel prediction heads are connected. Each prediction head consists of two fully connected layers, generating a prediction set of less than or equal to K instances corresponding to a proposal.

[0049] This invention uses an FPN (Feature Pyramid Network) with RoIAlign as the baseline detector and its basic network architecture. This architecture demonstrates powerful feature extraction and multi-scale feature fusion capabilities in object detection tasks. The FPN can effectively fuse feature maps at different levels to obtain rich semantic and spatial information, enabling the model to have good detection performance for targets of different sizes.

[0050] In traditional object detection models, each proposal typically predicts only one instance. This can easily lead to missed detections and false positives, especially in fruit detection scenarios where fruits are occluded or overlapping. This method proposes a multi-instance prediction approach that allows each proposal to predict a potentially highly overlapping set of instances, better handling the complex distribution of fruits. Specifically, for each proposal, the new method predicts not a single target, but rather a set of related ground truth instances. Compared to traditional single-instance prediction frameworks, this new approach significantly simplifies the learning process in dense scenes.

[0051] At the front end of the model, an image preprocessing module is used to normalize, crop, and scale the input fruit images to meet the model's input requirements and enhance data diversity. In the feature extraction stage, a network is used to extract features from the preprocessed images, obtaining feature maps at different scales. These feature maps contain rich feature information about the fruit, from low-level details to high-level semantic features.

[0052] In the multi-instance prediction stage, based on the extracted feature maps, multiple instances are predicted for each proposal through an additional prediction branch. Each instance prediction includes category and location information, enabling accurate location and identification of multiple fruits in the image. To further optimize the prediction results, a novel minimum matching loss function and a deduplication NMS post-processing procedure are introduced. The minimum matching loss function calculates the distance between the predicted instance set and the real instance set, serving as the loss function during training to guide the model's learning. The deduplication NMS post-processing removes duplicate predicted instances, improving the accuracy and reliability of the detection results.

[0053] The overall framework of the model is as follows Figure 1As shown, this invention selects the Feature Pyramid Network (FPN) with RoI Alignment as the baseline detector. In the FPN, the Region Proposal Network (RPN) branch is responsible for generating proposals, and the RCNN (or RoI) branch is used to predict the instances corresponding to the RoI proposals. This invention connects K-1 parallel prediction heads after the RoI branch, each prediction head consisting of two fully connected layers (i.e., fc1 and fc2), generating a set of no more than K instances corresponding to a proposal. In this embodiment, the hyperparameter K (i.e., the maximum number of instances a proposal can correspond to) is set to 2. Experiments show that setting K to 2 can well meet the recognition requirements in most cases.

[0054] The specific analysis is as follows:

[0055] The added multi-instance prediction head is one of the core modules of this model. In traditional object detection methods, each proposal predicts only one instance, which can easily lead to inaccurate detection results when fruits are densely distributed or occluded. The multi-instance prediction module breaks this limitation, allowing each proposal to predict a set of potentially highly overlapping instances. Its working principle is based on feature maps, generating no more than K-1 additional prediction branches in parallel for each proposal, each containing predictions of the fruit category and location. Specifically, a prediction head consisting of two fully connected layers is added after the output layer of the feature extraction network. Currently, most mainstream proposal-based detection frameworks use a single detection function that outputs a pair of prediction results (c, l), where c represents the class label with confidence, and l represents the relative coordinates of the target instance. This invention uses the same prediction result, but sets up K detection functions to generate a set of (ci, li) element pairs with no more than K elements. ci represents the category label with confidence, and li represents the relative coordinates of the target instance, i = 1, 2, 3, ..., K. Each (ci, li) pair represents the prediction result for a certain type of object to be detected. It's worth noting that ci can be predicted as the background class, meaning that this (ci, li) pair does not represent any object, i.e., the prediction result of the prediction function is empty. This design actually uses a parameter setting of K=2, so only one detection head is added to the original network. In this way, each proposal can generate multiple prediction instances, thereby improving the detection capability of fruits in complex scenes.

[0056] Step S5, Model Training: The prediction set is optimized to obtain the optimal solution by using the minimum matching loss function that combines the classification loss function and the regression loss function as the loss function of the feature extraction model; and the optimal solution is further optimized by the deduplication NMS post-processing module.

[0057] The loss function of this invention is defined based on the minimum weight matching of bipartite graphs, combining classification loss and regression loss to ensure that the model can simultaneously optimize instance location prediction and class prediction during training.

[0058] Minimum matching loss function: The minimum matching loss function is used to measure the difference between the predicted instance set and the real instance set, serving as the loss function during training to guide model parameter updates. In the fruit detection task, accurately calculating the loss between the predicted results and the real situation is crucial for model learning. Considering the following points: the predicted set may contain background classes, i.e., empty predictions; the number of elements in the real instance set may not reach K; potentially incorrect predictions may be retained because the search for the optimal situation will preserve even if the optimal solution is not correct. Therefore, this invention adopts the following strategy:

[0059] First, edges are connected between elements in different sets, with a threshold value θ set. An edge is only connected if the IoU between the rectangles corresponding to the two elements is greater than this threshold value; otherwise, the edge is not connected. In particular, for the background class in the prediction set, there is no need to calculate the IoU, and the edge connection step is skipped directly.

[0060] Then, for each connected edge, its weight is set to the weighted sum of the classification loss and regression loss between two (ci, li) pairs, that is, the traditional loss function calculation method is used.

[0061] Finally, we search for pairs of matching elements in these two sets, i.e., pairs of elements with edges, such that the sum of the total edge weights is minimized while maximizing the number of matching pairs. This sum of edge weights is the minimum matching loss function. Therefore, this problem becomes a minimum weight matching problem in a bipartite graph.

[0062] The threshold setting fundamentally eliminates obviously unreasonable erroneous predictions and also solves the problem of empty predictions that may exist in the prediction set. The minimum weight matching does not require that the number of elements in the two sets be equal, so it avoids the problem that the number of elements in the real instance set may not reach K. It ensures that, in the case of overlap, as many fruits as possible can be detected (i.e., the false negative rate is minimized), thereby minimizing the classification error (i.e., reducing the false positive rate).

[0063] The minimum weight matching problem in a bipartite graph can be transformed into a minimum cost maximum flow problem in network flow. This invention adopts this approach and uses the highly efficient SPFA algorithm for solution. By incorporating the minimum matching loss function into the overall loss function, the model can better learn how to accurately predict the location and category of fruits during training, thereby improving detection accuracy.

[0064] As mentioned earlier, the minimum matching loss function is used to measure the difference between the predicted instance set and the real instance set. For the predicted instance set... and real instance collection The minimum matching loss function is calculated based on minimum weight matching in a bipartite graph, transforming it into a minimum cost maximum flow problem in network flow. This is achieved by constructing a bipartite graph by adding edges, establishing a super source and super sink, using the original edge weights as costs, and setting the edge capacity to 1, thus transforming it into a minimum cost maximum flow problem. Represents a prediction instance With real-world examples The cost between them is calculated by using a weighted sum of traditional classification and regression losses, and then the problem is solved to find the matching scheme that minimizes the cost.

[0065] The classification loss uses the cross-entropy loss function to measure the accuracy of the model's predictions of fruit categories. For each predicted instance, its category prediction can be represented as a probability distribution, while the true category is a one-hot encoded vector. The cross-entropy loss function effectively measures the difference between the predicted probability distribution and the true category, prompting the model to learn accurate category features. Let the predicted category probability distribution be... The real category is Then the cross-entropy loss The calculation formula is:

[0066]

[0067] The regression loss uses the EIoU loss function to measure the model's accuracy in predicting fruit locations. The bounding box regression error is decomposed into three independent and directly optimizable components: overlap error, center point distance error, and width-to-height difference error. The absolute differences in width and height are penalized directly, allowing the model to fit the geometric features of the true bounding box more accurately and quickly. The formula for calculating this regression loss is as follows:

[0068]

[0069] The meanings of each parameter are as follows:

[0070] Prediction box With real frame The intersection-union ratio measures the degree of overlap between two entities;

[0071] The square of the Euclidean distance, used to quantify differences in coordinates or dimensions;

[0072] , : These are the center coordinates of the predicted bounding box and the ground truth bounding box, respectively;

[0073] , , , : These represent the width and height of the predicted bounding box and the ground truth bounding box, respectively;

[0074] : The diagonal length of the smallest closed region enclosing the predicted bounding box and the ground truth bounding box, used to normalize the center point distance;

[0075] , : The width and height of the smallest closed region enclosing the predicted bounding box and the ground truth bounding box, used to normalize the width and height difference.

[0076] Ultimately, the weight of each edge is also the cost of each edge after transforming it into a network flow problem. From regression loss and classification loss We get the weighted sum, that is

[0077]

[0078] in These are weighting coefficients used to balance the importance of regression and classification losses. They are adjusted experimentally. The value was found when At this time, the model achieves good performance in both detection precision and recall. This loss definition allows the model to focus on both the location and category information of instances during training, thereby improving the accuracy of fruit detection.

[0079] Deduplication NMS Post-processing: Deduplication NMS post-processing is used to remove duplicate instances from multi-instance prediction results, improving the accuracy and reliability of detection results. In fruit detection, since the multi-instance prediction module generates multiple prediction instances, there may be a large number of duplicate predictions. These duplicate predictions not only increase the computational load but also affect the accuracy of the detection results. The deduplication NMS post-processing module processes the set of prediction instances, retaining the most representative instances and removing those that highly overlap with other instances and have low confidence. Its implementation method differs from traditional NMS, which processes based on a single prediction box, while deduplication NMS processes based on an instance set. Specifically, traditional NMS first sorts the instance set according to the confidence of the prediction instances, placing instances with high confidence at the top. Then, it iterates through the sorted instance set, and for each instance, calculates its overlap with the retained instances (which can be measured using IoU, i.e., Intersection over Union). If the overlap between the instance and the retained instances exceeds a certain threshold and the confidence is lower than that of the retained instances, the instance is deleted. However, considering the multi-instance prediction mechanism and the characteristics of the loss function in this design, predictions from the same proposal will inevitably be different. This is determined by the definition of bipartite graph matching in minimum loss, where matching strictly requires a one-to-one relationship, and there is no many-to-many relationship. Therefore, this design will use the following method when calculating the overlap between each instance and the retained instances:

[0080] First, sort the instance set according to the confidence level of the predicted instances, and put the instances with higher confidence levels first;

[0081] Then, the sorted set of instances is traversed sequentially to determine whether the current instance and the instance retained in the previous stage come from the same proposal prediction. If they do, the instance is skipped directly. Otherwise, it is processed in the traditional way, that is, the overlap between the current instance and the instance retained in the previous stage is calculated. If the overlap exceeds the set threshold and the confidence level is lower than that of the instance retained in the previous stage, the current instance is deleted. Otherwise, the current instance is retained.

[0082] After inserting a judgment logic to determine whether they come from the same proposal, deduplication NMS can more effectively remove duplicate instances and obtain the final detection result, thus better adapting to the situation of highly overlapping instances in fruit detection.

[0083] During model training, the choice of optimizer is crucial to the model's convergence speed and performance. Common optimizers include Adam and SGD (Stochastic Gradient Descent), each with different characteristics and applicable scenarios.

[0084] SGD is a classic optimizer whose core idea is to update parameters by calculating the negative gradient direction of the current parameters. The mathematical formula is:

[0085]

[0086] in It's the learning rate. This is the gradient of the loss function. SGD is simple and efficient, performing well when dealing with mini-batch data or convex optimization problems. However, in non-convex optimization problems, SGD is prone to getting trapped in local minima or exhibiting oscillations, leading to slower convergence. Furthermore, the learning rate of SGD needs to be manually fine-tuned; a fixed learning rate may result in slow convergence in the early stages or oscillations in later stages.

[0087] The Adam (Adaptive Moment Estimation) optimizer combines the ideas of momentum estimation and adaptive learning rate, and is one of the most widely used optimizers in deep learning. Its core formula includes first-order moment estimation:

[0088]

[0089] Second-order moment estimation:

[0090]

[0091] And the parameter update formula:

[0092]

[0093] in and Typically, values ​​of 0.9 and 0.999 are used. The momentum term is a smoothing term. Adam's advantage lies in its ability to adaptively adjust the learning rate, with a larger learning rate when gradients are sparse and a smaller learning rate when gradients are dense, thus maintaining good update performance across different parameter dimensions. The introduction of the momentum term also helps Adam mitigate the problem of abrupt gradient direction changes, accelerating convergence and reducing oscillations.

[0094] After comparative experiments, this study selected Adam as the optimizer. In training the fruit detection model, due to the diversity of data and the complexity of the model, an optimizer that can adaptively adjust the learning rate, converge quickly, and is robust to hyperparameters is needed. Adam's adaptive learning rate feature can automatically adjust the learning rate according to the gradient of different parameter dimensions, avoiding the tedious process of manually adjusting the learning rate, and also better adapting to the complex data distribution in the fruit detection task. The momentum term helps accelerate model convergence, reduce oscillations during training, and improve training stability. In the early stages of training, Adam can quickly adjust parameters, guiding the model towards the optimal solution; in the later stages of training, the adaptive learning rate can prevent the model from missing the optimal solution due to an excessively large learning rate, thereby improving the final performance of the model. Adam's learning rate was set to 0.001. It is 0.9. It is 0.999. With parameters of 1e - 8, these parameters demonstrated good performance in the experiment, enabling the model to achieve high detection accuracy within a reasonable training time.

[0095] S6. Model Update: Update the model parameters in reverse based on the calculated value of the loss function;

[0096] The selection of hyperparameters has a significant impact on model performance. In order to find the optimal combination of hyperparameters, this study adopts a method combining cross-validation and grid search.

[0097] Cross-validation is an effective method for evaluating model performance and selecting hyperparameters. The dataset is divided into multiple subsets, typically K subsets (K-fold cross-validation). In each training iteration, one subset is selected as the validation set, and the remaining subsets are used as the training set. Through multiple training and validation iterations, performance metrics such as accuracy, recall, and F1 score are obtained for different hyperparameter combinations on the validation set. These performance metrics are then averaged to obtain the average performance of that hyperparameter combination, thus providing a more comprehensive evaluation of the impact of hyperparameters on model performance.

[0098] Grid search is an exhaustive method for searching hyperparameter combinations. For each hyperparameter that needs adjustment, a range and step size are defined. For the learning rate, the range can be defined as [0.0001, 0.01] with a step size of 0.0001; for the batch size, the range can be defined as [16, 64] with a step size of 16. Then, all possible combinations of these hyperparameters are trained and evaluated. Based on the average performance metric obtained from cross-validation, the hyperparameter combination with the best performance is selected as the final hyperparameter settings.

[0099] When adjusting the learning rate, it was found that a small learning rate (e.g., 0.0001) resulted in slow model convergence and excessively long training time; while a large learning rate (e.g., 0.01) caused the model to oscillate during training and fail to converge to a satisfactory result. After multiple experiments, it was determined that a learning rate of 0.001 enabled the model to converge within a reasonable time and achieve good performance on the validation set.

[0100] Regarding batch size adjustments, smaller batch sizes (e.g., 16) can make the model training process less stable and more susceptible to noise; larger batch sizes (e.g., 64) can improve training stability, but may lead to insufficient memory or inaccurate gradient calculations. Ultimately, a batch size of 32 was determined, achieving a good balance between training stability and computational efficiency.

[0101] By employing cross-validation and grid search, the hyperparameters such as learning rate and batch size were meticulously adjusted, enabling the model to achieve optimal performance in the fruit detection task.

[0102] In summary, the training process of the fruit detection model mainly includes steps such as data loading, model forward propagation, loss calculation, and backpropagation to update parameters.

[0103] During the data loading phase, a data loader is used to read data from the preprocessed fruit dataset. The data loader loads image data and corresponding annotation information (fruit category, location, etc.) into memory according to a set batch size and performs necessary preprocessing operations, such as normalization and data augmentation. Data augmentation methods such as random cropping, rotation, and scaling are used to increase data diversity and improve the model's generalization ability. The data loader also arranges the data in a specific order to ensure even access to all data during training.

[0104] During the forward propagation phase of the model, the loaded image data is input into the constructed fruit detection model. The model first extracts features from the image using a feature extraction network, obtaining feature maps at different scales. These feature maps contain rich feature information about the fruit, from low-level detailed features to high-level semantic features. Based on these feature maps, the multi-instance prediction module, through an additional prediction branch, predicts multiple instances for each proposal, outputting a set of predicted instances, each containing category and location information.

[0105] In the loss calculation phase, the loss function is calculated based on the predicted instance set and the real instance set. As mentioned earlier, a network is first constructed by adding edges, and the cost of each edge is obtained by a weighted sum of the regression loss and the classification loss. Then, the SPFA algorithm is used to solve for the minimum overall loss. This loss value reflects the difference between the model's current prediction results and the actual situation, and serves as the basis for updating the model's parameters.

[0106] In the backpropagation parameter update phase, based on the calculated loss value, the gradient of the loss function with respect to the model parameters is calculated using the backpropagation algorithm. The backpropagation algorithm is based on the chain rule, starting with the loss function and progressively calculating the gradient of each parameter. After calculating the gradient, the selected optimizer (such as Adam) updates the model parameters based on the gradient information. The optimizer adjusts the update step size and direction of the parameters according to the hyperparameter settings (such as learning rate, momentum, etc.), causing the model to update parameters in the direction that reduces the loss function, thereby gradually improving the model's performance. A parameter update is performed after each training batch. As training progresses, the model gradually converges to a better state, capable of accurately detecting the category and location of fruits.

[0107] To prevent model overfitting and accelerate model convergence, this invention employs optimization strategies such as early stopping and gradient pruning.

[0108] Early stopping is a common method to prevent model overfitting. During training, the model's loss on the training set usually decreases as training progresses, but the loss on the validation set may start to rise after a certain point, indicating that the model is beginning to overfit. Early stopping monitors performance metrics (such as loss and accuracy) on the validation set. When the performance on the validation set no longer improves within a certain number of training epochs, training is stopped, and the model parameters at this point are saved. Specifically, a counter is set; when the performance on the validation set does not improve, the counter is incremented by 1; when the counter exceeds a certain threshold (e.g., 10 epochs), the model is considered overfitted, and training is stopped. Early stopping avoids overfitting on the training set, improves the model's generalization ability, and enables it to better detect fruit in practical applications.

[0109] Gradient clipping is used to prevent gradient explosion and ensure the stability of model training. In deep learning, excessively large gradient values ​​can lead to excessively large parameter updates, causing the model to fail to converge or even diverge. Gradient clipping limits the magnitude of the gradient, keeping it within a reasonable range. Specifically, it calculates the norm of the gradient (e.g., L2 norm). If the gradient norm exceeds a set threshold (e.g., 5.0), the gradient is scaled down to make its norm equal to the threshold. Gradient clipping effectively avoids gradient explosion, making the model more stable during training, accelerating the model's convergence speed, and thus improving the training effect and performance of fruit detection models.

[0110] In fruit testing experiments, the performance of the model was comprehensively evaluated using metrics such as mean precision (mAP), recall, and precision.

[0111] mAP is a commonly used evaluation metric in object detection tasks. It comprehensively considers the model's precision at different recall rates, providing a more complete reflection of the model's detection performance. The specific calculation method is as follows: First, for each fruit category, calculate its precision value at different recall rates. Then, average these precision values ​​to obtain the average precision (AP) for that category. Averaging the AP values ​​for all fruit categories yields the mAP. In this experiment, the Intersection over Union (IoU) threshold used in calculating mAP ranged from 0.5 to 0.95, with a step size of 0.05. This is a commonly used evaluation metric in object detection, providing a more rigorous assessment of the model's performance at different IoU thresholds.

[0112] Recall represents the ratio of the number of fruits correctly detected by the model to the actual number of fruits. The formula is: Recall = TP / (TP + FN), where TP represents true positives (the number of fruits correctly detected by the model) and FN represents false negatives (the number of fruits that actually exist but were not detected by the model). Recall reflects the comprehensiveness of the model's fruit detection; a higher recall indicates fewer missed fruits.

[0113] Accuracy represents the ratio of the number of fruits correctly detected by the model to the total number of fruits detected by the model. The formula is: Precision = TP / (TP + FP), where FP represents false positives, i.e., the number of fruits incorrectly detected by the model. Accuracy reflects the accuracy of the model's detection results; the higher the accuracy, the fewer false positives the model has.

[0114] Experimental testing showed that the fruit detection model based on the multi-instance prediction method proposed in this invention achieved an mAP of 0.85, a recall of 0.82, and a precision of 0.87. Compared with Faster R-CNN, the mAP was improved by 0.08, the recall by 0.06, and the precision by 0.05; compared with YOLOv5, the mAP was improved by 0.12, the recall by 0.09, and the precision by 0.07. These data indicate that this embodiment significantly improves the precision, recall, and accuracy of fruit detection, enabling more accurate and comprehensive fruit detection.

[0115] Analysis of the experimental results demonstrates that the fruit detection model based on the multi-instance prediction method proposed in this invention exhibits significant advantages in complex scenarios. When faced with densely distributed and occluded fruits, the multi-instance prediction module can predict a set of potentially highly overlapping instances, effectively improving the recall and accuracy of detection and reducing false negatives and missed detections. The introduction of the minimum matching loss function and the deduplication NMS post-processing module further optimizes the model's detection performance, enabling it to more accurately locate and identify fruits.

[0116] Finally, it should be noted that the above control method can be converted into software program instructions. It can be implemented using a control system including a processor and memory, or it can be implemented using computer instructions stored in a non-transitory computer-readable storage medium. The integrated unit implemented as a software functional unit can be stored in a computer-readable storage medium. This software functional unit, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0117] In summary, the machine vision fruit detection method and system based on instance sets described above have the following beneficial effects:

[0118] (1) This invention improves the original network structure by adding multiple new prediction heads at its tail to modify the definition of each proposal in the prediction process. It proposes a multi-instance prediction idea, which allows each proposal to predict a set of potentially highly overlapping instances. Instead of predicting a single target, it predicts the set of related basic ground truth instances. Compared with the traditional single-instance prediction framework, the new scheme can greatly simplify the learning process in dense scenes, reduce missed detections caused by occlusion, and improve detection accuracy and robustness.

[0119] (2) The present invention targets a set of potentially highly overlapping instances and removes duplicate instances by calculating the overlap between each instance and the retained instances, thereby better adapting to the situation of highly overlapping instances in fruit detection; and based on the principle that predictions from the same proposal must be different, it adds a judgment logic on whether the current instance and the instance retained in the previous stage come from the same proposal, thereby reducing the overlap calculation.

[0120] (3) In this invention, for a set of potentially highly overlapping instances, the minimum matching loss function of the classification loss function and the regression loss function is combined to transform the connection between the elements of different sets that meet the threshold value into the minimum weight matching of the bipartite graph. This avoids the problem that the number of elements in the real instance set may not reach K. It ensures that each fruit can be detected as much as possible in the case of overlap, thereby minimizing the classification error and reducing the false detection rate. The threshold value is set to eliminate obviously unreasonable false predictions and solve the problem of possible empty predictions, thereby further improving the detection accuracy and robustness.

[0121] (4) This invention expands the dataset and improves the generalization ability of the model by using image enhancement technology, so that the model can learn the features of fruit under different rotation angles, different sizes, different local regions, and noise interference, and improve the adaptability to changes in fruit angle, size, partial occlusion, and noise interference, thereby improving the detection accuracy and robustness of the model.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit it; although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A machine vision fruit detection method based on an instance set, characterized in that, Includes the following steps: S1. Data Acquisition: Acquire images of fruits under different lighting conditions; S2. Data annotation: The fruit image is annotated with bounding boxes. A rectangular annotation box is drawn for each visible fruit part. The annotation information indicates the occlusion relationship between fruits and classifies and marks the variety and maturity of the fruits. S3, Data Preprocessing; S4. Input the preprocessed image data into the feature extraction model and output the predicted set of instances: The feature extraction model uses FPN with RoIAlign as the baseline detector as the basic network architecture. After the RoI branch, connect K-1 parallel prediction heads. Each prediction head consists of two fully connected layers to generate a prediction set of less than or equal to K instances corresponding to a proposal. S5. Model Training: Using the minimum matching loss function, which combines the classification loss function and the regression loss function, as the loss function of the feature extraction model, the optimal solution is obtained by solving the predicted instance set; the minimum matching loss function includes: First, edges are connected between elements in different sets, with a threshold value θ set. An edge is only connected if the IoU between the rectangles corresponding to the two elements is greater than the threshold value; otherwise, the edge is not connected. Then, for each connected edge, the edge weight is set to the weighted sum of the classification loss and regression loss between two (ci, li) pairs, where ci represents the class label with confidence and li represents the relative coordinates of the target instance, i=1, 2, 3, ..., K; Finally, we search for pairs of matching elements in these two sets, that is, pairs of elements with edges, such that the sum of the total edge weights is minimized while maximizing the number of matching pairs. This sum of edge weights is the minimum matching loss function. S6. Model Update: Update the model parameters in reverse based on the calculated value of the loss function.

2. The machine vision fruit detection method based on instance sets according to claim 1, characterized in that, The classification loss uses the cross-entropy loss function, the regression loss uses the EIoU loss function, and the edge weights are calculated using the following formula: in, These are the weighting coefficients. To regress the loss, For classifying losses, For the predicted class probability distribution, The true category is represented by the bounding box (IOU). With real frame The intersection and union ratio, The square of the Euclidean distance. , These are the width and height of the prediction box, respectively. , These are the width and height of the actual bounding box, respectively. , These are the width and height of the minimum closed region that encloses the predicted bounding box and the ground truth bounding box, respectively. cdot is the length of the diagonal of the smallest closed region that encloses the predicted bounding box and the ground truth bounding box.

3. The machine vision fruit detection method based on instance sets according to claim 1, characterized in that, In the process of connecting elements between different sets, for the background class in the prediction set, there is no need to calculate IoU, and the step of connecting the edges can be skipped directly.

4. The machine vision fruit detection method based on instance sets according to claim 1, characterized in that, S5 also includes: further optimizing the optimal solution by using a deduplication NMS post-processing module.

5. The machine vision fruit detection method based on instance sets according to claim 4, characterized in that, The deduplication NMS post-processing module includes: First, sort the instance set according to the confidence level of the predicted instances, and put the instances with higher confidence levels first; Then, the sorted set of instances is traversed sequentially to determine whether the current instance and the instance retained in the previous stage come from the same proposal prediction. If they do, the instance is skipped directly. Otherwise, it is processed in the traditional way, that is, the overlap between the current instance and the instance retained in the previous stage is calculated. If the overlap exceeds the set threshold and the confidence level is lower than that of the instance retained in the previous stage, the current instance is deleted. Otherwise, the current instance is retained.

6. The machine vision fruit detection method based on instance sets according to claim 1, characterized in that, In S4, the value of K is 2.

7. The machine vision fruit detection method based on instance sets according to claim 1, characterized in that, S6 includes: adjusting the hyperparameters using cross-validation and grid search methods.

8. The machine vision fruit detection method based on instance sets according to claim 1, characterized in that, S3 includes: processing the labeled data using image enhancement techniques, and then performing data normalization processing. The image enhancement techniques include rotation, scaling, cropping, and adding noise.

9. A machine vision fruit inspection system based on an instance set, characterized in that, include: At least one processor; and at least one memory communicatively connected to the processor, wherein: The memory stores program instructions that can be executed by the processor, and the processor can invoke the program instructions to perform the method as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the method as described in any one of claims 1 to 8.

Images