Heterogeneous convolution and agency attention fused multi-task apple quality grading detection method and device

Abstract

The invention relates to the field of computer vision, and provides a heterogeneous convolution and agency attention fused multi-task apple quality grading detection method and device. The method specifically comprises the steps of data acquisition and preprocessing; marking pictures in the data set, dividing the pictures into a training set, a verification set and a test set, and performing image enhancement on the training set; establishing a model, selecting a proper algorithm model as a framework, and performing subsequent modification, so that the model is more suitable for multi-task detection of the Apple; using the training set in the data set as the input of the network model to perform multi-round training until the model converges, and storing the model with the highest accuracy on the verification set in the training process as a final model; calculating the size of the apple by using a method for converting the size based on a pixel unit into the size of the real world; and performing grade judgment on the apples according to the combination of the detected maturity and scar disease conditions and the calculated sizes, and outputting a final apple quality grading result.

Description

Technical Field

[0001] The present invention relates to a multi-task detection method and device for apple maturity recognition, size calculation, and scar or pest detection based on deep learning, belonging to the technical field of computer vision. Background Art

[0002] In terms of apple picking and post-harvest processing, traditional grading relies mainly on manual picking and visual inspection, which has prominent problems such as large subjective judgment bias and high labor intensity. In recent years, machine vision-based inspection technology has begun to be applied to the agricultural product field, but existing solutions have shortcomings:

[0003] Feature engineering relies on manual design and has poor adaptability to complex field environments (complex occlusion environments lead to insufficient recognition accuracy); single-task models find it difficult to achieve multi-dimensional feature fusion analysis of maturity, scars, pests and diseases, etc.; there is a lack of a linkage mechanism between quality grade classification and harvesting decisions, and it is impossible to build a closed-loop "detection-prediction-execution" system.

[0004] With the development of deep learning, researchers have applied convolutional neural networks to agricultural products, leveraging their advantages in self-extraction of features to improve detection accuracy and speed. However, in real apple orchards, fruit is severely obscured by leaves and branches, as well as by each other. This reduces detection accuracy, making further quality assessments such as maturity identification and scar detection impossible. Summary of the Invention

[0005] To improve the accuracy of apple detection and thus complete apple quality grading, a method for categorizing quality using maturity, scar, and pest detection is proposed. A dataset synthesis method for occluded apple detection is used to synthesize a large number of occluded apple images, enhancing the model's detection accuracy and robustness. A new model architecture based on YOLO v11 is proposed, enabling multi-task scalability. A dataset annotation method for the maturity of apples of different color varieties is proposed, which can better detect and identify apple maturity. A multi-task method is proposed for apple quality grading using apple maturity classification, scar and disease detection, and apple size.

[0006] The specific solution of the present invention to solve the above technical problems includes the following steps:

[0007] A method for identifying apples in a complex orchard environment by using computer vision technology, and then further completing the grading of apples, includes: S1, data collection and preprocessing; S2, labeling the pictures in the data set, obtaining the labeled images of apples in the orchard and dividing them into training set, validation set and test set, and performing image enhancement on the training set pictures; S3, establishing a model, selecting a suitable algorithm model as a framework and performing subsequent modifications to make the model more suitable for multi-task detection of apples; S4, using the training set in the data set as the input of the network model for multiple rounds of training until the model converges, and saving the model with the highest accuracy on the validation set during the training process as the final model; S5, using a method based on pixel-based size conversion to real-world size to calculate the size of the apple; S6, establishing a front-end UI interface to enable users to input content through the UI interface and obtain recognition results and quality grading results; S7, designing a dedicated real-time detection device for this technical solution.

[0008] In some embodiments of the present invention, step S1 includes constructing a dataset of 7,000 images of different varieties of ripe red and yellow apples. The images are then preprocessed to remove blurry, duplicate, and unidentified images, and all remaining images are cropped to a standard 640×640 image size.

[0009] Step S2 includes: S21, labeling the dataset according to the invention's objectives, indicating the maturity level of the target apples and the presence or absence of scars or diseases; S22, dividing the dataset into a training set, a validation set, and a test set in a ratio of 8:1:1. Using a synthetic occlusion dataset method, images of occluded apples are added to the training set to enhance recognition of occluded apples; and S23, performing image enhancement on the training set images using various methods.

[0010] In the step S21, the purpose of the invention is to grade the quality of apples, and to judge according to the maturity of the apples and whether there are scars or diseases on the surface of the apples. The data set images need to be annotated with both maturity and scar diseases. The maturity annotation rule is to judge according to the proportion of the color on the surface of the apples and the fullness of the surface color of the fruit. Before annotation, the apples in the image are first segmented using a segmentation model to obtain a segmented mask. The mask is applied to the original image to obtain an RGB image containing only apples, and the RGB image is converted into an HSV image. The hue (Hue), saturation (Saturation) and lightness (Value) can be obtained according to the HSV image, and the color distribution can be obtained according to the hue, and then the color proportion can be obtained. The saturation and lightness are then processed to obtain the saturation of the color.

[0011] According to the maturity labeling rules, varieties that are red when mature: if the red proportion is less than 20% or the color saturation is less than 0.3, they are labeled as: redApple_1; if the red proportion reaches 20%-40% and the color saturation is between 0.3-0.45, they are labeled as: redApple_2; if the red proportion reaches 40%-60% and the color saturation is between 0.45-0.6, they are labeled as redApple_3; if the red proportion reaches 60%-80% and the color saturation is between 0.6-0.8, they are labeled as: redApple_4; if the red proportion reaches more than 80% and the color saturation is greater than 0.8, they are labeled as: redApple_5; for varieties that are yellow when mature, they are labeled as yellowApple_1, yellowApple_2, yellowApple_3, yellowApple_4, and yellowApple_5 according to the above rules.

[0012] The scar disease labeling rule is to classify whether the apple has defects based on whether there are scars or rot on the surface of the apple in the picture. If there are scars or rots, it is marked as damaged, and if there are no scars or rots, it is marked as fresh.

[0013] The dataset was annotated according to the above rules, using the x-anylabeling annotation tool for semi-automatic annotation. First, 500 images were manually annotated using the annotation tool. These 500 annotated images served as the dataset for training a model. After training, the model was saved. The remaining images in the dataset were trained using the model. The resulting annotations were manually adjusted. After 1,000 images were adjusted, they were added to the dataset for training the annotation model. This iterative cycle continued until the annotations were free of obvious errors.

[0014] In step S22, the dataset of the occluded apple is synthesized to obtain a mask of the original image to obtain the specific location of the target apple in the image. The Grounding-DINO model is used to detect the corresponding occluders in the corresponding image, such as leaves, branches, etc.; the SAM model is then used to segment the occluders to obtain the mask of the occluders. The occluders are covered on the specific location of the apple in the original image to obtain the required occluded apple image. The synthesized occluded apple image is placed in the training set to enhance the model's performance in detecting occluded targets and improve the robustness of the model.

[0015] In step S23, data augmentation is performed using random cropping, random flipping, and color jittering. Random cropping enables the model to learn different local features, improving robustness; random flipping increases the diversity of training data; and color jittering prevents the model from over-reliance on color information by randomly adjusting brightness, contrast, saturation, and hue.

[0016] In step S3, a model is established, using Yolo V11 as the basic framework. This includes a backbone network that extracts features from the input image. The backbone network uses multiple C3k2 modules to process feature extraction at different stages. C3k2 modules at different locations and the C2PSA module at the end of the backbone network then feed the extracted features at different scales into the neck modules at different locations. The neck modules are responsible for multi-scale feature fusion. By fusing feature maps from different stages of the backbone network, they enhance feature representation and optimize feature transfer. Finally, the information is fed into the detection head for processing, which outputs the final result.

[0017] The C3k2 module is an important feature extraction component in the model. It is one of the improvements of Yolo V11 compared to the previously proposed network structure. C3k2 introduces the multi-scale convolution kernel C3K. When the hyperparameter c3k=True, the bottleneck block is replaced by C3k, otherwise it is still the Bottleneck module.

[0018] In step S3, the model is modified by replacing the mid- and high-level C3k2 modules in the backbone network with the Inception Depthwise Convolution (IDConv) module; replacing the feature fusion module in the neck network with the Semantics and Detail Infusion (SDI) module; and adding the Agention attention mechanism before the detection head.

[0019] IDConv decomposes a large-kernel depthwise separable convolution into multiple parallel branches, including a small square kernel branch, an orthogonal band kernel branch, and an identity mapping branch. Leveraging the residual principle, it preserves information in some channels, preventing information loss. This convolutional module includes four branches with convolutional kernels of different scales for feature extraction. By replacing the high-level C3k2 modules in the backbone network, this multi-branch structure captures more complex semantic information, enabling multi-scale feature extraction and improving representation capabilities. The module utilizes depthwise separable convolution, which improves computational efficiency.

[0020] The SDI module is a semantic detail attention module. Its core concept is to better integrate the semantic information in high-level features with the detail information in low-level features by leveraging the complementarity of semantic and detail information and the Hadamard product, thereby obtaining a feature map that contains both semantic and detail information. The Hadamard product uses element-wise multiplication to fuse two feature maps into a single feature map.

[0021] The agent attention mechanism innovatively introduces an additional set of attention triplets (Q, K, V) with the token A, generating a proxy attention represented as a quadruple (Q, A, K, V). This additional set of proxy tokens A is incorporated into the traditional attention module. Compared to traditional attention mechanisms, agent attention boasts stronger task differentiation and cross-task feature synergy, significantly improving the model's multi-task performance while maintaining computational efficiency.

[0022] The C2PSA module is an advanced module in YOLO V11 for enhancing feature extraction. It embeds a multi-head attention mechanism and a feedforward neural network. The embedded PSA module implements a module with an attention mechanism. The introduction of PSA aims to enhance feature extraction capabilities through the multi-head attention mechanism and feedforward neural network.

[0023] The PSA module is a module based on the attention mechanism. The PSA module forms a pyramid-shaped feature map by splicing the convolution results of convolution kernels of different sizes, and then applies the attention mechanism on this feature map to extract richer feature information.

[0024] Furthermore, the PSA module is:

[0025] First, we perform multi-scale feature extraction, or SPC. Assuming the input tensor is X, we divide X into S groups. Large convolution kernels increase computational complexity, so we perform group convolution on each group. The specific number of groups is G = exp(2,(k-1) / 2), which is 2 raised to the power of (k-1) / 2. We then use convolution kernels of different sizes to extract features of different scales for these groups, and then concatenate the extracted multi-scale features. This process can be described as follows:

[0026]

[0027] Then the output of the SPC module is passed through the SE Weight Module to obtain the channel attention value. The formula is:

[0028]

[0029] The attention vectors Zi are concatenated and normalized to finally obtain the features after the attention mechanism:

[0030]

[0031] In step S3, the model is modified to make it more suitable for multi-task detection. The head network is modified to add two independent branches, one for maturity detection and the other for scar disease detection. These two branches share features extracted by the backbone and neck networks. New loss functions are used for each branch, with Focal loss used for the maturity branch. This technical solution labels scar disease detection with two labels, and uses a binary cross-entropy loss function for all scar disease branches. Uncertainty dynamic weighting is used to dynamically balance the importance of different tasks.

[0032]

[0033] Among them, Loss ripeness is the loss function calculation of the maturity branch, α t Is a balancing factor used to adjust the impact between positive and negative samples, p t is the model's predicted probability for the category, γ is the focus factor used to adjust the weight of difficult and easy samples; Loss damaged is the loss function calculation of the scar disease branch; Loss total is the calculation method of the total loss, where σ is a learnable parameter that should represent the uncertainty of the prediction of each task and is used to dynamically adjust the task weights.

[0034] In step S4, the images in the dataset are imported into the established model, and the corresponding parameters and parameter update methods are set for training. After each training round, the accuracy of the current model is calculated using the validation set, and the parameters are updated using the gradient descent method. The number of training rounds is set to 200, and the training process is continuously repeated until the model converges. After training is completed, the model with the highest accuracy on the validation set is saved as the final model.

[0035] The images in the test set are imported into the final model for testing, and the generalization ability of the final model is verified by testing it on unseen images.

[0036] In step S5, the size of the apple is estimated. During real-time detection, the size in pixels is converted into the real size in the real world based on the target-object distance and camera parameters based on the ordinary 2D image.

[0037]

[0038] Where X and Y are the estimated real-world length and width of the apple, x and y are the length and width of the rectangular bounding box of the apple detected in the image, f is the focal length of the camera, and Z is the horizontal distance from the camera to the actual object. Apple size is typically based on its transverse diameter (maximum cross-sectional diameter), so the longer of the calculated length and width is used as the transverse diameter.

[0039] Combining the completed tasks of apple maturity identification, size, and scar disease detection, apples are graded based on these three quality indicators, with four levels: special grade, first grade, second grade, and immature. When the maturity identification result is redApple_5 or yellowApple_5, the scar disease detection result is fresh, and the apple's horizontal diameter is greater than 80mm, the quality is classified as special grade. When the maturity identification result is redApple_4 or yellowApple_4, the scar disease detection result is fresh, and the size is 65mm-80mm, the quality is classified as first grade. When the scar disease detection result is damaged, the quality is classified as second grade regardless of the results of other tasks. When the maturity identification result is redApple_1-3 or yellowApple_1-3, and the scar disease detection result is fresh, the apple is classified as immature.

[0040] In step S6, a front-end UI interface is established, which can perform image detection, video detection, and real-time detection. The detection interface can perform apple recognition, maturity detection, or scar disease detection, and can also classify apple quality grades.

[0041] In order to implement the above technical solution, a special real-time monitoring device is designed.

[0042] The device includes a tripod base, a support plate, an electric telescopic rod, a power supply, an industrial camera, an electric rotating pan / tilt head, and a fill light. The tripod base is fixed to the orchard floor, and the support plate is fixed to the base. A hole is provided in the center of the support plate for inserting the electric telescopic rod into the tripod base. The electric rotating pan / tilt head is fixed to the top of the electric telescopic rod, and the industrial camera with a zoom lens is embedded in the electric rotating pan / tilt head. The power supply includes a solar panel and a battery box; the solar panel is fixed to the support plate, and the battery box is fixed to the bottom of the tripod to increase stability and power the device.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] This technical solution constructs an integrated multi-task model based on the Yolo V11 network framework. In the backbone network, the C3k2 modules at mid- and high-level are replaced with IDConv modules; in the neck network, the Contact module is replaced with SDI modules; an Agent attention mechanism is added; and two new branches are introduced in the head network: a maturity recognition branch and a scar disease detection branch. These branches enable apple object location detection, maturity recognition, and scar disease detection. The three tasks share the same feature extraction backbone network. Based on this modified model, complete multi-dimensional apple information can be simultaneously obtained during the forward propagation, improving the ability to detect local features. Sharing semantic features across multiple tasks helps low-resource labeling tasks benefit from high-resource tasks, alleviating imbalance issues. An uncertainty weighting strategy is employed to dynamically adjust the weights between tasks, eliminating the need for manual parameter tuning and facilitating the expansion of more apple-related tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This application provides a design flow chart for a method for grading apple quality based on maturity, size, scar and disease.

[0046] Figure 2 This is the model network framework diagram.

[0047] Figure 3 Schematic diagram of the C3k2 module structure used for feature extraction.

[0048] Figure 4 Schematic diagram of the PSA module structure of the attention mechanism module.

[0049] Figure 5 Schematic diagram of the SPC unit structure.

[0050] Figure 6 Schematic diagram of the IDConv module structure.

[0051] Figure 7 Schematic diagram of the Agent Attention module structure.

[0052] Figure 8 Diagram of the multi-task detection head framework.

[0053] Figure 9 The effect of converting RGB image to HSV image and performing color extraction.

[0054] Figure 10 This is the effect diagram of apple maturity classification in this application.

[0055] Figure 11 This is a diagram of the device designed in this application. DETAILED DESCRIPTION

[0056] The following specific embodiments describe the present invention in detail with reference to the accompanying drawings. The described specific embodiments are only for illustrating the above technical solutions and are not intended to limit the scope of protection of the claims.

[0057] like Figure 1 As shown, a multi-task solution for apple maturity identification, scar disease detection, and yield estimation, and then apple quality grading, the technical solution of the present invention includes the following specific steps:

[0058] Step 1: Data collection and preprocessing, and the data set is expanded using the synthetic occlusion method:

[0059] We collected images of apples from real orchards using public datasets within the RoboFlow platform or from other user-uploaded public data. We collected images of apples from different varieties, environments, and lighting conditions. A total of 8,000 images were collected, including images of Fuji, Gala, and Golden Delicious apples. Fuji and Gala apples are red when ripe, while Golden Delicious apples are yellow when ripe. These apple images from different datasets were consolidated and then preprocessed. This preprocessing step included removing similar, untargeted, and blurred images, and then cropping the images to a uniform size of 640×640.

[0060] Orchard environments are complex and ever-changing, and fruit is often obscured by branches, other fruit, or even the fruit itself, posing a significant challenge for automatic recognition. Statistics show that under naturally growing conditions, the occlusion rate of apples can reach 30%-50%, severely impacting recognition accuracy. To address this issue, a synthetic occlusion method is used to generate images of occluded apples to augment the dataset. First, a binary mask is generated from the original image to identify the specific location of the apple. The original image is then detected using the Grounding DINO model, a zero-shot model that can assist in detecting objects in images by combining file input. The original image and text related to the occluded objects are fed into the Grounding DINO model to detect the required occluded objects. The SAM model is then used to segment these occluded objects, generating occluded masks. Finally, the mask information is used to reposition the apples at the locations covered by the occluded objects, generating images of the occluded apples and adding them to the dataset.

[0061] Step 2: Develop annotation rules for the corresponding tasks and annotate the dataset. After annotation, divide the dataset into training set, validation set, and test set, and perform image enhancement on the training set images:

[0062] This method of grading apple quality requires obtaining information about its maturity, size, and surface scars and diseases. Therefore, it is necessary to identify the maturity of the apples and detect scar diseases. The maturity level is determined by the color ratio and color richness of the apple surface, and the apple skin is inspected to determine whether there are scar diseases:

[0063] First, use the OpenCV function cvtColor() to convert the RGB images in the dataset into HSV images. Then, calculate the color proportions based on the hue of the HSV image, and the color saturation based on the saturation and value. Apples that mature red are categorized into five maturity levels: red_1, red_2, red_3, red_4, and red_5. Apples that mature yellow are categorized into yellow_1, yellow_2, yellow_3, yellow_4, and yellow_5. For scar diseases, apples with scars, rot, or defects on the skin are labeled "damaged." Otherwise, they are labeled "fresh."

[0064] Use x-anylabeling, a semi-automatic annotation tool, to perform labeling according to the labeling rules. First, manually label 500 apple images with their maturity level and scar disease status. Using these 500 manually labeled images as a dataset, select a model to train and obtain a model suitable for this task. Import the model into the labeling tool x-anylabeling, and use the model to automatically label the remaining dataset images. Manually adjust the 1,000 automatically labeled images and add them to the dataset for training the labeling model and retrain it. Repeat this cycle until the model's labeling results are free of obvious errors, and the labeling is complete. Divide the labeled dataset into training, validation, and test sets in an 8:1:1 ratio.

[0065] An image enhancement module was set up to perform image enhancement operations on the images in the training set to improve model performance and enhance model robustness. This technical solution correlates ripeness judgment with the color of the apple skin. To avoid image color changes caused by image enhancement operations, safe and reliable pixel-based enhancement methods were selected, such as asymmetric brightness adjustment, independent adjustment of color channels, and adaptive histogram equalization. Enhancement operations were performed on the entire image: random cropping, random flipping, and random rotation were used for data enhancement.

[0066] Asymmetric brightness adjustment: By controlling the brightness scaling factor, brightness can be adjusted in only one direction, preventing color distortion caused by low brightness and loss of color characteristics.

[0067] Independent color channel adjustment: Perturbations are added only to non-primary color channels, preserving the primary color features associated with maturity.

[0068] Contrast-limited adaptive histogram equalization (CLAHE): This method enhances local contrast by dividing the image into multiple small blocks and performing histogram equalization on each block. This method can significantly improve the visual quality of the image. To prevent noise in certain areas caused by excessive histogram enhancement, CLAHE uses a contrast limit parameter, clipLimit, to limit the extent of histogram enhancement for each block, thereby preventing noise amplification.

[0069] RandomCrop: enables the model to learn different local features and improve robustness.

[0070] Random Horizontal Flip: Increase the diversity of training data.

[0071] Random Rotation: allows the model to adapt to inputs in different directions.

[0072] Step 3: Select a suitable algorithm model as the framework and modify it to make it more suitable for the task of detecting apple maturity and scar diseases:

[0073] Figure 2-5 This is the Yolo V11 network structure diagram and its module diagram.

[0074] According to one embodiment of the present invention, the Yolo V11 model is selected as the basic framework. The network framework includes a backbone network for feature extraction, which extracts multi-level semantic features from the input image; a neck network that performs further feature extraction and feature fusion based on the information obtained from the backbone network, aggregates features at different levels of the backbone, and enhances multi-scale detection capabilities; and finally, a detection head that converts and further processes the feature information extracted from the backbone network and the neck network, extracting key information such as the target's location, category, and confidence level, so as to ultimately determine the specific situation of the target in the image.

[0075] In step S3, the model is modified by replacing the mid- and high-level C3k2 modules in the backbone network with the Inception Depthwise Convolution (IDConv) module; replacing the feature fusion module in the neck network with the Semantics and Detail Infusion (SDI) module; and adding the Agention attention mechanism before the detection head.

[0076] The modified backbone network consists of multiple convolutional layers, two C3k2 modules, two IDConv modules, one SPPF module, and one C2PSA module. The original RGB image input to the backbone network first undergoes preliminary feature extraction through the convolutional layers, which is equivalent to downsampling. This reduces the input image size from H×W to H / 2×W / 2, extracts low-level features, compresses information, and enhances feature representation by increasing the number of channels. The backbone network contains one C3k2 module and two IDConv modules. These three modules, located in different locations, handle feature extraction at different stages of the backbone. The C3k2 module is a new structure in the YOLO V11 network. The parameter c3k controls the switching between the C3k module and the Bottleneck module as needed. When the parameter is False, the C3k2 module uses the Bottleneck module, degenerating into the C2f module. When the parameter is True, the C3k module is used for extraction. When the output of the convolutional layer reaches the C3k2 module, it first passes through a CBS module, that is, a convolutional layer, a BN layer, and a SiLu activation function to reduce the number of channels of the C3k2 input. Then it passes through n C3k or Bottleneck modules to extract features, and the processed features are combined with the direct channel through the residual structure and output.

[0077] Figure 6 This is the IDConv module structure diagram.

[0078] When passing through the IDConv module, the input X is split along the channel dimension into four groups, corresponding to four branches: the identity mapping branch, the small square kernel branch, and the orthogonal band kernel branch. The small square kernel branch uses 3×3 depthwise separable convolutions for feature extraction; the orthogonal band kernel branch uses 1×k and k×1 depthwise separable convolutions for feature extraction; and the identity mapping branch directly outputs the convolutional kernel using the residual principle. The outputs of the four branches are then concatenated to produce the final feature map. The IDConv module decomposes the large kernel depthwise separable convolution into multiple parallel branches, including the small square kernel branch, the orthogonal band kernel branch, and the identity mapping branch. Using the residual principle, it preserves information in some channels to prevent information loss. This convolutional module contains four branches with convolutional kernels of different scales for feature extraction. Replacing the high-level C3k2 module in the backbone network allows the multi-branch structure to capture more complex semantic information, perform multi-scale feature extraction, and improve representation capabilities. The module utilizes depthwise separable convolutions to improve computational efficiency.

[0079] Finally, the SPPF module and C2PSA module are implemented. The SPPF module uses one CBS block and three consecutive max pooling operations to increase the receptive field, capture multi-scale contextual information, maintain spatial resolution, and fuse multi-scale features. After the SPPF module, the C2PSA module is introduced. The input is split into two parts using a 1×1 convolution. One part is processed by multiple PSAB blocks, which include a multi-head attention mechanism and a feedforward neural network. The processed features are then combined with the original features and output after a 1×1 convolution to restore the number of channels.

[0080] The Neck network adopts the design concept of the feature pyramid structure (FPN) to extract multi-scale features from different levels of the backbone network, and fuses features of different scales through top-down path kernel lateral connections to generate semantically rich and different resolution feature maps for subsequent detection. In the network structure used in the present invention, the neck network receives features extracted from three different positions in the backbone network. The feature maps of the shallower layers have high resolution and contain more position kernel detail information, while the feature maps of the deeper layers have low resolution and contain richer semantic information. The Neck part performs upsampling and downsampling operations on the extracted features, and passes through multiple C3k2 modules to fuse these features of different scales to form a multi-scale feature representation. The use of this feature pyramid structure enables the network to effectively process targets of different scales, and improves the overall performance of the model for the detection of small targets such as apples in pictures.

[0081] Figure 7 This is the SDI module structure diagram.

[0082] The SDI module is used for feature fusion in the neck network. This module is a semantic detail attention module. Its core concept is to leverage the complementarity of semantic and detail information and the Hadamard product to better integrate the semantic information in high-level features with the detail information in low-level features, thereby obtaining a feature map that contains both semantic and detail information. The Hadamard product uses element-wise multiplication to fuse two feature maps into a single feature map.

[0083] like Figure 8 It is a multi-task head module.

[0084] To better meet the multitasking requirements of the present invention and simultaneously output the apple's maturity level and scar disease status, two detection head branches were added: one for maturity recognition and the other for scar disease detection. Different loss functions were designed to calculate the corresponding losses, and dynamic weighting of uncertainty was used to dynamically balance the importance of different tasks when calculating the total loss. These branches share information extracted and processed by the backbone and neck networks, converting this information into the desired output results for the corresponding tasks. For example, the original branch detection head outputs a bounding box; the maturity branch outputs the apple's maturity level; and the scar disease branch outputs the presence or absence of scar disease, and sometimes the location of the scar disease.

[0085]

[0086] Among them, Loss ripeness is the loss function calculation of the maturity branch, α t Is a balancing factor used to adjust the impact between positive and negative samples, p t is the model's predicted probability for the category, γ is the focus factor used to adjust the weight of difficult and easy samples; Loss damaged is the loss function calculation of the scar disease branch; Loss total This is the calculation method for total loss, where σ is a learnable parameter that represents the uncertainty of each task's prediction and is used to dynamically adjust task weights. When the training loss of a task is too large, the model automatically increases the σ value for that task, thereby reducing its contribution to the total loss and, in effect, reducing the weight of that task. When the training loss of a task decreases or stabilizes, the model automatically decreases the σ value, increasing the loss contribution of that task and increasing its weight.

[0087] An agent attention module is added before the detection head. This innovative mechanism introduces an additional set of attention triplets (Q, K, V) representing the token A, generating a proxy attention quadruple (Q, A, K, V). This additional set of proxy tokens A is incorporated into the traditional attention module. Compared to traditional attention mechanisms, agent attention offers stronger task differentiation and cross-task feature collaboration capabilities, significantly improving the model's multi-task performance while maintaining computational efficiency.

[0088] Step 4: Train the model and keep the one with the highest accuracy on the validation set as the optimal model:

[0089] Initialize the established model and import the labeled dataset images into the model for training. Forward propagation is performed with gradients reset to zero. After forward propagation, the loss function for each task is calculated, and backpropagation is performed based on the loss function to update the parameters. After each training round, the current model is validated using the validation set data, and the validation set accuracy is calculated. The training rounds are set to 200, and the training process is repeated until the model converges. After training is complete, the model with the highest validation set accuracy is saved as the final model. The final model is tested using images from the test set, using mean average precision (mAP) and F1 score as evaluation metrics.

[0090] Step 5, calculate the size of the apple:

[0091] The size of the apple is estimated. During real-time detection, the size in pixels is converted to the real size in the real world based on the target-object distance and camera parameters based on ordinary 2D images.

[0092]

[0093] Combining the completed tasks of apple maturity identification, size, and scar disease detection, apples are graded according to these three quality indicators, with four levels: special grade, first grade, second grade, and immature. When the maturity identification result is redApple_5 or yellowApple_5, the scar disease detection result is fresh, and the length or width of the apple fruit is greater than 80mm with a difference of less than 10mm between the length and width, the quality is classified as special grade. When the maturity identification result is redApple_4 or yellowApple_4, the scar disease detection result is fresh, and the size is 65mm-80mm, the quality is classified as first grade. When the scar disease detection result is damaged, the quality is classified as second grade regardless of the results of other tasks. When the maturity identification result is redApple_1-3 or yellowApple_1-3, and the scar disease detection result is fresh, the quality is classified as immature.

[0094] Step 6: Use PyQt5 to design the front-end UI interface. This interface supports image detection, video detection, and real-time detection. The detection interface can be used to identify apples, detect maturity or scar diseases, and classify apple quality.

[0095] The present invention also designs a special real-time detection device for the task of apple quality grading, such as Figure 11 As shown:

[0096] In an optional embodiment, the real-time apple detection device uses a tripod base, a support plate, an electric telescopic rod, a power supply device, an industrial camera, and an electric rotating pan-tilt head. A tripod base 7 is fixed on the ground of the orchard, and a support plate 5 is fixed on the base 7. A hole is set in the center of the support plate 5 for inserting and fixing the electric telescopic rod 3 through the hole. The electric rotating pan-tilt head 2 is fixed to the top of the electric telescopic rod 3, and the electric rotating pan-tilt head 2 can change the angle of the camera 1. The industrial camera 1 with a zoom lens is embedded in the electric rotating pan-tilt head 2 to collect information about apples in the orchard. The power supply device includes a solar panel 4 and a battery box 6; the solar panel 4 is fixed to the support plate 5, and the battery box 6 is fixed to the bottom of the tripod 7 to increase stability and power the device.

[0097] The above describes the various steps of the present invention in a specific order, but it does not mean that the steps must be executed in the above specific order. Some steps can be executed concurrently as long as the required functions can be achieved.

[0098] The above implementation scheme is only a representative technical solution of the present invention and is not intended to limit the scope of protection. Any functional adjustment, equivalent replacement or adaptive improvement based on the core technical solution of the present invention should be deemed to be within the scope of coverage of the claims of this patent. Although the technical features have been explained in detail through diagrams, the application scenarios of the present invention are not limited to the illustrated cases. Technical personnel in related fields can achieve reasonable changes and expansions of the technical solution through different implementation methods based on the technical principles of this patent while maintaining the essence of innovation.

Claims

1. A multi-task apple quality grading method integrating heterogeneous convolution and proxy attention, characterized by: The steps include: S1, data collection and preprocessing; S2. Label the images in the dataset to obtain labeled images of apples in the orchard and divide them into training set, validation set and test set, and perform image enhancement on the training set images; S3. Build a model, select a suitable algorithm model as the framework and make subsequent modifications to make the model more suitable for multi-task detection of apples; S4. The training set in the dataset is used as the input of the network model and multiple rounds of training are performed until the model converges. The model with the highest accuracy on the validation set during the training process is saved as the final model. S5. Calculate the size of the apple using a method that converts pixel-based dimensions into real-world dimensions. S6. Establish a front-end UI interface to enable users to input content through the UI interface and obtain recognition results and quality grading results; S7. Design a dedicated real-time detection device for this technical solution; According to the multi-task apple quality grading method integrating multidimensional features, the method is characterized in that S2 includes: S21, annotating the data set according to the purpose of the invention, converting the RGB image into an HSV image for extracting color information, classifying the apple maturity level according to the hue, saturation and brightness in the HSV image, and annotating the apple maturity information and scar disease information in the image; S22, dividing the data set images into a training set, a validation set and a test set in a ratio of 8:1:1; S23, designing a variety of different methods to perform image enhancement on the training set images, such as random cropping, random flipping, color dithering, etc. According to the multi-task apple quality grading method integrating multi-dimensional features, S3 includes: selecting a basic framework of a Yolo V11 model, the structure of which includes a backbone network for feature extraction, a neck network for multi-scale feature fusion, and a head part for processing the extracted information and outputting the final result; The backbone uses multiple C3k2 modules for feature extraction at different scales. Two IDConv modules replace the C3k2 modules used for feature extraction at the mid- and high-level layers of the backbone network, leveraging a multi-branch structure to capture more complex semantic information. The backbone network also incorporates a C2PSA module with a multi-head attention mechanism, which enhances local feature extraction and improves model performance. The neck network uses the SDI module for feature fusion. This module leverages the complementarity of semantic and detail information and the Hadamard product to better fuse the semantic information in high-level features with the detail information in low-level features, resulting in a feature map that contains both semantic and detail information. The C3k2 module is used during the fusion process for further feature extraction. The head network adds a maturity classification branch and a scar disease detection branch, transforming the model into a multi-task model. An Agent Attention module is added at the junction of the neck and head networks. Compared to traditional attention mechanisms, Agent Attention has stronger task differentiation and cross-task feature collaboration capabilities, significantly improving the model's multi-task performance while maintaining computational efficiency. New loss functions are used for each of the two additional branches: According to the multi-task apple quality grading method integrating multi-dimensional features and the established model, it is characterized in that images in a data set are imported into the established model and parameters and parameter update methods are set for training, the training process is continuously iterated, and the trained model is evaluated using a validation set during the iterative process until the model converges, and the model with the highest accuracy is saved; According to the multi-task apple quality grading method integrating multi-dimensional features, it is characterized in that a certain conversion method is used to convert the size in pixel units into the real size in the real world. The specific conversion process is: Where X, Y are the estimated length and width of the apple in the real world, x, y are the length and width of the rectangular box of the apple target detected in the image, f is the focal length of the camera, and Z is the horizontal distance between the camera and the actual target; According to the multi-task apple quality grading method integrating multi-dimensional features, it is characterized by establishing a front-end UI interface, the interface content includes image detection, video detection and real-time detection, and can output and display the detection results; A dedicated real-time detection device for the multi-task apple quality grading method integrating multi-dimensional features is characterized in that the dedicated real-time detection device comprises: a tripod base, a support plate, an electric telescopic rod, a power supply device, an industrial camera, and an electric rotating pan-tilt head. A tripod base 7 is fixed on the ground of the orchard, a support plate 5 is fixed on the base 7, and a hole is provided in the center of the support plate 5 for inserting and fixing the electric telescopic rod 3 through the hole. The electric rotating pan-tilt head 2 is fixed to the top of the electric telescopic rod 3, and the electric rotating pan-tilt head 2 can change the angle of the camera 1. An industrial camera 1 with a zoom lens is embedded in the electric rotating pan-tilt head 2 to collect information about apples in the orchard. The power supply device includes a solar panel 4 and a battery box 6; the solar panel 4 is fixed to the support plate 5, and the battery box 6 is fixed to the bottom of the tripod 7 to increase stability and provide power to the device.

Images