Agricultural product detection and sorting method based on adaptive gated visual perception

Abstract

The application discloses a kind of agricultural product detection and sorting method based on adaptive gating visual perception, which includes image acquisition, visual feature extraction, abnormal perception, quality evaluation and visual servo control module.By constructing multi-dimensional visual feature model including texture, color, saturation, edge, brightness, color consistency and dark area proportion, feature interaction expansion and adaptive gating mechanism are introduced to realize abnormal detection under complex defects.Combining size, color, shape and defect influence index, quality score is obtained through dynamic weight learning.According to the detection result, the target position is obtained, and mixed visual servo is used to realize accurate positioning and intelligent sorting of agricultural products.The application improves the accuracy of agricultural product detection, environmental adaptability and automated sorting capability.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, smart agriculture, machine learning, and automated control technology for agricultural robots, specifically to a method for detecting and sorting agricultural products based on adaptive gating visual perception. Background Technology

[0002] With the rapid development of smart agriculture and agricultural automation technologies, agricultural product quality testing and intelligent sorting have gradually become important technical means to improve agricultural production efficiency and product standardization. Traditional agricultural product quality testing mainly relies on manual observation and experience-based judgment, which results in low testing efficiency, susceptibility to subjective factors of testing personnel, and difficulty in meeting the needs for rapid, accurate, and consistent testing in large-scale agricultural production.

[0003] In recent years, agricultural product inspection methods based on computer vision technology have been widely applied. These methods acquire images of agricultural products and extract visual information such as color, texture, and morphology to achieve non-contact quality inspection. However, existing visual inspection methods typically use fixed visual features or single evaluation indicators for analysis. In actual agricultural environments, factors such as changes in lighting, target posture, surface reflection, and background interference can easily cause visual feature distortion, leading to a decrease in detection accuracy. Furthermore, different types of agricultural product defects have different visual manifestations. For example, rotten areas typically show color changes and increased dark areas, while mechanical damage usually manifests as changes in texture and edge structure. Traditional fixed-weight feature fusion methods struggle to automatically adjust the feature contribution level according to different defect states.

[0004] Furthermore, existing methods for evaluating the quality of agricultural products typically rely on fixed indicator weights, classifying products based on single factors such as size, color, or appearance. This lack of comprehensive analysis of the impact of defective areas leads to discrepancies between quality evaluation results and actual commodity value. Therefore, how to dynamically adjust the weights of different evaluation indicators based on the current state of agricultural products to achieve more accurate quality assessment is a problem that needs to be addressed in the field of smart agriculture.

[0005] In intelligent sorting, existing robotic sorting systems typically employ a method where the vision detection module and the mechanical execution module operate independently. The detection results are insufficient to directly guide the robot in performing precise grasping and sorting operations. Due to the randomness and complexity of the agricultural environment, the position, posture, and surface condition of agricultural products vary significantly. Traditional single-vision positioning methods are easily affected by errors, reducing the efficiency and stability of robot sorting.

[0006] Therefore, a method is needed that can integrate adaptive visual perception, dynamic quality evaluation, and intelligent execution control. By enhancing the ability to express visual features, it can improve the ability to detect defects in agricultural products under complex environments. At the same time, by combining dynamic quality evaluation models and visual servo control technology, it can realize the integrated processing of agricultural product detection, grading, and automatic sorting.

[0007] To address the aforementioned issues, this invention proposes an agricultural product detection and sorting method based on adaptive gating visual perception. By constructing an adaptive gating visual anomaly perception model, dynamic fusion of visual features is achieved; by learning a quality evaluation model through dynamic weights, comprehensive quality assessment of agricultural products is realized; and by combining visual positioning with robot sorting control, intelligent detection and automated sorting of agricultural products are achieved. Summary of the Invention

[0008] This invention addresses the problems existing in the intelligent detection of agricultural products, such as insufficient visual feature representation capabilities, poor detection stability under complex defect conditions, fixed weights of quality evaluation indicators, and difficulty in directly applying detection results to automatic sorting control. It proposes an agricultural product detection and sorting method based on adaptive gating visual perception.

[0009] This invention constructs an adaptive gated visual perception model to dynamically adjust multidimensional visual features in agricultural product images; it achieves adaptive fusion of different quality indicators through a dynamic weighted quality evaluation model; and it further combines eye-to-hand hybrid visual servo control to realize a closed-loop connection between detection results and robot sorting actions. To achieve the above objectives, the following steps are followed:

[0010] Step 1: Agricultural product image acquisition and preprocessing. Acquire image data of the agricultural products to be detected, and perform target region extraction, size normalization and illumination correction on the images to obtain standardized input images, providing a data foundation for subsequent visual feature extraction.

[0011] Step 2: Construct a multidimensional visual anomaly perception model based on an adaptive gating mechanism. Input the standardized agricultural product image into the multidimensional visual anomaly perception model to extract visual features such as texture, color, saturation, edge structure, brightness, color consistency, and dark area ratio. Construct a visual feature interaction extension model to provide input for subsequent gating weight calculation.

[0012] Step 2.1: Construct a seven-dimensional visual anomaly feature vector. Visual features are extracted from the preprocessed agricultural product image to obtain texture features, color features, saturation features, edge structure features, brightness features, color consistency features, and dark area proportion features, thus constructing a seven-dimensional visual anomaly feature vector.

[0013] ,

[0014] in, Indicates texture anomaly features, Indicates color deviation characteristics. Indicates saturation characteristics, Indicates edge structure features, Indicates brightness deviation characteristics. Indicates color consistency characteristics. Indicates the proportion characteristics of the dark area;

[0015] Step 2.2: Construct a feature interaction extension model based on the seven-dimensional visual anomaly features, and obtain the enhanced visual feature vector by calculating the second-order interaction relationship between different visual features:

[0016] ,

[0017] in, Represents an extended visual feature vector. Indicates the first The visual feature and the first Interaction features between visual features;

[0018] Step 3: Construct an adaptive gating mechanism. After obtaining the extended visual feature vector, construct an adaptive gating mechanism to dynamically adjust the importance of different visual features according to the defect status in the current agricultural product image, so that the detection model can automatically select more effective visual information for different types of abnormal regions.

[0019] Step 3.1: Input the extended visual feature vector into the gating weight generation module, and calculate the dynamic gating weights corresponding to each feature based on the current visual state of the agricultural product image:

[0020] ,

[0021] in, Indicates the first The gating weights corresponding to each visual feature Indicates the gating parameters, Indicates the bias term. This represents the Sigmoid activation function;

[0022] Step 3.2: Dynamically adjust the importance of different visual features based on the obtained gating weights:

[0023] ,

[0024] in: Indicates the number after gating adjustment 3D visual features; Represents original visual features; Indicates the corresponding gating weight;

[0025] The adjusted visual features are represented as follows:

[0026] ,

[0027] Depending on the different defect states, the gating mechanism automatically increases the importance of key visual features.

[0028] Step 4: Construct a comprehensive quality evaluation model based on dynamic weight learning, which integrates indicators of agricultural product size, color, shape, and defect impact, adaptively adjusts the weights of quality evaluation indicators according to the state of different test objects, obtains a comprehensive quality score, and completes the classification of agricultural product quality grades based on the score results.

[0029] Step 4.1: Calculate the standardized values ​​of visual anomaly features and perform standardization processing on the visual anomaly features:

[0030] No. The degree of visual feature anomaly is represented as follows:

[0031] ,

[0032] in: This represents the anomalous response value after standardization of the k-th dimension visual feature; Indicating the first normal sample Mean of 3D visual features; Indicating the first normal sample Standard deviation of 3D visual features; This represents a very small constant to prevent the denominator from being zero;

[0033] Step 4.2: Calculate the adaptive gating anomaly score by fusing the standardized visual anomaly features with the dynamic gating weights to obtain the final anomaly score.

[0034] ,

[0035] in, This indicates the anomaly score obtained based on the adaptive gating mechanism. Indicates the first Dynamic weights of visual features This indicates the corresponding visual abnormality features. This score is used to determine whether agricultural products have any abnormal defects.

[0036] Further set anomaly detection thresholds When the following conditions are met: If the sample is found to have quality abnormalities, it will be determined that the agricultural product has quality abnormalities and will proceed to the subsequent quality evaluation process; otherwise, it will be determined as a normal sample.

[0037] Step 5: Construct a dynamic weighted quality evaluation model. After anomaly detection is completed, conduct a comprehensive analysis of the factors affecting the size, color, shape, and defects of agricultural products, establish a dynamic weighted quality evaluation model, and realize the automatic evaluation of agricultural products of different quality grades.

[0038] Step 5.1: Calculate the defect impact index. First, obtain the defect area information based on the anomaly detection results, and then calculate the defect impact index:

[0039] ,

[0040] in, Represents the probability of a defect. Indicates the proportion of the defective area. Indicates the severity of the defect;

[0041] Step 5.2: Calculate the comprehensive quality score, integrating factors such as size, color, shape, and defects to establish a comprehensive quality evaluation model for agricultural products.

[0042] ,

[0043] in, This indicates the overall quality score of agricultural products. , , These represent the evaluation criteria for size, color, and shape, respectively. , , , This indicates the weights of the evaluation indicators obtained through dynamic optimization based on the correlation between historical sample quality labels and evaluation indicators.

[0044] Step 6: Visual positioning and coordinate transformation. After completing the quality inspection and grading of agricultural products, the center position of the target area of ​​the agricultural product is obtained through the visual positioning module, and the transformation between the image coordinate system, the camera coordinate system and the robot coordinate system is completed using the camera calibration parameters, so as to realize the data association between the inspection results and the mechanical actuator.

[0045] Step 6.1: Obtain the image coordinates of the target agricultural product area. Based on the anomaly detection results obtained in Step 4 and the quality grade information obtained in Step 5, locate the target agricultural product area.

[0046] Obtain the pixel set of agricultural product regions through image segmentation:

[0047] ,

[0048] in: This represents the set of pixels representing the target region of agricultural products. , Indicates the first The coordinates of each target pixel in the image coordinate system; Indicates the number of pixels in the target area;

[0049] Calculate the center location of the target region:

[0050] ,

[0051] ,

[0052] in:( , () represents the two-dimensional coordinates of the center of the agricultural product in the image coordinate system;

[0053] Step 6.2: Image coordinate to camera coordinate transformation. Based on the camera imaging model, the two-dimensional image coordinates are transformed into three-dimensional camera coordinates. The transformation relationship is as follows:

[0054] ,

[0055] in: Indicates the target camera coordinates; Represents the camera intrinsic parameter matrix; This indicates the depth information of the target point from the camera;

[0056] Step 6.3: Transformation from camera coordinate system to robot coordinate system. The transformation relationship is as follows:

[0057] ,

[0058] in: This indicates the position of the target point in the robot coordinate system; Indicates the position of the target point in the camera coordinate system; This represents the rotation matrix between the camera coordinate system and the robot coordinate system; This represents the translation vector between two coordinate systems.

[0059] Step 7: Eye-to-Hand hybrid visual servo control. Based on the target spatial position obtained in Step 6, the mechanical actuator is driven to complete the grasping and sorting of agricultural products using visual servo control methods. Workspace information is acquired through a camera, and hybrid control is performed by combining image error and spatial position error, enabling the robot to adjust its motion state in real time based on visual feedback.

[0060] Step 7.1: Establish an image spatial error model. Based on the difference between the target image position obtained from visual detection and the expected position, establish the image error:

[0061] ,

[0062] in: Indicates image spatial error; , () represents the current target image coordinates; , () represents the desired image coordinates of the target;

[0063] Step 7.2: Establish a spatial position error model. Based on the robot coordinate system position obtained in Step 6, calculate the target spatial error:

[0064] ,

[0065] in: Indicates spatial position error; , , () indicates the current position of the target in the robot coordinate system; , , () indicates the desired location of the target;

[0066] Step 7.3: Construct a hybrid visual servo control model, fusing image errors and spatial position errors to establish a hybrid visual servo control law:

[0067] ,

[0068] in: Indicates the robot's motion control quantities; Represents the image spatial control gain matrix; Represents the spatial position control gain matrix; Indicates image error; Indicates spatial error;

[0069] Step 7.4: Complete intelligent sorting. Based on the quality grade results and visual servo control output, drive the robot to complete the grasping, moving and sorting of agricultural products, and realize intelligent detection and automatic sorting of agricultural products.

[0070] The beneficial effects of this invention are:

[0071] 1. By constructing a multi-dimensional feature fusion mechanism based on adaptive gating visual perception, the multi-source visual information such as color, texture, edge, and brightness in agricultural product images is comprehensively represented. The adaptive gating module is used to dynamically adjust the weights of different visual features, so that the detection process can automatically select key visual information according to different agricultural product states, thereby improving the accuracy and robustness of agricultural product quality detection in complex environments.

[0072] 2. By introducing a dynamic feature weight adjustment strategy, the information redundancy and feature failure problems caused by fixed feature combinations in traditional methods are avoided, enabling the system to effectively identify different defect types and improving the detection capability of various abnormalities such as surface damage, color anomalies, and morphological changes.

[0073] 3. An adaptive sorting control method based on visual feedback is adopted. By integrating visual servo control strategies at different stages, the robot has a faster response speed in the long-distance movement stage and higher positioning accuracy in the approaching target stage, thereby improving the motion stability and control accuracy in the sorting process. Attached Figure Description

[0074] Figure 1 This is an overall flowchart of an embodiment of the present invention; Detailed Implementation

[0075] The following is in conjunction with the appendix Figure 1 The flowchart and specific embodiments of the present invention further illustrate the method for detecting and sorting agricultural products based on adaptive gating visual perception proposed in this invention, but the scope of protection of this invention is not limited to the following embodiments.

[0076] This embodiment uses Python 3.13 as the development environment, OpenCV 4.13 to complete the acquisition, preprocessing and visual feature extraction of agricultural product images, and scikit-learn 1.7 to complete the visual feature data processing, dynamic weight learning and quality evaluation model construction. It is combined with industrial cameras, robotic arms and controllers to form an intelligent detection and sorting system for agricultural products.

[0077] A method for detecting and sorting agricultural products based on adaptive gating visual perception includes the following steps:

[0078] Step 1: Acquire and preprocess image data of the agricultural products to be inspected. Images of the agricultural products to be inspected are acquired using an industrial camera. The acquired products include agricultural products of different varieties, different ripeness levels, and different defect states. To reduce the impact of complex backgrounds and ambient lighting on the inspection results, the acquired raw images undergo target region extraction, image size normalization, and lighting correction processing in sequence to obtain standardized agricultural product images.

[0079] First, the acquired RGB image is represented as:

[0080] ,

[0081] in, , , These represent the pixels in the image. The corresponding red, green, and blue color channels. Then, the image is uniformly scaled to a preset size:

[0082] ,

[0083] in, This represents the normalized image; and These represent the normalized image width and height, respectively. The preprocessed image serves as the input for the subsequent visual anomaly detection model.

[0084] Step 2: Construct a multidimensional visual anomaly perception model based on an adaptive gating mechanism. Input the standardized agricultural product image into the multidimensional visual anomaly perception model to extract visual features such as texture, color, saturation, edge structure, brightness, color consistency, and dark area ratio. Construct a visual feature interaction extension model to provide input for subsequent gating weight calculation.

[0085] Step 2.1: Extract multidimensional visual features from the preprocessed agricultural product images and construct a seven-dimensional visual anomaly feature vector;

[0086] First, the RGB image is converted to the HSV color space. The conversion result is shown as follows:

[0087] ,

[0088] in, , , These represent the hue, saturation, and brightness components, respectively. Seven-dimensional visual anomaly features are extracted from the converted image to construct a visual feature vector:

[0089] ,

[0090] in, Indicates texture anomaly features, Indicates color deviation characteristics. Indicates saturation characteristics, Indicates edge structure features, Indicates brightness deviation characteristics. Indicates color consistency characteristics. Indicates the proportion characteristics of the dark area;

[0091] The color deviation feature is calculated using the Euclidean distance between the current detection area and the standard color template.

[0092] ,

[0093] in, , , These represent the average values ​​of the color samples of normal agricultural products.

[0094] Texture anomaly features are used to calculate texture entropy using the gray-level co-occurrence matrix:

[0095] ,

[0096] in, Represents the gray levels in the gray-level co-occurrence matrix. and The probability of them occurring simultaneously; This indicates the number of gray levels.

[0097] The gradient magnitude of edge structure features is calculated using the Sobel operator:

[0098] ,

[0099] in, and They represent the first The gradient values ​​of each pixel in the horizontal and vertical directions; Indicates the total number of pixels in the target area;

[0100] The dark area ratio characteristic is defined as:

[0101] ,

[0102] in, This indicates the number of pixels whose brightness is below a preset threshold; Indicates the total number of pixels in the target area;

[0103] Step 2.2: Construct a visual feature interaction extension model. Since agricultural product quality defects are usually represented by multiple visual features, a second-order interaction extension is performed on the seven-dimensional visual anomaly features to enhance the correlation between different visual features. The extension form is as follows:

[0104] ,

[0105] in, Represents an extended visual feature vector. Indicates the first The visual feature and the first The interaction features between visual features, and the expanded visual feature vectors are used as input to the adaptive gating mechanism. By simultaneously retaining the original visual feature information and the interaction information between features, the model's ability to perceive complex quality problems such as decay, mechanical damage, abnormal maturity and surface defects is improved, providing richer visual information for subsequent dynamic gating weight calculation and anomaly scoring.

[0106] Step 3: Construct an adaptive gating mechanism. After obtaining the extended visual feature vector, construct an adaptive gating mechanism to dynamically adjust the importance of different visual features according to the defect status in the current agricultural product image, so that the detection model can automatically select more effective visual information for different types of abnormal regions.

[0107] Traditional visual inspection methods typically use fixed weights to fuse different visual features, which cannot adapt to the visual differences between different agricultural products and different defect types. This invention introduces dynamic gating weights, enabling each visual feature to adaptively adjust according to the current detection state;

[0108] Step 3.1: Calculate the dynamic gating weights. Input the extended visual feature vector obtained in Step 2.2 into the gating weight calculation model to calculate the... Gating weights corresponding to each visual feature:

[0109] ,

[0110] in, Indicates the first The gating weights corresponding to each visual feature Indicates the gating parameters, Indicates the bias term. This represents the Sigmoid activation function;

[0111] Furthermore, the gating input is represented as:

[0112] ,

[0113] in: Indicates the first The gating response value corresponding to each visual feature; Indicates the first The weight parameters corresponding to each gated unit; This represents the expanded visual feature vector; Indicates the bias parameter;

[0114] The gating response value is further normalized using the Sigmoid function to obtain the dynamic gating weights.

[0115] ,

[0116] in, Indicates the first The dynamic gating weights of each visual feature have a value range of 0 to 1.

[0117] The gating weights corresponding to all visual features are represented as follows:

[0118] ,

[0119] in, Represents the set of visual feature gating weights;

[0120] Step 3.2: Adaptive adjustment of visual features. Based on the dynamic gating weights obtained in Step 3.1, adjust the visual features of each dimension:

[0121] ,

[0122] in: Indicates the number after gating adjustment 3D visual features; Represents original visual features; Indicates the corresponding gating weight;

[0123] The adjusted visual features are represented as follows:

[0124] ,

[0125] Depending on the different defect states, the gating mechanism automatically increases the importance of key visual features. Specifically: when detecting areas such as spoilage or dark spots in agricultural products, the gating mechanism increases the weight of brightness features and dark area ratio features; when detecting abnormalities such as mechanical damage or epidermal breakage, it increases the weight of texture features and edge structure features; and when detecting abnormal maturity, it increases the weight of color-related features.

[0126] Through the above adaptive adjustment process, the model can automatically select effective visual information according to different defect types, thereby improving its adaptability to detecting agricultural product anomalies in complex environments.

[0127] Step 4: Anomaly score calculation. Based on the gated fusion visual features obtained in Step 3, the degree of anomaly in the agricultural product image is quantified to obtain an adaptive gated anomaly score.

[0128] Step 4.1: Calculate the standardized values ​​of visual anomaly features. Since the dimensions and numerical ranges of different visual features differ, the visual features of each dimension are first normalized:

[0129] No. The degree of visual feature anomaly is represented as follows:

[0130] ,

[0131] in: This represents the anomalous response value after standardization of the k-th dimension visual feature; Indicating the first normal sample Mean of 3D visual features; Indicating the first normal sample Standard deviation of 3D visual features; This represents a very small constant to prevent the denominator from being zero;

[0132] Step 4.2: Calculate the adaptive gating anomaly score by fusing the standardized visual anomaly features with the dynamic gating weights to obtain the final anomaly score.

[0133] ,

[0134] in, This indicates the anomaly score obtained based on the adaptive gating mechanism. Indicates the first Dynamic weights of visual features Indicates the corresponding visual abnormality features;

[0135] Further set anomaly detection thresholds When the following conditions are met: If the sample is found to be abnormal, it is determined that the agricultural product has a quality abnormality and proceeds to the subsequent quality evaluation process; otherwise, it is determined to be a normal sample.

[0136] The above-mentioned anomaly scoring method enables the detection of agricultural product anomalies based on dynamic adjustment of visual features.

[0137] Step 5: Construct a dynamic weighted quality evaluation model. After anomaly detection is completed, conduct a comprehensive analysis of the factors affecting the size, color, shape, and defects of agricultural products, establish a dynamic weighted quality evaluation model, and realize the automatic evaluation of agricultural products of different quality grades.

[0138] Step 5.1: Calculate the defect impact index. First, obtain the defect area information based on the anomaly detection results, and then calculate the defect impact index:

[0139] ,

[0140] in, Represents the probability of a defect. Indicates the proportion of the defective area. Indicates the severity of the defect;

[0141] The percentage of defective areas is calculated as follows:

[0142] ,

[0143] in: Indicates the number of pixels in the defect area; This indicates the total number of pixels in the target area for agricultural products;

[0144] The severity of defects is calculated based on a comprehensive assessment of color anomalies, texture anomalies, and morphological changes.

[0145] ,

[0146] in: Indicates the degree of color abnormality; Indicates the degree of texture anomaly; Indicates the degree of morphological abnormality; , , Indicates the corresponding evaluation weight;

[0147] Step 5.2: Calculate the comprehensive quality score, integrating factors such as size, color, shape, and defects to establish a comprehensive quality evaluation model for agricultural products.

[0148] ,

[0149] in, This indicates the overall quality score of agricultural products. , , These represent the evaluation criteria for size, color, and shape, respectively. , , , This indicates the weights of the evaluation indicators obtained through dynamic optimization based on the correlation between historical sample quality labels and evaluation indicators;

[0150] Agricultural products are graded based on their overall quality score:

[0151] ,

[0152] in: Indicates the quality grade; , This indicates the threshold for classifying quality grades;

[0153] The dynamic weighted quality evaluation model enables automatic determination of the quality grade of agricultural products, providing a basis for subsequent intelligent sorting.

[0154] Step 6: Visual positioning and coordinate transformation. After completing the quality inspection and grading of agricultural products, it is necessary to further determine the spatial position of the target agricultural products in the actual working space to provide accurate position coordinates for subsequent mechanical sorting.

[0155] This invention obtains the center position of the target area of ​​agricultural products through a visual positioning module, and uses camera calibration parameters to complete the transformation between the image coordinate system, the camera coordinate system and the robot coordinate system, thereby realizing the data association between the detection results and the mechanical actuator.

[0156] Step 6.1: Obtain the image coordinates of the target area of ​​agricultural products. Based on the anomaly detection results obtained in Step 4 and the quality grade information obtained in Step 5, locate the target agricultural product area.

[0157] Obtain the pixel set of agricultural product regions through image segmentation:

[0158] ,

[0159] in: This represents the set of pixels representing the target region of agricultural products. , Indicates the first The coordinates of each target pixel in the image coordinate system; Indicates the number of pixels in the target area;

[0160] Calculate the center location of the target region:

[0161] ,

[0162] ,

[0163] in:( , () represents the two-dimensional coordinates of the center of the agricultural product in the image coordinate system;

[0164] Step 6.2: Image coordinate to camera coordinate transformation. Based on the camera imaging model, convert the two-dimensional image coordinates into three-dimensional camera coordinates:

[0165] The conversion relationship is as follows: ,Right now

[0166] ,

[0167] in:( , , () represent the three-dimensional coordinates of the target point in the camera coordinate system; Represents the camera intrinsic parameter matrix; This indicates the depth information of the target point from the camera;

[0168] The camera intrinsic parameter matrix is ​​represented as follows:

[0169] ,

[0170] in: , These represent the camera's focal length in the horizontal and vertical directions, respectively; , () represents the coordinates of the principal point of the image;

[0171] Step 6.3: Transformation from camera coordinate system to robot coordinate system. In order to realize information interaction between the vision system and the mechanical actuator, the target point needs to be transformed from the camera coordinate system to the robot base coordinate system:

[0172] The transformation relationship is as follows:

[0173] ,

[0174] in: This indicates the position of the target point in the robot coordinate system; Indicates the position of the target point in the camera coordinate system; This represents the rotation matrix between the camera coordinate system and the robot coordinate system; This represents the translation vector between two coordinate systems;

[0175] Through the above coordinate transformation process, an accurate mapping between agricultural product testing results and mechanical sorting positions is achieved.

[0176] Step 7: Eye-to-Hand hybrid vision servo control. Based on the target spatial position obtained in Step 6, the vision servo control method is used to drive the mechanical actuator to complete the grasping and sorting of agricultural products.

[0177] This invention adopts an Eye-to-Hand structure, in which the vision sensor is fixedly installed on the outside of the robot, acquires workspace information through a camera, and combines image error and spatial position error for hybrid control to improve positioning accuracy during the sorting process;

[0178] Step 7.1: Establish an image spatial error model. Based on the difference between the target image position obtained from visual detection and the expected position, establish the image error:

[0179] ,

[0180] in: Indicates image spatial error; , () represents the current target image coordinates; , () represents the desired image coordinates of the target;

[0181] Step 7.2: Establish a spatial position error model. Based on the robot coordinate system position obtained in Step 6, calculate the target spatial error:

[0182] ,

[0183] in: Indicates spatial position error; , , () indicates the current position of the target in the robot coordinate system; , , () indicates the desired location of the target;

[0184] Step 7.3: Construct a hybrid visual servo control model, fusing image errors and spatial position errors to establish a hybrid visual servo control law:

[0185] ,

[0186] in: Indicates the robot's motion control quantities; Represents the image spatial control gain matrix; Represents the spatial position control gain matrix; Indicates image error; Indicates spatial error;

[0187] By simultaneously utilizing image information and spatial pose information, the robot is able to adjust its motion state in real time based on visual feedback.

[0188] Step 7.4: Complete the intelligent sorting of agricultural products. Based on the results of the hybrid vision servo control, the robot end effector moves to the location of the target agricultural product and performs the corresponding sorting action according to the quality level obtained in Step 5.

[0189] The specific process includes:

[0190] (1) The vision system continuously acquires images of the working area;

[0191] (2) The detection model outputs the quality category and spatial location of agricultural products;

[0192] (3) The control system adjusts the mechanical motion trajectory based on visual and spatial errors;

[0193] (4) The mechanical actuator completes the grasping, moving, and releasing actions;

[0194] (5) Agricultural products of different quality grades are transported to the corresponding collection areas.

[0195] Through the above steps, intelligent detection and automatic sorting of agricultural products based on adaptive gating visual perception, dynamic quality evaluation, and hybrid visual servo control can be achieved.

Claims

1. A method for detecting and sorting agricultural products based on adaptive gating visual perception, characterized in that, Includes the following steps: Step 1: Agricultural product image acquisition and preprocessing. Acquire image data of agricultural products to be detected, and perform target region extraction, size normalization and illumination correction on the images to obtain standardized input images, providing a data foundation for subsequent visual feature extraction. Step 2: Construct a multidimensional visual anomaly perception model based on an adaptive gating mechanism. Input the standardized agricultural product image into the multidimensional visual anomaly perception model to extract visual features such as texture, color, saturation, edge structure, brightness, color consistency, and dark area ratio. Construct a visual feature interaction extension model to provide input for subsequent gating weight calculation. Step 3: Construct an adaptive gating mechanism. After obtaining the extended visual feature vector, construct an adaptive gating mechanism to dynamically adjust the importance of different visual features according to the defect status in the current agricultural product image, so that the detection model can automatically select more effective visual information for different types of abnormal regions. Step 4: Construct a comprehensive quality evaluation model based on dynamic weight learning, which integrates indicators of agricultural product size, color, shape, and defect impact, adaptively adjusts the weights of quality evaluation indicators according to the state of different test objects, obtains a comprehensive quality score, and completes the classification of agricultural product quality grades based on the score results. Step 5: Construct a dynamic weighted quality evaluation model. After anomaly detection is completed, conduct a comprehensive analysis of the factors affecting the size, color, shape, and defects of agricultural products, establish a dynamic weighted quality evaluation model, and realize the automatic evaluation of agricultural products of different quality grades. Step 6: Visual positioning and coordinate transformation. After completing the quality inspection and grading of agricultural products, the center position of the target area of ​​the agricultural product is obtained through the visual positioning module, and the transformation between the image coordinate system, the camera coordinate system and the robot coordinate system is completed using the camera calibration parameters, so as to realize the data association between the inspection results and the mechanical actuator. Step 7: Eye-to-Hand hybrid visual servo control. Based on the target spatial position obtained in Step 6, the mechanical actuator is driven to complete the grasping and sorting of agricultural products using the visual servo control method. Workspace information is obtained through the camera, and hybrid control is performed by combining image error and spatial position error, so that the robot can adjust its motion state in real time according to visual feedback.

2. The method for detecting and sorting agricultural products based on adaptive gating visual perception according to claim 1, characterized in that, The multidimensional visual anomaly perception model based on adaptive gating mechanism includes the following steps: Step 2.1: Construct a seven-dimensional visual anomaly feature vector. Visual features are extracted from the preprocessed agricultural product image to obtain texture features, color features, saturation features, edge structure features, brightness features, color consistency features, and dark area proportion features, thus constructing a seven-dimensional visual anomaly feature vector. ， in, Indicates texture anomaly features, Indicates color deviation characteristics. Indicates saturation characteristics, Indicates edge structure features, Indicates brightness deviation characteristics. Indicates color consistency characteristics. Indicates the proportion characteristics of the dark area; Step 2.2: Construct a feature interaction extension model based on the seven-dimensional visual anomaly features, and obtain the enhanced visual feature vector by calculating the second-order interaction relationship between different visual features: ， in, Represents an extended visual feature vector. Indicates the first The visual feature and the first Interaction features between visual features.

3. The method for detecting and sorting agricultural products based on adaptive gating visual perception according to claim 1, characterized in that, The adaptive gating mechanism includes the following steps: Step 3.1: Input the extended visual feature vector into the gating weight generation module, and calculate the dynamic gating weights corresponding to each feature based on the current visual state of the agricultural product image: ， in, Indicates the first The gating weights corresponding to each visual feature Indicates the gating parameters, Indicates the bias term. This represents the Sigmoid activation function; Step 3.2: Dynamically adjust the importance of different visual features based on the obtained gating weights: ， in: Indicates the number after gating adjustment 3D visual features; Represents original visual features; The corresponding gating weights are represented as follows: ， Depending on the different defect states, the gating mechanism automatically increases the importance of key visual features.

4. The method for detecting and sorting agricultural products based on adaptive gating visual perception according to claim 1, characterized in that, The comprehensive quality evaluation model based on dynamic weight learning includes the following steps: Step 4.1: Calculate the standardized values ​​of visual anomaly features and perform standardization processing on the visual anomaly features: No. The degree of visual feature anomaly is represented as follows: ， in: This represents the anomalous response value after standardization of the k-th dimension visual feature; Indicating the first normal sample Mean of 3D visual features; Indicating the first normal sample Standard deviation of 3D visual features; This represents a very small constant to prevent the denominator from being zero; Step 4.2: Calculate the adaptive gating anomaly score by fusing the standardized visual anomaly features with the dynamic gating weights to obtain the final anomaly score. ， in, This indicates the anomaly score obtained based on the adaptive gating mechanism. Indicates the first Dynamic weights of visual features This score indicates the corresponding visual abnormality features and is used to determine whether agricultural products have abnormal defects. Further set anomaly detection thresholds When the following conditions are met: If the quality of the agricultural product is abnormal, it will be determined that the product has an abnormality and will proceed to the subsequent quality evaluation process; otherwise, it will be determined as a normal sample.

5. The method for detecting and sorting agricultural products based on adaptive gating visual perception according to claim 1, characterized in that, The construction of the dynamic weighted quality evaluation model includes the following steps: Step 5.1: Calculate the defect impact index. First, obtain the defect area information based on the anomaly detection results, and then calculate the defect impact index: ， in, Represents the probability of a defect. Indicates the proportion of the defective area. Indicates the severity of the defect; Step 5.2: Calculate the comprehensive quality score, integrating factors such as size, color, shape, and defects to establish a comprehensive quality evaluation model for agricultural products. ， in, This indicates the overall quality score of agricultural products. , , These represent the evaluation criteria for size, color, and shape, respectively. , , , This indicates the weights of the evaluation indicators obtained through dynamic optimization based on the correlation between historical sample quality labels and evaluation indicators.

6. The method for detecting and sorting agricultural products based on adaptive gating visual perception according to claim 1, characterized in that, The visual positioning and coordinate transformation method includes the following steps: Step 6.1: Obtain the image coordinates of the target agricultural product area. Based on the anomaly detection results obtained in Step 4 and the quality grade information obtained in Step 5, locate the target agricultural product area. Obtain the pixel set of agricultural product regions through image segmentation: ， in: This represents the set of pixels representing the target region of agricultural products. , Indicates the first The coordinates of each target pixel in the image coordinate system; Indicates the number of pixels in the target area; Calculate the center location of the target region: ， ， in:( , () represents the two-dimensional coordinates of the center of the agricultural product in the image coordinate system; Step 6.2: Image coordinate to camera coordinate transformation. Based on the camera imaging model, the two-dimensional image coordinates are transformed into three-dimensional camera coordinates. The transformation relationship is as follows: ， in: Indicates the target camera coordinates; Represents the camera intrinsic parameter matrix; This indicates the depth information of the target point from the camera; Step 6.3: Transformation from camera coordinate system to robot coordinate system. The transformation relationship is as follows: ， in: This indicates the position of the target point in the robot coordinate system; Indicates the position of the target point in the camera coordinate system; This represents the rotation matrix between the camera coordinate system and the robot coordinate system; This represents the translation vector between two coordinate systems.

7. The method for detecting and sorting agricultural products based on adaptive gating visual perception according to claim 1, characterized in that, The Eye-to-Hand hybrid vision servo control method includes the following steps: Step 7.1: Establish an image spatial error model. Based on the difference between the target image position obtained from visual detection and the expected position, establish the image error: ， in: Indicates image spatial error; , () represents the current target image coordinates; , () represents the desired image coordinates of the target; Step 7.2: Establish a spatial position error model. Based on the robot coordinate system position obtained in Step 6, calculate the target spatial error: ， in: Indicates spatial position error; , , () indicates the current position of the target in the robot coordinate system; , , () indicates the desired location of the target; Step 7.3: Construct a hybrid visual servo control model, fusing image errors and spatial position errors to establish a hybrid visual servo control law: ， in: Indicates the robot's motion control quantities; Represents the image spatial control gain matrix; Represents the spatial position control gain matrix; Indicates image error; Indicates spatial error; Step 7.4: Complete intelligent sorting. Based on the quality grade results and visual servo control output, drive the robot to complete the grasping, moving and sorting of agricultural products, and realize intelligent detection and automatic sorting of agricultural products.

Images