Strawberry nondestructive sorting method and system based on multi-modal fusion and force feedback control

Abstract

The application discloses a strawberry nondestructive sorting method and system based on multi-modal fusion and force feedback control, which comprises the following steps: collecting images of strawberries of different varieties, ripeness and quality states, including color, texture, shape and other characteristic parameters; placing the strawberries to be detected on a visual recognition station, acquiring multi-angle images through a high-resolution camera, and performing preprocessing such as light compensation and background segmentation; establishing a multi-modal feature extraction branch, extracting color features, edge features, texture features and shape features in parallel, calculating feature scores, adjusting feature weights through a weight attention mechanism, and calculating a comprehensive similarity score based on a weighted fusion algorithm; performing strawberry quality grading determination, outputting the quality category and defect type of the strawberries; and outputting the grading result and driving a mechanical arm and a dynamically adjusted mechanical claw to realize adaptive grabbing and sorting. The application realizes automatic detection and grading of the appearance quality of strawberries through the cooperative work of visual recognition hardware and software.

Description

Technical Field

[0001] This invention relates to the interdisciplinary fields of computer vision, artificial intelligence, and agricultural automation, specifically to a method and system for intelligent detection and adaptive sorting of strawberry quality based on multimodal feature fusion and closed-loop feedback control. In particular, it is an intelligent technology that can simultaneously determine strawberry maturity, identify damage, and detect diseases, and achieve non-destructive adaptive sorting through a vision-mechanical linkage closed-loop feedback control system. Background Technology

[0002] Strawberry picking and quality grading mainly rely on manual labor, which suffers from low efficiency, high costs, unstable quality, secondary damage, high labor intensity, and low standardization. Although some automated sorting equipment exists, the following technical shortcomings remain: Single-color recognition technology.

[0003] It can only determine maturity, but cannot identify damage and disease, and is greatly affected by light conditions; traditional image processing technology: low accuracy, easily affected by light, and limited feature extraction ability; fixed force grasping technology: cannot adapt to the hardness differences of strawberries at different maturity levels, and is prone to secondary damage; recognition and execution separation technology: cannot adjust the recognition strategy according to the actual grasping situation, and the system has poor robustness.

[0004] Existing technologies generally have an accuracy rate of less than 90%, a damage rate of 5%-8% during sorting, and a processing speed of less than 1000 pieces / hour, which cannot meet the needs of large-scale commercialization. Summary of the Invention

[0005] To address the shortcomings and deficiencies of existing technologies in strawberry quality inspection and sorting, this invention starts with multimodal feature fusion and closed-loop feedback control. It introduces a multimodal feature weighted fusion algorithm for feature extraction to adapt to the inspection of strawberries under different varieties and lighting conditions. Simultaneously, a closed-loop feedback control algorithm replaces the traditional open-loop control algorithm, dynamically adjusting the gripping force based on the strawberry size and real-time force feedback to avoid strawberry damage and improve sorting accuracy.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A non-destructive strawberry sorting method based on multimodal fusion and force feedback control includes the following steps:

[0008] S1. Acquire strawberry images;

[0009] S2. Perform background segmentation, image enhancement, location coordinate positioning, and size normalization on the strawberry image;

[0010] S3. Establish a multimodal feature extraction branch to extract color features, edge features, texture features, and shape features in parallel, and obtain color similarity score Sc, edge similarity score Se, texture similarity score St, and shape similarity score Ss, respectively; wherein the color features are extracted in the HSV color space, the edge features are extracted using the adaptive Canny operator, the texture features are extracted using the gray-level co-occurrence matrix, and the shape features are extracted based on contour analysis.

[0011] S4. Through a weighted attention mechanism, dynamically adjust each weight parameter based on strawberry variety, lighting conditions, and texture complexity: color similarity weight. Edge similarity weight Texture similarity weight Shape similarity weight The overall similarity score is calculated based on a weighted fusion algorithm.

[0012] S5. Determine the quality grade of strawberries based on the comprehensive similarity score, and output the quality category and defect type of strawberries;

[0013] S6. Based on the grading results, output instructions to control the gripper to perform adaptive gripping and sorting. The gripping force is dynamically adjusted according to the size of the strawberry and real-time force feedback. A PID control algorithm is used to achieve closed-loop feedback control to ensure that the gripping force is within the threshold range.

[0014] S7. Establish a parameter optimization database, use manually reviewed and corrected parameters as optimization data, and continuously optimize through adaptive parameter adjustment.

[0015] The strawberry images used are from open-source datasets or real-time camera images; and the images contain various types of defects.

[0016] The GrabCut algorithm is used to distinguish between the foreground and background, and the CLAHE method is used for enhancement. The three-dimensional position coordinates are located based on the edge contour of the enhanced image, the image size is normalized to 224×224, and the pixel value is normalized to [0,1].

[0017] The similarity scores for each feature are calculated as follows:

[0018] Extract hue, saturation, and lightness from the HSV color space; calculate the color histogram and color moments; and calculate the color similarity score. ;in, For HSV space Euclidean distance. The maximum possible distance;

[0019] An adaptive Canny operator is used for edge detection, calculating contour integrity score, edge density, edge length, edge curvature, and edge similarity score. ;in, The maximum number of pixels at the edge of the connected region. This represents the total number of edge pixels.

[0020] Multi-texture features, including energy, contrast, entropy, correlation, homogeneity, dissimilarity, and angular second moment, are extracted in four directions (0°, 45°, 90°, and 135°) using the gray-level co-occurrence matrix. Texture similarity scores are then calculated. cos(θ); Cosine similarity is used to calculate texture feature similarity, with a value range of [0,1], where θ is the angle between the strawberry texture feature vector to be tested and the standard texture feature vector;

[0021] Shape features such as the outline area, perimeter, aspect ratio, roundness, eccentricity, and rectangularity of the strawberry are extracted to obtain a shape similarity score. Shape feature matching degree, range [0,1].

[0022] The weights are dynamically adjusted using the following three functions via an attention mechanism:

[0023] Color similarity weighting function based on variety, lighting, and texture :

[0024] ;

[0025] Edge similarity weight adjustment function based on variety and illumination :

[0026] ;

[0027] Texture similarity weight adjustment function based on variety and texture :

[0028] ;

[0029] This leads to the shape similarity weight adjustment function. :

[0030] ;

[0031] in, is the light intensity factor (range [0,1]); Variety factor (range [0,1]); is the texture complexity factor (range [0,1]);

[0032] The overall similarity score .

[0033] The quality grading is determined as follows: a score ≥ 0.90 indicates a Grade 1 product; a score ≤ 0.75 ≤ 90% indicates a Grade 2 product; otherwise, it is considered unqualified.

[0034] The step of outputting instructions based on the grading results to control the gripper for adaptive grasping and sorting, wherein the gripping force is dynamically adjusted according to the size of the strawberries and real-time force feedback, including:

[0035] Based on the location of the target strawberry to be picked, position control commands are output to the encoders of each axis of the robotic arm, which enable the robotic arm to adjust each axis to move along the planned path from the start to the target strawberry, and to adjust the gripping force, opening angle and direction of the end gripper in real time to achieve non-destructive gripping of the strawberry.

[0036] The PID control algorithm is used to achieve closed-loop feedback control, ensuring that the grip strength is within a threshold range, including: during strawberry picking, providing a target grip strength. Real-time monitoring of actual gripping force via force sensor Calculation error - Input the PID controller and calculate the adjusted grip force according to the PID control function. ;

[0037] in: Indicates initial grip strength. Indicates the grip strength of the target. Indicates actual grip strength. Represents the proportionality coefficient. Represents the integral coefficient. Represents the differential coefficient. Indicates error. Represents the integral term. Let represent the advantage of the differential term, and kp, ki, and kd be the proportional, integral, and differential coefficients, respectively.

[0038] The parameter optimization database records the optimal parameter combinations for different strawberry varieties, different light conditions, different ripening stages, different combinations of defect types, and different shooting angles and background interference. The parameters are fine-tuned regularly to continuously optimize performance.

[0039] The strawberry non-destructive sorting system based on multimodal fusion and force feedback control includes a vision acquisition device, a picking and sorting device, and a host computer. The host computer communicates with the vision acquisition device and the picking and sorting device via Ethernet or industrial bus.

[0040] The visual acquisition equipment includes an industrial depth camera and a high color rendering index LED ring light source, which work together to acquire strawberry images under the control of a host computer.

[0041] The host computer is equipped with a front-end human-computer interaction module, a control back-end, and a data management module;

[0042] The human-computer interaction module uses a human-computer interaction interface to interact with users to achieve manual review and parameter adjustment. Users input open-source datasets of strawberry images, as well as various parameters and formulas for image processing, multimodal feature fusion, and the control process of picking and sorting equipment, as well as control instructions for on-site visual acquisition equipment and picking and sorting equipment, and the results of image processing, multimodal feature fusion, and quality grading are displayed in graphical and chart form.

[0043] The control backend includes the following program modules: data acquisition module, image processing module, multimodal feature fusion module, and robotic arm control module;

[0044] The data acquisition module receives open-source datasets input by the human-computer interaction interface, or strawberry images captured on-site by a real-time camera of a visual acquisition device; the images contain various types of defects.

[0045] The image processing module performs preprocessing operations on the received strawberry image: background segmentation, image enhancement, and size normalization.

[0046] The multimodal feature fusion module is used for multimodal feature extraction, feature weighted fusion and scoring, as well as quality grading and location coordinates of strawberries;

[0047] The mechanical picking control module outputs position control commands to the encoders of each axis of the robotic arm based on the position coordinates of the target strawberry to be picked. This enables the robotic arm to adjust each axis to move to the target strawberry according to the planned path. It also calculates the opening angle and direction of the multi-finger gripper for strawberry picking.

[0048] The data management module is used to record data.

[0049] The picking and sorting equipment consists of a robotic arm installed on-site and a dynamically adjustable mechanical gripper at the end of the robotic arm. The gripper has a built-in pressure sensor, gripper controller, and adaptive flexible gripper. The force feedback module monitors the force feedback in real time and uploads it to the host computer. The gripper controller adjusts the gripping force, opening angle, and direction in real time according to the strawberry size, surface condition, and sorting instructions output by the host computer, so as to achieve non-destructive gripping of the strawberries.

[0050] The present invention has the following beneficial effects and advantages:

[0051] 1. Introducing a multimodal feature weighted fusion algorithm: Simultaneously fusing four features—color, edge, texture, and shape—to adapt to strawberry detection under different varieties and lighting conditions, improving the situation of single feature recognition and overcoming the problem of low target detection accuracy caused by ignoring multidimensional features.

[0052] 2. Simplify the image preprocessing process: retain the two core steps of background segmentation and image enhancement, and remove redundant illumination compensation and noise removal steps. While ensuring image quality, the computational complexity is greatly reduced and the real-time performance of the system is improved.

[0053] 3. Closed-loop feedback control algorithm is adopted: the gripping force is dynamically adjusted according to the size of the strawberries and real-time force feedback, which can effectively reduce the damage rate during the sorting process, especially the damage rate of overripe strawberries is reduced to below 2%, thus improving the sorting quality.

[0054] 4. Dynamic weight adjustment mechanism: The weights of each feature are dynamically adjusted based on the strawberry variety, lighting conditions and texture complexity through an attention mechanism, without the need for manual intervention, and the system has strong self-adaptability.

[0055] 5. Wide applicability: This method is not only applicable to the quality inspection and sorting of strawberries, but also to other small-target scenarios and the quality inspection and sorting of other fruits and vegetables. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the overall system structure and application scenarios.

[0057] Figure 2 This is a diagram of the overall system architecture.

[0058] Figure 3 This is a flowchart of the software processing.

[0059] Figure 4 This is a flowchart of a multimodal feature extraction and fusion algorithm.

[0060] Figure 5 This is a flowchart of closed-loop feedback control.

[0061] Figure 6 This is a schematic diagram of the image of the target strawberry at angle 1.

[0062] Figure 7 This is a schematic diagram of the image of the target strawberry at angle 2.

[0063] Figure 8 Example images for testing different varieties of strawberries.

[0064] Figure 9 This is a comparison chart showing the accuracy and speed of recognition after adding modules. Detailed Implementation

[0065] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0067] like Figure 1 The diagram shows the overall system structure and application scenario. This invention discloses a method for inspecting and sorting strawberry maturity and damage based on visual recognition. Through the collaborative work of a visual acquisition device and host computer software, it achieves automated detection and grading of strawberry appearance quality. The visual acquisition device is designed considering the size and irregular shape of strawberries, as well as the need for multi-angle imaging. A high-resolution camera with at least 5 megapixels is used to ensure clear capture of subtle damage and texture changes on the strawberry surface. The camera is equipped with a lens of suitable focal length, and the working distance is set within the range of 300-500mm according to the production line layout. The host computer software is mainly used to extract multimodal features, locate the strawberry's position coordinates, determine quality grading, and provide whether the picking meets the standards. The picking and sorting equipment includes a robotic arm and a dynamically adjustable robotic gripper at the end of the robotic arm. This robotic gripper has a built-in pressure sensor and an adaptive flexible gripper that can adjust the gripping force in real time according to the strawberry size, surface condition, and sorting instructions output by the host computer or controller, achieving non-destructive gripping of the strawberries. The controllable light source uses a high color rendering index (Ra ≥ 90) white LED ring light source with a color temperature set at 5500K–6500K to simulate natural sunlight and reduce color misjudgment caused by color shift due to lighting conditions. Simultaneously, the light source brightness can be adjusted via software to adapt to imaging needs under different ambient light conditions. Figure 2 The diagram shows the overall architecture of the strawberry quality inspection and sorting system of this invention, which mainly includes a visual acquisition device, a picking and sorting device, and a host computer. The host computer communicates with the visual acquisition device and the picking and sorting device via Ethernet or an industrial bus.

[0068] Furthermore, the visual acquisition module consists of on-site hardware, including an industrial depth camera and a high color rendering index LED ring light source. These two components work together and are controlled by a host computer to acquire images of strawberries. Specifically, the on-site location is a work platform with several strawberry plants or an actual strawberry field. A high-resolution industrial depth camera with at least 5 megapixels (capturing 3D image data including depth information) is used. The industrial depth camera is mounted around the work platform or the actual strawberry field, or installed on the robotic arm of the picking and sorting equipment. The industrial depth camera takes multi-angle shots within its working distance.

[0069] Furthermore, the host computer is equipped with a front-end human-computer interaction module, a control back-end, and a data management module.

[0070] The human-computer interaction module uses a user interface to interact with users for manual review and parameter adjustment. Users can input open-source datasets of strawberry images, as well as various parameters and formulas for image processing, multimodal feature fusion, and the control process of picking and sorting equipment. It also provides control commands for the on-site visual acquisition and picking and sorting equipment, and displays the results of image processing, multimodal feature fusion, and quality grading in graphical and chart formats. Furthermore, it allows for graphical visualization of twin simulations of the on-site picking and sorting equipment.

[0071] The control backend includes the following program modules: data acquisition module, image processing module, multimodal feature fusion module, and robotic arm control module.

[0072] Data acquisition module: This is a program module set up in the host computer control background. It can receive open-source datasets input by humans through the human-computer interaction interface, or strawberry images captured on-site by real-time cameras of visual acquisition devices; the images contain various types of defects.

[0073] Image processing module: This is a program module located in the host computer control background, responsible for preprocessing the received strawberry images. This includes background segmentation, image enhancement, and size normalization.

[0074] Multimodal feature fusion module: This is a program module set up in the host computer control background. It is responsible for multimodal feature extraction, feature weighted fusion and scoring, as well as quality grading and locating the strawberry's coordinates.

[0075] Mechanical picking control module: This is a program module set in the host computer control background. It is responsible for outputting position control commands to the encoders of each axis of the robotic arm according to the position coordinates of the target strawberry to be picked. This enables the robotic arm to adjust each axis to move to the target strawberry according to the planned path. It also calculates the opening angle and direction of the multi-finger gripper for strawberry picking.

[0076] Data Management Module: This is a program module set up in the host computer control backend, responsible for recording data.

[0077] Furthermore, the picking and sorting equipment consists of on-site hardware, including a robotic arm and a dynamically adjustable mechanical gripper at the end of the robotic arm, positioned on-site (at a workstation platform or in the strawberry field). This mechanical gripper incorporates a force feedback module (pressure sensor), a gripper controller, and an adaptive flexible gripper. The force feedback module monitors and transmits the force feedback data to a host computer in real time. The gripper controller adjusts the gripping force, opening angle, and direction in real time based on the strawberry size, surface condition, and sorting instructions output by the host computer, achieving non-destructive gripping of the strawberries. The robotic arm and mechanical gripper in this invention are existing technologies.

[0078] like Figure 3 As shown, this is the overall process for strawberry quality testing and sorting according to the present invention. Figure 4 The diagram shows the flowchart of the multimodal feature extraction and fusion algorithm. The method of this invention mainly includes the following 7 steps:

[0079] Step 1: Acquire images of the target strawberry

[0080] The target strawberry images in this example can be from open-source datasets or strawberry images taken by real-time cameras; ensure that the images collected in this step have multiple defect types (mechanical damage, fungal diseases, physiological defects, overripe, unripe) and that there is sufficient image data for each defect type.

[0081] The target strawberry image in this example can be obtained from an open-source dataset, i.e., a publicly available strawberry quality detection dataset.

[0082] When capturing strawberry images using a camera, appropriate camera parameters need to be adjusted, including: exposure time, acquisition frequency, resolution, and white balance; real-time multi-angle image capture of target strawberries using an industrial camera; technical parameters include:

[0083] Exposure time: Adjustable from 1 / 100 second to 1 / 1000 second, supports automatic exposure;

[0084] Image format: RGB 24-bit true color, resolution 2592×1944 or higher;

[0085] Acquisition frequency: no less than 10fps, supports triggered acquisition;

[0086] White balance: Automatic white balance or manual setting, color temperature error 98%.

[0087] During image acquisition, a high-resolution industrial camera with no less than 5 million pixels is used to take pictures from multiple angles within a working distance of 300-500mm, that is, at least 3 angles with equal intervals of 120°. A high color rendering index LED ring light source is used for illumination, with Ra≥90, color temperature 5500K-6500K, and dimming range of 0-100%.

[0088] Step 2: Image preprocessing; The strawberry image undergoes background segmentation, image enhancement, location coordinate positioning, and size normalization. Background segmentation uses the GrabCut algorithm, and image enhancement uses the CLAHE method. Details are as follows:

[0089] The GrabCut algorithm is used to distinguish between foreground and background.

[0090] The CLAHE (Contrast Limiting Adaptive Histogram Equalization) method can improve detail clarity, increase contrast by more than 30%, and preserve edge details while avoiding over-enhancement.

[0091] Position coordinate positioning: The three-dimensional position coordinates are located based on the edge contour of the enhanced image.

[0092] Image normalization processes normalize the image size to 224×224 and the pixel values ​​to [0,1].

[0093] Step 3: Parallel extraction of multimodal features;

[0094] A multimodal feature extraction branch is established to extract color features, edge features, texture features, and shape features in parallel, obtaining color similarity scores Sc, edge similarity scores Se, texture similarity scores St, and shape similarity scores Ss, respectively. Details are as follows:

[0095] 1) Color feature extraction;

[0096] Hue, saturation, and lightness are extracted from the HSV color space, and a color histogram (32×32×32) is calculated. Color moments (mean, variance, skewness, and kurtosis) are also calculated. This feature is mainly used to identify abnormal maturity conditions such as overripe (manifested as hue shift and reduced saturation) and underripe (manifested as excessively high lightness and insufficient saturation), as well as uneven color distribution caused by physiological defects.

[0097] 2) Edge feature extraction;

[0098] An adaptive Canny operator is used for edge detection. By setting a high-to-low threshold ratio of 2:1, adaptive calculations are performed, including: calculating contour integrity score, edge density, edge length, and edge curvature. This feature is mainly used to identify mechanical damage. When there is physical damage such as collision or squeezing on the strawberry surface, the edge contour will show discontinuities such as breaks and gaps, resulting in a significant decrease in contour integrity score.

[0099] 3) Texture feature extraction;

[0100] Thirteen texture features were extracted using the Gray-Level Co-occurrence Matrix (GLCM) in four directions: 0°, 45°, 90°, and 135°. These features included energy, contrast, entropy, correlation, homogeneity, dissimilarity, and second moment of angle. These features are primarily used to identify fungal diseases (such as gray mold and powdery mildew). Diseased areas typically exhibit increased texture roughness and decreased uniformity, specifically manifested as increased entropy and decreased energy. The system can also identify surface texture anomalies caused by physiological defects.

[0101] 4) Shape feature extraction;

[0102] Extract the outline area and perimeter of the strawberry, and calculate shape features such as aspect ratio, roundness, eccentricity, and rectangularity. These features are mainly used to comprehensively evaluate the overall morphological integrity of the strawberry, as mechanical damage is often accompanied by abnormal shape parameters (such as decreased roundness and increased eccentricity).

[0103] 5) Calculate the similarity score for each feature:

[0104] 5a) Color similarity score ;in, For HSV space Euclidean distance. The maximum possible distance ( );

[0105] 5b) Edge similarity score ;in, The maximum number of pixels at the edge of the connected region (after morphological closing operation). This represents the total number of edge pixels.

[0106] 5c) Texture similarity score cos(θ); Cosine similarity is used to calculate texture feature similarity, with a value range of [0,1], where θ is the angle between the strawberry texture feature vector to be tested and the standard texture feature vector;

[0107] 5d) Shape similarity score Shape feature matching degree, range [0,1].

[0108] The multimodal features extracted above are all directly related to the similarity scores of each feature calculated in step 4.

[0109] Step 4: Weighted fusion scoring;

[0110] 1) Weights are dynamically adjusted;

[0111] The "original model" refers to the basic strawberry quality detection model before the introduction of the multimodal feature weighted fusion algorithm. In this model, the weights of the four features—color, edge, texture, and shape—are fixed values ​​and cannot be adaptively adjusted according to different scenarios, which limits its detection accuracy in complex practical applications.

[0112] This invention introduces a multimodal feature weighted fusion algorithm into the original model to adapt to the detection of strawberries of different varieties and under different lighting conditions.

[0113] Through an attention mechanism, specifically a weight adjustment function, the weight parameters are dynamically adjusted based on the strawberry variety, lighting conditions, and texture complexity: color similarity weight. Edge similarity weight Texture similarity weight Shape similarity weight ;

[0114] Weight adjustment function:

[0115] Color similarity weighting function based on variety, lighting, and texture :

[0116] ;

[0117] Edge similarity weight adjustment function based on variety and illumination :

[0118] ;

[0119] Texture similarity weight adjustment function based on variety and texture :

[0120] ;

[0121] This leads to the shape similarity weight adjustment function. :

[0122] ;

[0123] in:

[0124] is the light intensity factor (range [0,1]);

[0125] Variety factor (range [0,1]);

[0126] is the texture complexity factor (range [0,1]);

[0127] 2) Overall similarity score .

[0128] Step 5: Quality grading determination;

[0129] Strawberry quality is graded based on overall similarity score; grading criteria:

[0130] Grade 1 (Premium): Score ≥ 0.90; sorted and harvested.

[0131] Grade II (Good): 0.75 ≤ Score ≤ 90%; sorting and picking.

[0132] Otherwise, it will be deemed unqualified and will not be picked.

[0133] Based on actual operating data, the weighting coefficients are automatically adjusted, namely color similarity weight Wc, edge similarity weight We, texture similarity weight Wt, shape similarity weight Ws, and threshold parameters, namely first-grade product judgment threshold 0.90, second-grade product judgment threshold 0.75, and feature thresholds for various defect recognition, such as contour integrity <0.6, texture entropy >0.85, saturation <0.4, etc.

[0134] Establish a parameter optimization database to record the optimal parameter combinations under different scenarios, namely different strawberry varieties (such as Hongyan, Zhangji, Sweet Charlie, etc.), different lighting conditions (such as cloudy / sunny days, changes in light source color temperature, and ambient light intensity), different ripening stages (unripe, moderately ripe, overripe), different combinations of defect types (single defect or compound defect), and different shooting angles and background interference.

[0135] We fine-tune the parameters weekly to continuously optimize system performance.

[0136] Defect type identification mechanism:

[0137] Mechanical damage: mainly identified by a significant decrease in contour integrity score (<0.6) and local edge density anomalies in edge features, manifested as edge breakage, gaps, and irregular contours.

[0138] Fungal diseases: mainly detected by significantly increased entropy (>0.85) and decreased energy (<0.15) in texture features, manifested as surface mold spots, gray mold, and increased texture roughness.

[0139] Physiological defects: judged by the combination of uneven saturation distribution (coefficient of variation > 0.3) and abnormal texture features (contrast > 0.6 or homogeneity < 0.4), which are manifested as abnormal surface color and texture caused by internal tissue lesions.

[0140] Overripe: Identified by hue value shift (H>20° or H<340°) and saturation reduction (S<0.4) in color features, manifested as a darker color that loses its bright red hue.

[0141] Unripe: This is determined by excessively high brightness (V>0.8) and insufficient saturation (S<0.3) in the color characteristics. It is characterized by a white or light pink color, lacking the deep red of ripe strawberries.

[0142] The system outputs the quality category of strawberries (Grade 1, Grade 2, Ungraded) and specific defect types (mechanical damage, fungal diseases, physiological defects, overripe, unripe).

[0143] Step 6: Robotic Arm Control Flow (Adaptive Grasping and Sorting):

[0144] like Figure 5 The diagram shown is a flowchart of closed-loop feedback control.

[0145] Based on the location of the target strawberry to be picked, position control commands are output to the encoders of each axis of the robotic arm. This enables the robotic arm to adjust each axis to move along the planned path from the start to the target strawberry, and to adjust the gripping force, opening angle and direction in real time to achieve non-destructive gripping of the strawberry.

[0146] During strawberry picking, a target grip strength is given. Real-time monitoring of actual gripping force via force sensor An error is generated for each crawling process. - The error is processed proportionally, integrally, and derivatively by the PID controller (which consists of the proportional, integral, and derivative terms shown in the diagram), thus obtaining the adjusted gripping force according to the PID control algorithm function. Finally, these adjusted gripping forces are fed into the gripping module for further adjustments. The core of closed-loop feedback control is the force feedback mechanism.

[0147] PID control function:

[0148]

[0149] in: This indicates the adjusted grip strength; Indicates initial grip strength; Indicates the grip strength of the target; Indicates actual grip strength; Indicates the proportionality coefficient; Indicates the integral coefficient; Represents the differential coefficient; Indicates error; Indicates the integral term; The differential term represents the advantage; where kp=0.5, ki=0.1, and kd=0.05, ensuring that the gripping force is dynamically adjusted within the range of 0.05N-0.2N. Closed-loop feedback control does not directly fix the gripping force, but adjusts it based on the error, reducing fluctuations and thus avoiding damage to strawberries of different ripeness and grasping failures. While the result obtained after closed-loop feedback control is not necessarily the globally optimal solution, it is more generalized than open-loop control, effectively preventing strawberry damage, and without increasing the algorithm's complexity.

[0150] Step 7: Use the manually reviewed and corrected parameters as optimization data, and continuously optimize system performance through adaptive parameter adjustment.

[0151] In practical applications, the detection process of this invention mainly includes the following steps:

[0152] Step 1: Image Input

[0153] The preprocessed image is then fed into the multimodal feature extraction module for feature extraction.

[0154] Step 2: Feature Extraction

[0155] The image input feature extraction module extracts color features, edge features, texture features, and shape features from the input image. This feature extraction module includes four processing methods: color space conversion, edge detection, texture analysis, and shape analysis.

[0156] Step 3: Feature Fusion

[0157] The feature vectors generated by the feature extraction module are input into the weighted fusion module for feature fusion. The weighted fusion module weights the feature vectors multiple times and fuses the features into the last layer. Each fused feature vector is scored independently, and finally a comprehensive similarity score is generated.

[0158] Step 4: Quality Inspection

[0159] To perform quality detection on strawberry images, a multimodal feature weighted fusion algorithm is introduced into the original model to adapt to the detection of strawberries of different varieties and under different lighting conditions.

[0160] Step 5: Grading Determination

[0161] The quality is graded based on the overall similarity score, outputting the strawberry's quality category (Grade 1, Grade 2, Substandard) and defect type (mechanical damage, fungal disease, physiological defect, overripe, unripe). Substandard strawberries are secondary products that do not meet the quality standards but are still usable.

[0162] Figure 6 and Figure 7 The two images marked with "maturity" and "abnormality" in the middle represent two angles of the same object being tested. Figure 8 To identify the annotation results. Figure 9 This is a comparison chart of the recognition accuracy and recognition speed of the method of the present invention. Green represents recognition accuracy, which is expressed as %, and black represents recognition speed, which is expressed as milliseconds.

[0163] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media containing computer-usable program code.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit its scope of protection. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading the present invention, they can still make various changes, modifications or equivalent substitutions to the specific implementation of the application, but these changes, modifications or equivalent substitutions are all within the scope of protection of the claims pending approval.

Claims

1. A non-destructive strawberry sorting method based on multimodal fusion and force feedback control, characterized in that, Includes the following steps: S1. Acquire strawberry images; S2. Perform background segmentation, image enhancement, location coordinate positioning, and size normalization on the strawberry image; S3. Establish a multimodal feature extraction branch to extract color features, edge features, texture features, and shape features in parallel, and obtain color similarity score Sc, edge similarity score Se, texture similarity score St, and shape similarity score Ss, respectively; wherein the color features are extracted in the HSV color space, the edge features are extracted using the adaptive Canny operator, the texture features are extracted using the gray-level co-occurrence matrix, and the shape features are extracted based on contour analysis. S4. Through a weighted attention mechanism, dynamically adjust each weight parameter based on strawberry variety, lighting conditions, and texture complexity: color similarity weight. Edge similarity weight Texture similarity weight Shape similarity weight The overall similarity score is calculated based on a weighted fusion algorithm. S5. Determine the quality grade of strawberries based on the comprehensive similarity score, and output the quality category and defect type of strawberries; S6. Based on the grading results, output instructions to control the gripper to perform adaptive gripping and sorting. The gripping force is dynamically adjusted according to the size of the strawberry and real-time force feedback. A PID control algorithm is used to achieve closed-loop feedback control to ensure that the gripping force is within the threshold range. S7. Establish a parameter optimization database, use manually reviewed and corrected parameters as optimization data, and continuously optimize through adaptive parameter adjustment.

2. The strawberry non-destructive sorting method based on multimodal fusion and force feedback control according to claim 1, characterized in that, The strawberry images used are from open-source datasets or real-time camera images; and the images contain various types of defects.

3. The strawberry non-destructive sorting method based on multimodal fusion and force feedback control according to claim 1, characterized in that, The GrabCut algorithm is used to distinguish between the foreground and background, and the CLAHE method is used for enhancement. The three-dimensional position coordinates are located based on the edge contour of the enhanced image, the image size is normalized to 224×224, and the pixel value is normalized to [0,1].

4. The strawberry non-destructive sorting method based on multimodal fusion and force feedback control according to claim 1, characterized in that, The similarity scores for each feature are calculated as follows: Extract hue, saturation, and lightness from the HSV color space; calculate the color histogram and color moments; and calculate the color similarity score. ;in, For HSV space Euclidean distance. The maximum possible distance; An adaptive Canny operator is used for edge detection, calculating contour integrity score, edge density, edge length, edge curvature, and edge similarity score. ;in, The maximum number of pixels at the edge of the connected region. This represents the total number of edge pixels. Multi-texture features, including energy, contrast, entropy, correlation, homogeneity, dissimilarity, and angular second moment, are extracted in four directions (0°, 45°, 90°, and 135°) using the gray-level co-occurrence matrix. Texture similarity scores are then calculated. cos(θ); Cosine similarity is used to calculate texture feature similarity, with a value range of [0,1], where θ is the angle between the strawberry texture feature vector to be tested and the standard texture feature vector; Shape features such as the outline area, perimeter, aspect ratio, roundness, eccentricity, and rectangularity of the strawberry are extracted to obtain a shape similarity score. Shape feature matching degree, range [0,1].

5. The strawberry non-destructive sorting method based on multimodal fusion and force feedback control according to claim 1, characterized in that, The weights are dynamically adjusted using the following three functions via an attention mechanism: Color similarity weighting function based on variety, lighting, and texture : ； Edge similarity weight adjustment function based on variety and illumination : ； Texture similarity weight adjustment function based on variety and texture : ； This leads to the shape similarity weight adjustment function. : ； in, is the light intensity factor (range [0,1]); Variety factor (range [0,1]); is the texture complexity factor (range [0,1]); The overall similarity score .

6. The strawberry non-destructive sorting method based on multimodal fusion and force feedback control according to claim 1, characterized in that, The quality grading is determined as follows: a score ≥ 0.90 indicates a Grade 1 product; a score ≤ 0.75 ≤ 90% indicates a Grade 2 product; otherwise, it is considered unqualified.

7. The strawberry non-destructive sorting method based on multimodal fusion and force feedback control according to claim 1, characterized in that, The step of outputting instructions based on the grading results to control the gripper for adaptive grasping and sorting, wherein the gripping force is dynamically adjusted according to the size of the strawberries and real-time force feedback, including: Based on the location of the target strawberry to be picked, position control commands are output to the encoders of each axis of the robotic arm, which enable the robotic arm to adjust each axis to move along the planned path from the start to the target strawberry, and to adjust the gripping force, opening angle and direction of the end gripper in real time to achieve non-destructive gripping of the strawberry. The PID control algorithm is used to achieve closed-loop feedback control, ensuring that the grip strength is within a threshold range, including: during strawberry picking, providing a target grip strength. Real-time monitoring of actual gripping force via force sensor Calculation error - Input the PID controller and calculate the adjusted grip force according to the PID control function. ; in: Indicates initial grip strength. Indicates the grip strength of the target. Indicates actual grip strength. Represents the proportionality coefficient. Represents the integral coefficient. Denotes the differential coefficient. Indicates error. Represents the integral term. Let represent the advantage of the differential term, and kp, ki, and kd be the proportional, integral, and differential coefficients, respectively.

8. The strawberry non-destructive sorting method based on multimodal fusion and force feedback control according to claim 1, characterized in that, The parameter optimization database records the optimal parameter combinations for different strawberry varieties, different light conditions, different ripening stages, different combinations of defect types, and different shooting angles and background interference. The parameters are fine-tuned regularly to continuously optimize performance.

9. A strawberry non-destructive sorting system based on multimodal fusion and force feedback control, characterized in that, It includes visual acquisition equipment, picking and sorting equipment, and host computer. The host computer communicates with the visual acquisition equipment and picking and sorting equipment via Ethernet or industrial bus. The visual acquisition equipment includes an industrial depth camera and a high color rendering index LED ring light source, which work together to acquire strawberry images under the control of a host computer. The host computer is equipped with a front-end human-computer interaction module, a control back-end, and a data management module; The human-computer interaction module uses a human-computer interaction interface to interact with users to achieve manual review and parameter adjustment. Users input open-source datasets of strawberry images, as well as various parameters and formulas for image processing, multimodal feature fusion, and the control process of picking and sorting equipment, as well as control instructions for on-site visual acquisition equipment and picking and sorting equipment, and the results of image processing, multimodal feature fusion, and quality grading are displayed in graphical and chart form. The control backend includes the following program modules: data acquisition module, image processing module, multimodal feature fusion module, and robotic arm control module; The data acquisition module receives open-source datasets input by the human-computer interaction interface, or strawberry images captured on-site by a real-time camera of a visual acquisition device; The image contains multiple defect types; The image processing module performs preprocessing operations on the received strawberry image: background segmentation, image enhancement, and size normalization. The multimodal feature fusion module is used for multimodal feature extraction, feature weighted fusion and scoring, as well as quality grading and location coordinates of strawberries; The mechanical picking control module outputs position control commands to the encoders of each axis of the robotic arm based on the position coordinates of the target strawberry to be picked. This enables the robotic arm to adjust each axis to move to the target strawberry according to the planned path. It also calculates the opening angle and direction of the multi-finger gripper for strawberry picking. The data management module is used to record data.

10. The strawberry non-destructive sorting system based on multimodal fusion and force feedback control according to claim 9, characterized in that, The picking and sorting equipment consists of a robotic arm installed on-site and a dynamically adjustable mechanical gripper at the end of the robotic arm. The gripper has a built-in pressure sensor, gripper controller, and adaptive flexible gripper. The force feedback module monitors the force feedback in real time and uploads it to the host computer. The gripper controller adjusts the gripping force, opening angle, and direction in real time according to the strawberry size, surface condition, and sorting instructions output by the host computer, so as to achieve non-destructive gripping of the strawberries.

Images