A method for fruit quality testing, a robot, and terminal equipment

Abstract

This invention provides a fruit quality inspection method, robot, and terminal equipment. The method, applied to a fruit quality inspection robot, specifically involves scanning the plant canopy to obtain canopy images, identifying target fruits, and adjusting the robotic arm's acquisition posture to achieve target fruit recognition and localization. A vision system acquires images of the target fruit's appearance, enabling detection of external surface features. A spectral acquisition system collects spectral data of the target fruit, enabling internal quality inspection. The inspection is continuous, covering the entire growth stage of the cherry tomato plant. Furthermore, a deep learning model is used for fruit quality assessment, ensuring rapid response and completion of quality inspection tasks in orchard environments. Therefore, this invention enables efficient, precise, and automated monitoring of fruit growth stages in orchards, improving overall fruit quality and market competitiveness, and promoting agricultural mechanization and automation.

Description

Technical Field

[0001] This invention relates to the field of fruit quality testing technology, and more specifically, to a fruit quality testing method, a robot, and a terminal device. Background Technology

[0002] With the rapid development of the global fruit industry, improving fruit quality has become a key factor in driving market competitiveness. Currently, fruit quality assessment mainly relies on manual labor, and is largely concentrated on post-harvest sorting, lacking real-time monitoring of the growth stages.

[0003] In recent years, intelligent agricultural equipment has been gradually applied to fruit cultivation, including harvesting robots and sorting robots. However, research on non-destructive testing for fruit growth stages remains relatively weak. Traditional monitoring methods not only consume a lot of time and manpower but also rely on experience-based judgment, lack scientific basis, and are prone to misjudgment.

[0004] Although drone remote sensing technology has been applied in agricultural monitoring, due to the wide distribution of fruits and their influence by natural factors such as sunlight and rainfall, and the fact that fruits cover the entire canopy and branches of fruit trees during their growth, drones cannot monitor the entire appearance of fruits like they can monitor crops. Therefore, single remote sensing technology cannot meet the needs of full-process monitoring of fruit quality.

[0005] Therefore, developing a non-destructive testing system based on robotics to accurately assess the quality of fruit at different growth stages is an effective way to solve the above problems. Summary of the Invention

[0006] The main objective of this invention is to provide a method, robot, and terminal equipment for fruit quality testing, aiming to solve the technical problem of insufficient quality monitoring during the fruit growth stage in the prior art.

[0007] In a first aspect, the present invention provides a fruit quality detection method applied to a fruit quality detection robot. The fruit quality detection robot includes a mobile chassis, a robotic arm, a navigation system, a vision system, and a spectral acquisition system. The robotic arm and the navigation system are mounted on the mobile chassis, and the vision system and the spectral acquisition system are mounted on the robotic arm. The detection method includes:

[0008] Based on navigation planning and orchard environment data collected by the navigation system, the mobile chassis is controlled to move towards the target detection area;

[0009] Upon reaching the target detection area, the robotic arm is controlled to adjust to the pre-detection state and scans the plant canopy through the vision system to obtain a canopy image.

[0010] Based on the canopy image, the target fruit is identified and the robotic arm's acquisition posture is adjusted.

[0011] The vision system is controlled to acquire images of the fruit appearance of the target fruit.

[0012] The system is controlled to acquire the spectral data of the target fruit.

[0013] The fruit appearance image is analyzed based on the first deep learning model, the surface features of the target fruit are identified and the quality is evaluated to obtain the external physical quality score of the target fruit.

[0014] The fruit spectral data is analyzed based on a second deep learning model to predict the internal quality score of the target fruit.

[0015] The quality test results of the target fruit are obtained by combining the external physical quality score and the internal quality score of the fruit.

[0016] Furthermore, the vision system includes a robotic arm depth camera, and the process of identifying target fruits and adjusting the robotic arm's acquisition posture based on the canopy image includes:

[0017] The canopy image is input into a trained YOLO object detection model to identify the target fruit in the canopy image and generate the bounding box of the target fruit and the corresponding confidence information.

[0018] Calculate the center point of the bounding box of the target fruit to obtain the image coordinates of the target fruit in the canopy image;

[0019] Based on the image coordinates of the center point, the depth information of the canopy image, and the calibration parameters of the robotic arm, coordinate transformation is performed on the coordinate system of the robotic arm, and the motion trajectory of the robotic arm is generated.

[0020] The robotic arm's acquisition posture is adjusted according to its movement trajectory.

[0021] Further, the step of analyzing the fruit appearance image based on the first deep learning model, identifying the fruit surface features of the target fruit and performing quality assessment to obtain the external physical quality score of the target fruit includes:

[0022] The fruit appearance image is input into the trained YOLO object detection model to automatically identify and locate the target fruit in the fruit appearance image;

[0023] The surface features of the target fruit are extracted using image segmentation technology, and the color distribution is statistically analyzed using color analysis to evaluate the ripeness and color difference of the target fruit.

[0024] The defect features of the target fruit are extracted and quantitatively analyzed to calculate the defect area and distribution.

[0025] The external physical quality score of the target fruit is obtained based on its maturity, color difference, defect area and distribution.

[0026] Furthermore, the step of analyzing the fruit spectral data based on the second deep learning model to predict the internal quality score of the target fruit includes:

[0027] The spectral data of the fruit are identified by band selection and feature extraction algorithms to identify spectral characteristic peaks related to the internal quality of the fruit.

[0028] The spectral feature peaks are input into a trained fruit internal quality prediction model to predict the internal quality score of the target fruit.

[0029] Furthermore, the detection method further includes:

[0030] Edge computing technology is used to preprocess the fruit appearance image;

[0031] The spectral data of the fruit were preprocessed using standard normal transformation and multivariate scattering correction.

[0032] Furthermore, the detection method further includes:

[0033] Based on the fruit appearance image and the fruit spectral data, a growth trend prediction model is constructed using a long short-term memory network and a temporal convolutional network. The growth trend prediction model can use historical fruit growth data to establish a non-linear time series relationship and predict the future ripening time, final quality and potential disease risk of the fruit.

[0034] Secondly, the present invention provides a fruit quality inspection robot, employing a fruit quality inspection method as described in the first aspect. The fruit quality inspection robot includes: a mobile chassis, a robotic arm, a navigation system, a vision system, a spectral acquisition system, and a main control system. The robotic arm, the navigation system, and the main control system are mounted on the mobile chassis, and the vision system and the spectral acquisition system are mounted on the robotic arm. The main control system is used for:

[0035] Based on navigation planning and orchard environment data collected by the navigation system, the mobile chassis is controlled to move towards the target detection area;

[0036] Upon reaching the target detection area, the robotic arm is controlled to adjust to the pre-detection state and scans the plant canopy through the vision system to obtain a canopy image.

[0037] Based on the canopy image, the target fruit is identified and the robotic arm's acquisition posture is adjusted.

[0038] The vision system is controlled to acquire images of the fruit appearance of the target fruit.

[0039] The system is controlled to acquire the spectral data of the target fruit.

[0040] The fruit appearance image is analyzed based on the first deep learning model, the surface features of the target fruit are identified and the quality is evaluated to obtain the external physical quality score of the target fruit.

[0041] The fruit spectral data is analyzed based on a second deep learning model to predict the internal quality score of the target fruit.

[0042] The quality test results of the target fruit are obtained by combining the external physical quality score and the internal quality score of the fruit.

[0043] Furthermore, the fruit quality inspection robot also includes a light-shielding structure, and the spectral acquisition system includes a visible-near-infrared spectrometer. Both the visible-near-infrared spectrometer and the light-shielding structure are located at the end of the robotic arm, and the light-shielding structure is located directly in front of the acquisition of the visible-near-infrared spectrometer.

[0044] Furthermore, the navigation system includes a lidar, a forward-looking depth camera, and an inertial measurement unit; the mobile chassis includes a tracked chassis; and the fruit quality inspection robot also includes a device voltage conversion module, a device charging interface, a power display module, and a chassis communication interface mounted on the tracked chassis.

[0045] Thirdly, the present invention provides a terminal device, comprising: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement a fruit quality detection method as described in the first aspect.

[0046] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0047] This invention integrates visual recognition, spectral acquisition, image processing, and deep learning technologies. Specifically, it scans the plant canopy to obtain canopy images, then identifies target fruits based on these images and adjusts the robotic arm's acquisition posture to achieve target fruit recognition and localization. The vision system acquires images of the target fruit's appearance, enabling the detection of external features (such as pests and diseases, surface defects, etc.), while the spectral acquisition system collects spectral data to assess internal fruit quality (such as sugar content, acidity, maturity, etc.). This monitoring is continuous, covering the entire growth stage of the cherry tomato plant. Furthermore, a deep learning model is used for fruit quality assessment, ensuring rapid response and completion of quality inspection tasks in orchard environments. Additionally, the robot can flexibly adjust its working mode according to different operational conditions, ensuring stable operation in various environments. Therefore, this invention can efficiently and accurately monitor the fruit growth stages in orchards, improving the overall quality and market competitiveness of fruits, and promoting the development of agricultural mechanization and automation. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the structure of a fruit quality inspection robot provided in an embodiment of the present invention;

[0049] Figure 2 This is a structural schematic diagram from another perspective of a fruit quality inspection robot provided in one embodiment of the present invention;

[0050] Figure 3 This is a partial structural schematic diagram of a fruit quality inspection robot provided in one embodiment of the present invention;

[0051] Figure 4 This is a flowchart illustrating a fruit quality testing method according to an embodiment of the present invention;

[0052] Figure 5 This is a schematic diagram illustrating the process of a fruit quality inspection robot performing fruit quality inspection tasks according to an embodiment of the present invention;

[0053] Figure 6 This is a flowchart illustrating a depth vision-based robotic arm-spectrometer data acquisition posture algorithm according to an embodiment of the present invention.

[0054] Figure 7 This is a flowchart illustrating a multimodal fruit quality assessment algorithm based on spectral and visual fusion provided in an embodiment of the present invention.

[0055] Figure 8 This is a schematic diagram of the structure of a terminal device provided in an embodiment of the present invention.

[0056] in:

[0057] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings.

[0058] Explanation of reference numerals in the attached figures:

[0059] 1. Mobile chassis; 11. Chassis tracks; 12. Track drive wheel; 13. Track driven wheel; 21. Vision system; 22. Spectral acquisition system; 23. Robotic arm; 24. Robotic arm end effector; 3. Navigation system; 31. LiDAR; 32. Forward-looking depth camera; 33. Inertial measurement unit; 4. Main control system; 41. Equipment voltage conversion module; 42. Equipment charging interface; 43. Power display module; 44. Chassis communication interface; 5. Sunshade structure. Detailed Implementation

[0060] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0061] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0062] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection, a direct connection, or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0063] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0064] Please see Figure 1 , Figure 1 This is a schematic diagram of the structure of a fruit quality inspection robot provided in an embodiment of the present invention.

[0065] This invention discloses a fruit quality inspection robot for non-destructive quality inspection of fruits during their growth stages. Specifically, it combines navigation technology, visual recognition, spectral acquisition, and edge computing technology to automatically navigate within orchards and accurately monitor the growth status and quality characteristics of fruits. The fruits include cherry tomatoes.

[0066] Specifically, refer to Figure 1-3 The fruit quality inspection robot includes: a mobile chassis 1, a robotic arm 23, a navigation system 3, a vision system 21, and a spectral acquisition system 22. The robotic arm 23 and the navigation system 3 are mounted on the mobile chassis 1, and the vision system 21 and the spectral acquisition system 22 are mounted on the robotic arm 23.

[0067] Reference Figure 1-2 The mobile chassis 1 includes a tracked chassis, which includes chassis tracks 11, track drive wheels 12, and track driven wheels 13. The navigation system 3 includes a lidar 31, a forward-looking depth camera 32, and an inertial measurement unit (IMU) 33.

[0068] The tracked chassis utilizes differential control, ensuring the robot's stability and flexibility in complex orchard terrain, especially on muddy or uneven surfaces. Path planning and intelligent obstacle avoidance algorithms ensure stable movement in complex orchard environments, achieving precise positioning and autonomous navigation.

[0069] The robotic arm 23 includes a six-degree-of-freedom robotic arm. The six-degree-of-freedom robotic arm has high-precision motion control capabilities and can accurately locate the coordinates of the target fruit and perform tasks based on the position data provided by the vision system 21.

[0070] The vision system 21 includes a robotic arm depth camera. The robotic arm depth camera is located at the end of the robotic arm 23. To achieve accurate monitoring of fruit, the vision system 21 of this embodiment acquires canopy images and fruit appearance images through the robotic arm depth camera, and combines this with deep learning algorithms for target detection, fruit recognition, and localization. Image preprocessing and denoising algorithms effectively remove the influence of light and shadow in the environment, improving recognition accuracy. Utilizing the optimized target detection algorithm, the robot can quickly and accurately identify target fruits in complex environments, providing accurate location information for subsequent quality inspection.

[0071] In one specific embodiment, both the forward-looking depth camera 32 and the robotic arm depth camera include a binocular stereo camera.

[0072] The spectral acquisition system 22 includes a visible-near-infrared spectrometer, i.e., a VIS-NIR spectrometer. It analyzes the internal quality characteristics of fruits, such as sugar content, ripeness, and soluble solids content, by acquiring spectral data. The spectral data is processed using techniques such as standard normal transformation (SNV) and multivariate scattering correction (MSC) to reduce the impact of noise and ensure the accuracy of the test results.

[0073] This invention combines deep learning algorithms to establish a fruit quality prediction model based on fruit appearance images and fruit spectral data, which can evaluate fruit quality in real time and provide accurate data support for orchard management.

[0074] Specifically, the robot's fruit quality inspection system consists of a VIS-NIR spectrometer, a robotic arm depth camera, control and drive circuits, and data transmission links. It is used to collect the fruit's reflectance spectral data and appearance image information. The spectrometer works by irradiating the fruit surface with a light source of a specific wavelength to obtain the reflectance spectrum, and then analyzing the fruit's quality based on different spectral characteristics. After preprocessing, the spectral data is processed in real-time by an edge computing unit, and machine learning algorithms are used to predict and analyze the fruit's internal quality parameters such as sugar content and vitamin C content. After preprocessing the fruit surface image information, deep learning algorithms are used to analyze its surface condition, and the fruit's quality is determined by the combined internal and external quality assessments. The robotic arm depth camera and spectrometer work in conjunction with the robotic arm 23. The robotic arm 23 can rotate around its end-effector rotation axis 24 to change the position of the VIS-NIR spectrometer and depth camera mounted on it, enabling accurate non-destructive testing of the target fruit and ensuring that the testing process does not affect the fruit's quality.

[0075] Reference Figure 3In one specific embodiment, the fruit quality inspection robot further includes a light-shielding structure 5. Both the visible light-near infrared spectrometer and the light-shielding structure 5 are disposed at the end of the robotic arm 23, and the light-shielding structure 5 is located directly in front of the acquisition of the visible light-near infrared spectrometer.

[0076] In one specific embodiment, the light-shielding structure 5 includes a light-shielding silicone for the spectral acquisition head.

[0077] Reference Figure 2 In one specific embodiment, the fruit quality inspection robot further includes a device voltage conversion module 41, a device charging interface 42, a power display module 43, and a chassis communication interface 44, all mounted on the tracked chassis.

[0078] In addition, the fruit quality robot can change its end effector according to different operating conditions, extending the robot's applicability throughout the fruit growth cycle, and utilizing the characteristics of different sensors to collect the required data more accurately.

[0079] The present invention provides a fruit quality testing method, which is applied to the fruit quality testing robot described in the above embodiments, and can perform non-destructive quality testing on fruit during its growth stages.

[0080] Please see Figure 4 , Figure 4 This is a schematic flowchart of a fruit quality testing method according to an embodiment of the present invention. The testing method includes the following steps:

[0081] S100: Based on navigation planning and orchard environment data collected by the navigation system 3, control the mobile chassis 1 to move to the target detection area.

[0082] Reference Figure 5 When the fruit quality inspection robot needs to perform fruit (such as cherry tomatoes) quality inspection, the cherry tomato quality inspection program is started automatically or manually. The robot starts from the starting point, goes to the sampling area or the user-preset area, plans its path along the rows of orchards and moves to the target inspection area.

[0083] Navigation System 3 acquires data from sensors in real time and combines it with SLAM technology to dynamically adjust the path and autonomously avoid obstacles, ensuring that the robot can drive efficiently and autonomously in the orchard and complete its mission objectives.

[0084] S200. After reaching the target detection area, control the robotic arm 23 to adjust to the pre-detection state, and scan the plant canopy through the vision system 21 to obtain a canopy image.

[0085] After the robot arrives at the target orchard area, the robotic arm 23 adjusts to the pre-detection action, scans the canopy structure of fruit (such as cherry tomatoes) plants through the robotic arm depth camera of the vision system 21, locates the current fruit tree number by combining the pre-stored orchard map, and uploads the canopy image information to the central server in real time.

[0086] S300. Based on the canopy image, identify the target fruit and adjust the acquisition posture of the robotic arm 23.

[0087] The robot uses canopy images to identify the precise location of the fruit and adjusts the position of the end effector of the robotic arm 23 in real time.

[0088] S400, Control the vision system 21 to acquire the fruit appearance image of the target fruit;

[0089] S500: Control the spectral acquisition system 22 to acquire the fruit spectral data of the target fruit.

[0090] Taking cherry tomatoes as an example, the VIS-NIR spectrometer probe of the spectral acquisition system 22 is aligned with the cherry tomato fruit, ensuring close contact between the spectrometer and the fruit. Simultaneously, scanning is initiated to collect spectral data of the cherry tomato's surface and interior, while a camera simultaneously captures images of the fruit's appearance (including color, shape, and texture). The collected data undergoes real-time preprocessing (such as noise reduction and normalization) via an edge computing module.

[0091] S600. Analyze the fruit appearance image based on the first deep learning model, identify the fruit surface features of the target fruit and perform quality assessment to obtain the external physical quality score of the target fruit.

[0092] S700. Analyze the spectral data of the fruit based on the second deep learning model to predict the internal quality score of the target fruit.

[0093] The robot analyzes fruit appearance image data based on a first deep learning model (such as the YOLO object detection model) to identify features such as fruit color, surface defects, and pests and diseases; it simultaneously analyzes fruit spectral data and uses a second deep learning model (such as the PLS or SVM model) to predict internal quality indicators such as fruit sugar content and vitamin content.

[0094] S800. By combining the external physical quality score and the internal quality score of the target fruit, the quality detection result of the target fruit is obtained.

[0095] By comprehensively analyzing the internal components and external characteristics of fruits, real-time non-destructive testing of fruit quality can be achieved.

[0096] Reference Figure 5After a single plant is inspected, the robot moves to the next target plant according to the path planned by the navigation system 3, repeating steps S200 to S800 until the inspection procedure is completed.

[0097] After completing the inspection task, the robot returns to the charging base station and uploads the complete inspection data (including spectral data, image analysis results, and quality assessment report) to the cloud server via wireless network. This simultaneously updates the orchard database and generates visual reports. Once the data upload is complete, the robot automatically docks with the charging station to charge. After charging is finished, it enters a low-power sleep mode, awaiting the next task command to wake it up.

[0098] A fruit quality inspection robot according to an embodiment of the present invention further includes a main control system 4, the main control system 4 being used for:

[0099] Based on navigation planning and orchard environment data collected by the navigation system 3, the mobile chassis 1 is controlled to move to the target detection area;

[0100] Upon reaching the target detection area, the robotic arm 23 is controlled to adjust to the pre-detection state and scans the plant canopy through the vision system 21 to obtain a canopy image;

[0101] Based on the canopy image, the target fruit is identified and the acquisition posture of the robotic arm 23 is adjusted;

[0102] The vision system 21 is controlled to acquire images of the fruit appearance of the target fruit.

[0103] The spectral acquisition system 22 is controlled to acquire the spectral data of the target fruit.

[0104] The fruit appearance image is analyzed based on the first deep learning model, the surface features of the target fruit are identified and the quality is evaluated to obtain the external physical quality score of the target fruit.

[0105] The fruit spectral data is analyzed based on a second deep learning model to predict the internal quality score of the target fruit.

[0106] The quality test results of the target fruit are obtained by combining the external physical quality score and the internal quality score of the fruit.

[0107] In this embodiment, the main control system 4 can execute all the steps and functions of a fruit quality detection method provided in any embodiment. The specific functions of the main control system 4 will not be described in detail here.

[0108] In order to accurately obtain the quality status of fruit at different growth stages, a fruit quality detection method according to an embodiment of the present invention mainly consists of the following two algorithms: a robot arm-spectrometer data acquisition posture algorithm based on depth vision and a multimodal fruit quality assessment algorithm based on spectral and visual fusion.

[0109] Specifically, the posture algorithm for robotic arm-spectrometer data acquisition based on depth vision is as follows:

[0110] In one specific embodiment, step S300, which identifies the target fruit and adjusts the acquisition posture of the robotic arm 23 based on the canopy image, includes the following steps:

[0111] S310. Input the canopy image into the trained YOLO object detection model to identify the target fruit in the canopy image and generate the bounding box of the target fruit and the corresponding confidence information.

[0112] S320. Calculate the center point of the bounding box of the target fruit to obtain the image coordinates of the target fruit in the canopy image;

[0113] S330. Based on the image coordinates of the center point, the depth information of the canopy image, and the calibration parameters of the robotic arm 23, perform coordinate transformation on the coordinate system of the robotic arm 23 and generate the motion trajectory of the robotic arm 23.

[0114] S340. Adjust the acquisition posture of the robotic arm 23 according to the movement trajectory of the robotic arm 23.

[0115] Spectroscopic detection utilizes a light source emitting light of different wavelengths, and obtains intrinsic information about a substance by detecting the reflectance spectral response after interaction with the sample. The spectrometer used in this invention covers a wavelength range of approximately 200 to 1700 nm. The light source employs electrically pulsed modulated LEDs and OLEDs to achieve better uniformity and reproducibility; furthermore, ambient noise can be removed through Fourier transform signal filtering. To eliminate ambient light interference and improve the signal-to-noise ratio and measurement accuracy, detection is typically performed in a dark chamber.

[0116] Because spectral detection requires a dark box environment, to improve the accuracy of spectral data, the instrument needs to be placed close to the fruit to create a dark box detection state. Therefore, this embodiment proposes a depth vision-based robotic arm-spectrometer data acquisition posture algorithm. This algorithm needs to be used in conjunction with the robotic arm 23, a depth camera, the spectrometer located at the end of the robotic arm 23, and its light-shielding structure 5. The positions of each device at the end of the robotic arm 23 are as follows: Figure 3 As shown.

[0117] Therefore, when conducting fruit quality testing, the robotic arm 23 will automatically adjust its acquisition posture based on the real-time canopy image data and the fruit position, so that the spectrometer can be precisely aligned with the surface of the target fruit, thereby enabling efficient and non-destructive acquisition of spectral data.

[0118] Specifically, the fruit quality inspection robot, as an automated quality inspection system applied to the fruit growth stage, employs a depth vision-based robotic arm-spectrometer data acquisition posture algorithm. The specific process is as follows: Figure 6 As shown:

[0119] S6-1 Image Acquisition Stage: The system uses a robotic arm depth camera to acquire canopy images and corresponding depth data in real time. At the same time, the acquired images are preprocessed (such as noise reduction and image enhancement) to ensure that clearer and higher quality image input can be obtained in the subsequent target detection stage.

[0120] S6-2 Target Detection Stage: The preprocessed image is input into the trained YOLO target detection model. The model can quickly and accurately detect targets in the canopy image (i.e., target detection fruits) and generate bounding boxes of the targets and corresponding confidence information to ensure that the position and size of each target are effectively identified.

[0121] S6-3 Target Center Point Extraction Stage: The system calculates the center point of the bounding box for each detected target to obtain the precise location of the object in the canopy image, and filters and verifies multiple targets to ensure that the selected target is the best object to be operated on.

[0122] S6-4 Coordinate Transformation Stage: The system uses the intrinsic and extrinsic parameters of the robotic arm's depth camera to convert the image coordinates and depth information of the target center point into corresponding three-dimensional spatial coordinates. Then, combined with the calibration parameters of the robotic arm 23, these coordinates are corrected so that the transformed coordinates are perfectly aligned with the coordinate system of the robotic arm 23, laying the foundation for subsequent operations.

[0123] S6-5 Command Sending and Motion Planning Stage: The converted and corrected target coordinates are transmitted to the control system of the robotic arm 23. The control system formulates a suitable motion trajectory based on these coordinates and commands the robotic arm 23 to start moving, ensuring that the robotic arm 23 can move to the predetermined target position quickly and accurately.

[0124] S6-6 Real-time Feedback and Dynamic Correction Phase: To ensure the VIS-NIR spectrometer is correctly aligned with the target (i.e., the fruit being detected), the robotic arm 23 acquires new depth canopy images and target data in real time during its movement. Continuous target detection and coordinate transformation monitor the dynamic changes in the target position, and the movement trajectory is finely adjusted based on real-time data to ensure the robotic arm 23 maintains optimal alignment with the target throughout its movement. In the end-effector attitude adjustment and data acquisition phase, as the robotic arm 23 approaches the target area, a final fine-tuning of its attitude is performed to ensure the end-effector (spectrometer) is closely aligned with the target. Based on this, the spectral data acquisition task is initiated. After spectral acquisition, the spectral data is automatically compared with historical data and a signal-to-noise ratio analysis is performed to ensure the accuracy of the data acquisition.

[0125] The multimodal fruit quality assessment algorithm, which integrates spectral and visual information, comprehensively analyzes the internal components and external characteristics of fruits through multimodal fusion of spectral and visual information. This enables real-time, non-destructive fruit detection. Furthermore, it combines prediction algorithms to forecast growth conditions and large language models to provide water and fertilizer recommendations for subsequent planting stages. The algorithm flow is as follows: Figure 7 As shown.

[0126] The external characteristics of fruit mainly include the color, defects, diseases and pests on the surface of the fruit, while the internal characteristics refer to the internal components of the fruit, such as sugar content, vitamin C content and other quality parameters.

[0127] In one specific embodiment, the detection method further includes the following steps:

[0128] S610. The fruit appearance image is preprocessed using edge computing technology.

[0129] The fruit appearance images are preprocessed in real time by the robot's edge computing module (such as cropping, color correction, noise reduction, and normalization).

[0130] In one specific embodiment, step S600 analyzes the fruit appearance image based on a first deep learning model, identifies the fruit surface features of the target fruit, performs quality assessment, and obtains the external physical quality score of the target fruit, including the following steps:

[0131] S620. Input the fruit appearance image into the trained YOLO target detection model to automatically identify and locate the target fruit in the fruit appearance image;

[0132] S630. Extract the surface features of the target fruit using image segmentation technology, and combine color analysis to statistically analyze color distribution to evaluate the ripeness and color difference of the target fruit.

[0133] S640. Extract and quantify the defect features of the target fruit, and calculate the defect area and distribution;

[0134] S650. Based on the maturity and color difference, defect area and distribution of the target fruit, obtain the external physical quality score of the target fruit.

[0135] In this embodiment, cherry tomatoes are used as an example for target detection. The present invention first uses a high-resolution depth camera to acquire fruit appearance images of cherry tomatoes, and performs a series of preprocessing operations on the fruit appearance images, such as cropping, color correction and noise reduction, to improve the accuracy of subsequent analysis.

[0136] Subsequently, the pre-processed fruit appearance image is input into the pre-trained YOLO object detection model to achieve automatic recognition and accurate localization of the fruit region.

[0137] Next, image segmentation technology was used to extract surface features of the fruit, and color analysis methods were combined to statistically analyze the distribution of major colors to assess the maturity and color difference of the fruit. Simultaneously, for defects on the fruit surface such as cracks, spots, diseases, and pests, feature extraction and quantitative analysis methods were used to calculate their area and distribution, thus forming a comprehensive scoring index for evaluating the external physical quality of cherry tomatoes.

[0138] In one specific embodiment, the detection method further includes the following steps:

[0139] S710. The fruit spectral data are preprocessed using standard normal transformation and multivariate scattering correction methods.

[0140] The spectrometer scans the fruit in the short ultraviolet to near-infrared band to collect its reflectance spectral data. In the data processing, this embodiment employs Standard Normal Transform (SNV) and Multivariate Scatter Correction (MSC) methods to preprocess the raw fruit spectral data, eliminating noise interference, improving the spectral signal-to-noise ratio, and standardizing the data.

[0141] In one specific embodiment, step S700 analyzes the fruit spectral data based on a second deep learning model to predict the internal quality score of the target fruit, including the following steps:

[0142] S720. Identify the spectral data of the fruit through band selection and feature extraction algorithms to identify spectral characteristic peaks related to the internal quality of the fruit.

[0143] S730. Input the spectral feature peaks into the trained fruit internal quality prediction model to predict the fruit internal quality score of the target fruit.

[0144] Before steps S720-S730, quality characteristics such as sugar content, soluble solids, and maturity of the fruit are extracted using spectral data and deep learning technology to establish a non-destructive testing model. Specifically, spectral feature peaks related to the internal quality of the fruit are identified through band selection and feature extraction algorithms. These feature peaks represent important quality indicators such as sugar content and vitamin C content. Next, traditional spectral analysis methods such as principal component analysis (PCA), partial least squares regression (PLSR), and convolutional neural networks (CNN) are combined to analyze the spectral data and construct a fruit internal quality prediction model. Through training and optimization, this model can accurately map spectral features to specific quality parameters to achieve non-destructive testing.

[0145] Therefore, in this embodiment, the spectral data of the target fruit will be identified by band selection and feature extraction algorithms to identify the spectral feature peaks related to the internal quality of the fruit. Then, the spectral feature peaks will be input into the trained fruit internal quality prediction model to predict the internal quality score of the target fruit.

[0146] Taking cherry tomatoes as an example, the sugar content and vitamin C models used in this embodiment have an accuracy of over 98% and an R² coefficient of determination of over 0.92 after thousands of training iterations. Therefore, the fruit internal quality prediction model can accurately perform non-destructive detection of the internal characteristics of cherry tomatoes.

[0147] In one specific embodiment, the detection method further includes the following steps:

[0148] S800. Based on the fruit appearance image and the fruit spectral data, a growth trend prediction model is constructed using a long short-term memory network and a temporal convolutional network. The growth trend prediction model can use historical fruit growth data to establish a nonlinear time series relationship and predict the future ripening time, final quality, and potential disease risks of the fruit.

[0149] Reference Figure 7 In addition to achieving real-time non-destructive detection of fruit quality, this embodiment further incorporates a prediction algorithm to predict the growth status of the fruit.

[0150] Specifically, based on multi-temporal spectral and visual data, this embodiment employs a combination of Long Short-Term Memory (LSTM) and Temporal Convolutional Network (TCN) to construct a growth trend prediction model. This model can utilize historical growth data, including fruit color changes, sugar accumulation trends, and disease development, to establish non-linear time-series relationships and predict key parameters for future growth stages, such as ripening time, final quality level, and potential disease risk. Through dynamic model updates, the prediction parameters can be adjusted for different planting environments and management practices, thereby improving the adaptability and accuracy of the predictions.

[0151] In one specific embodiment, the detection method further includes the following steps:

[0152] S900 introduces a large language model to provide water and fertilizer management suggestions for subsequent fruit planting stages.

[0153] Reference Figure 7 This embodiment also introduces a Large Language Model (LLM) to provide water and fertilizer management recommendations for subsequent planting stages. Combining fruit growth prediction results with environmental parameters such as temperature and light intensity, the LLM can generate personalized water and fertilizer management plans based on a large amount of agricultural data and expert knowledge.

[0154] For example, in the early stages of growth, if a rapid fruit enlargement rate and low sugar accumulation are predicted, the big data model can suggest increasing potassium fertilizer to promote sugar synthesis. If the risk of disease is high, the big data model can recommend appropriate prevention and control measures and green prevention and control programs. Through continuous learning and optimization, the big data model can dynamically adjust its recommendations, improving the accuracy and intelligence of planting management, thereby achieving efficient and sustainable fruit planting management.

[0155] In summary, the fruit quality detection method provided by this invention integrates visual recognition, spectral acquisition, image processing, and deep learning technologies. Specifically, it scans the plant canopy to obtain canopy images, then identifies the target fruit based on these images and adjusts the acquisition posture of the robotic arm 23 to achieve target fruit recognition and localization. The visual system 21 acquires images of the target fruit's appearance to detect external surface features (such as pest and disease conditions, surface defects, etc.), and the spectral acquisition system 22 acquires spectral data of the target fruit to detect internal quality (such as sugar content, acidity, maturity, etc.). The detection is continuous, covering the entire growth stage of the cherry tomato plant. Furthermore, the deep learning model-based fruit quality assessment ensures rapid response and completion of quality detection tasks in orchard environments. Additionally, the robot can flexibly adjust its working mode according to different operational conditions, ensuring stable operation in various environments. Therefore, this invention can efficiently and accurately monitor the fruit growth stages in orchards, improving the overall quality and market competitiveness of fruits, and promoting the development of agricultural mechanization and automation.

[0156] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a terminal device according to an embodiment of the present invention. The terminal device includes:

[0157] The processor 100, the memory 200, and the computer program stored in the memory 200 and configured to be executed by the processor 100, wherein the processor 100 executes the computer program to implement a fruit quality detection method as described in the above embodiments.

[0158] The processor 100 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0159] The memory 200 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 200 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 200 and called by the processor 100 to execute a fruit quality detection method according to an embodiment of the present invention.

[0160] Input / output interface 300 is used to realize information input and output;

[0161] The communication interface 400 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0162] Bus 500 transmits information between various components of the device (e.g., processor 100, memory 200, input / output interface 300, and communication interface 400);

[0163] The processor 100, memory 200, input / output interface 300 and communication interface 400 are connected to each other within the device via bus 500.

[0164] This invention also provides a computer-readable storage medium storing a computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute a fruit quality detection method according to the above embodiments.

[0165] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0166] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0167] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0168] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0169] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0170] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0171] It should be understood that in this invention, "at least one (item)" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0172] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0173] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0174] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0175] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0176] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention.

Claims

1. A method for detecting fruit quality, characterized in that, The fruit quality inspection robot includes a mobile chassis, a robotic arm, a navigation system, a vision system, and a spectral acquisition system. The robotic arm and the navigation system are mounted on the mobile chassis, and the vision system and the spectral acquisition system are mounted on the robotic arm. The detection method includes: Based on navigation planning and orchard environment data collected by the navigation system, the mobile chassis is controlled to move towards the target detection area; Upon reaching the target detection area, the robotic arm is controlled to adjust to the pre-detection state and scans the plant canopy through the vision system to obtain a canopy image. Based on the canopy image, the target fruit is identified and the robotic arm's acquisition posture is adjusted. The vision system is controlled to acquire images of the fruit appearance of the target fruit. The system is controlled to acquire the spectral data of the target fruit. The fruit appearance image is analyzed based on the first deep learning model, the surface features of the target fruit are identified and the quality is evaluated to obtain the external physical quality score of the target fruit. The fruit spectral data is analyzed based on a second deep learning model to predict the internal quality score of the target fruit. The quality test results of the target fruit are obtained by combining the external physical quality score and the internal quality score of the fruit.

2. The fruit quality testing method according to claim 1, characterized in that, The vision system includes a robotic arm depth camera. The process of identifying target fruits and adjusting the robotic arm's acquisition posture based on the canopy image includes: The canopy image is input into a trained YOLO object detection model to identify the target fruit in the canopy image and generate the bounding box of the target fruit and the corresponding confidence information. Calculate the center point of the bounding box of the target fruit to obtain the image coordinates of the target fruit in the canopy image; Based on the image coordinates of the center point, the depth information of the canopy image, and the calibration parameters of the robotic arm, coordinate transformation is performed on the coordinate system of the robotic arm, and the motion trajectory of the robotic arm is generated. The robotic arm's acquisition posture is adjusted according to its movement trajectory.

3. The fruit quality testing method according to claim 1, characterized in that, The process of analyzing the fruit appearance image based on the first deep learning model, identifying the fruit surface features of the target fruit, and performing quality assessment to obtain the external physical quality score of the target fruit includes: The fruit appearance image is input into the trained YOLO object detection model to automatically identify and locate the target fruit in the fruit appearance image; The surface features of the target fruit are extracted using image segmentation technology, and the color distribution is statistically analyzed using color analysis to evaluate the ripeness and color difference of the target fruit. The defect features of the target fruit are extracted and quantitatively analyzed to calculate the defect area and distribution. The external physical quality score of the target fruit is obtained based on its maturity, color difference, defect area and distribution.

4. The fruit quality testing method according to claim 1, characterized in that, The step of analyzing the fruit spectral data based on a second deep learning model to predict the internal quality score of the target fruit includes: The spectral data of the fruit are identified by band selection and feature extraction algorithms to identify spectral characteristic peaks related to the internal quality of the fruit. The spectral feature peaks are input into a trained fruit internal quality prediction model to predict the internal quality score of the target fruit.

5. The fruit quality testing method according to claim 1, characterized in that, The detection method further includes: Edge computing technology is used to preprocess the fruit appearance image; The spectral data of the fruit were preprocessed using standard normal transformation and multivariate scattering correction methods.

6. The fruit quality testing method according to claim 1, characterized in that, The detection method further includes: Based on the fruit appearance image and the fruit spectral data, a growth trend prediction model is constructed using a long short-term memory network and a temporal convolutional network. The growth trend prediction model can use historical fruit growth data to establish a non-linear time series relationship and predict the future ripening time, final quality and potential disease risk of the fruit.

7. A fruit quality inspection robot, characterized in that, A fruit quality detection method according to any one of claims 1 to 6 is adopted, wherein the fruit quality detection robot comprises: a mobile chassis, a robotic arm, a navigation system, a vision system, a spectral acquisition system, and a main control system; the robotic arm, the navigation system, and the main control system are mounted on the mobile chassis, and the vision system and the spectral acquisition system are mounted on the robotic arm; the main control system is used for: Based on navigation planning and orchard environment data collected by the navigation system, the mobile chassis is controlled to move towards the target detection area; Upon reaching the target detection area, the robotic arm is controlled to adjust to the pre-detection state and scans the plant canopy through the vision system to obtain a canopy image. Based on the canopy image, the target fruit is identified and the robotic arm's acquisition posture is adjusted. The vision system is controlled to acquire images of the fruit appearance of the target fruit. The system is controlled to acquire the spectral data of the target fruit. The fruit appearance image is analyzed based on the first deep learning model, the surface features of the target fruit are identified and the quality is evaluated to obtain the external physical quality score of the target fruit. The fruit spectral data is analyzed based on a second deep learning model to predict the internal quality score of the target fruit. The quality test results of the target fruit are obtained by combining the external physical quality score and the internal quality score of the fruit.

8. A fruit quality inspection robot according to claim 7, characterized in that, The fruit quality inspection robot also includes a light-shielding structure. The spectral acquisition system includes a visible-near-infrared spectrometer. Both the visible-near-infrared spectrometer and the light-shielding structure are located at the end of the robotic arm, and the light-shielding structure is located directly in front of the acquisition of the visible-near-infrared spectrometer.

9. A fruit quality inspection robot according to claim 7, characterized in that, The navigation system includes a lidar, a forward-looking depth camera, and an inertial measurement unit. The mobile chassis includes a tracked chassis. The fruit quality inspection robot also includes a device voltage conversion module, a device charging interface, a power display module, and a chassis communication interface mounted on the tracked chassis.

10. A terminal device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements a fruit quality detection method as described in any one of claims 1 to 6.

Images