Fruit quality detection method, robot and terminal equipment

Abstract

The invention provides a fruit quality detection method, a robot and terminal equipment, and the method is applied to the fruit quality detection robot, and specifically, a plant canopy is scanned to obtain a canopy image so as to recognize a target detection fruit and adjust the collection posture of a mechanical arm, so that target recognition and positioning of the fruit are realized; fruit appearance images of target detection fruits are collected through the visual system, fruit external appearance characteristic detection is achieved, fruit spectrum data of the target detection fruits are collected through the spectrum collection system, fruit internal quality detection is achieved, and detection surrounds the whole growth stage of the whole cherry tomato plant and is continuous; in addition, fruit quality evaluation is carried out based on a deep learning model, and it is ensured that a quality detection task can be rapidly responded and completed in an orchard environment. Therefore, the fruit growth stage can be efficiently, accurately and automatically monitored in the orchard, the overall quality and market competitiveness of the fruits are improved, and agricultural mechanization and automation development is promoted.

Description

Technical Field

[0001] The present invention relates to the technical field of fruit quality detection, and in particular to a fruit quality detection method, a robot and a terminal device. Background Art

[0002] With the rapid development of the global fruit industry, improving fruit quality has become a key factor in driving market competitiveness. Currently, fruit quality assessment relies mainly on manual labor, focusing on post-harvest sorting and lacking real-time monitoring of the growth stage.

[0003] In recent years, intelligent agricultural equipment, including harvesting and sorting robots, has been increasingly adopted in fruit cultivation. However, research on non-destructive testing of fruit during its growth phase remains relatively limited. Traditional monitoring methods are not only time-consuming and labor-intensive, but also rely on empirical judgment, lack scientific evidence, and are prone to misjudgment.

[0004] Although drone remote sensing technology has been partially applied in agricultural monitoring, due to the wide distribution of fruits and the influence of natural factors such as light and precipitation, and the fact that fruits spread throughout the entire crown and branches of the fruit trees during their growth process, drones cannot monitor the entire picture of fruits like they monitor crops. A single remote sensing technology cannot meet the needs of full-process monitoring of fruit quality.

[0005] Therefore, developing a non-destructive testing system based on robotics technology to accurately evaluate the quality of fruits during their growth stages has become an effective way to solve the above problems. Summary of the Invention

[0006] The main purpose of the present invention is to provide a fruit quality detection method, robot and terminal equipment, aiming to solve the technical problem of insufficient quality monitoring of fruit growth stages in the prior art.

[0007] In a first aspect, the present invention provides a fruit quality detection method, which is applied to a fruit quality detection robot. The fruit quality detection robot includes a mobile chassis, a robotic arm, a navigation system, a visual system, and a spectrum acquisition system. The robotic arm and the navigation system are arranged on the mobile chassis, and the visual system and the spectrum acquisition system are arranged on the robotic arm. The detection method includes:

[0008] Based on the navigation plan and the orchard environmental data collected by the navigation system, controlling the mobile chassis to move to a target detection area;

[0009] After reaching the target detection area, the robot arm is controlled to adjust to a pre-detection state, and the visual system is used to scan the plant canopy to obtain a canopy image;

[0010] According to the canopy image, identifying the target and detecting the fruit and adjusting the collection posture of the robotic arm;

[0011] Controlling the visual system to collect the appearance image of the target fruit;

[0012] Controlling the spectrum acquisition system to acquire fruit spectrum data of the target detection fruit;

[0013] Analyzing the fruit appearance image based on the first deep learning model, identifying the surface features of the target fruit and performing a quality assessment to obtain an external physical quality score of the target fruit;

[0014] Analyze the fruit spectral data based on the second deep learning model to predict the internal quality score of the target fruit;

[0015] The external physical quality score of the target fruit and the internal quality score of the fruit are combined to obtain the quality detection result of the target fruit.

[0016] Furthermore, the visual system includes a robotic arm depth camera, and the identification of the target and detection of the fruit and adjustment of the collection posture of the robotic arm according to the canopy image include:

[0017] Input the canopy image into a trained YOLO object detection model to identify the target detection fruit in the canopy image and generate a bounding box of the target detection fruit and corresponding confidence information;

[0018] Calculating a center point of a bounding box of the target detected fruit to obtain image coordinates of the target detected fruit in the canopy image;

[0019] Performing coordinate transformation on the coordinate system of the robotic arm according to the image coordinates of the center point, the depth information of the canopy image, and the calibration parameters of the robotic arm, and generating a motion trajectory of the robotic arm;

[0020] Adjust the collection posture of the robotic arm according to the motion trajectory of the robotic arm.

[0021] Furthermore, the analyzing the fruit appearance image based on the first deep learning model, identifying the surface features of the target fruit and performing quality assessment to obtain the external physical quality score of the target fruit includes:

[0022] Inputting the fruit appearance image into a trained YOLO target detection model to automatically identify and locate the target detection fruit in the fruit appearance image;

[0023] Utilizing image segmentation technology to extract surface features of the target fruit, and combining color analysis to statistically analyze color distribution, the maturity and color difference of the target fruit are evaluated;

[0024] Extracting and quantitatively analyzing the defect features of the target fruit, and calculating the defect area and distribution;

[0025] The external physical quality score of the target fruit is obtained according to the maturity and color difference, defect area and distribution of the target fruit.

[0026] Furthermore, the analyzing the fruit spectral data based on the second deep learning model to predict the fruit internal quality score of the target fruit includes:

[0027] Identifying the fruit spectral data through a band selection and feature extraction algorithm to identify spectral characteristic peaks related to the internal quality of the fruit;

[0028] The spectral characteristic peak is input into a trained fruit internal quality prediction model to predict the fruit internal quality score of the target fruit.

[0029] Furthermore, the detection method further comprises:

[0030] Preprocessing the fruit appearance image using edge computing technology;

[0031] The fruit spectral data were preprocessed using standard normal transformation and multivariate scatter correction methods.

[0032] Furthermore, the detection method further comprises:

[0033] Based on the fruit appearance image and the fruit spectral data, a growth trend prediction model is constructed using a long short-term memory network and a temporal convolutional network; wherein, the growth trend prediction model can use the historical growth data of the fruit to establish a nonlinear time series relationship, and predict the future maturity time, final quality and potential disease risk of the fruit.

[0034] In a second aspect, the present invention provides a fruit quality inspection robot, which uses the fruit quality inspection method described in the first aspect. The fruit quality inspection robot includes: a mobile chassis, a robotic arm, a navigation system, a visual system, a spectrum acquisition system, and a main control system. The robotic arm, the navigation system, and the main control system are arranged on the mobile chassis, and the visual system and the spectrum acquisition system are arranged on the robotic arm. The main control system is used to:

[0035] Based on the navigation plan and the orchard environmental data collected by the navigation system, controlling the mobile chassis to move to a target detection area;

[0036] After reaching the target detection area, the robot arm is controlled to adjust to a pre-detection state, and the visual system is used to scan the plant canopy to obtain a canopy image;

[0037] According to the canopy image, identifying the target and detecting the fruit and adjusting the collection posture of the robotic arm;

[0038] Controlling the visual system to collect the appearance image of the target fruit;

[0039] Controlling the spectrum acquisition system to acquire fruit spectrum data of the target detection fruit;

[0040] Analyzing the fruit appearance image based on the first deep learning model, identifying the surface features of the target fruit and performing a quality assessment to obtain an external physical quality score of the target fruit;

[0041] Analyze the fruit spectral data based on the second deep learning model to predict the internal quality score of the target fruit;

[0042] The external physical quality score of the target fruit and the internal quality score of the fruit are combined to obtain the quality detection result of the target fruit.

[0043] Furthermore, the fruit quality inspection robot also includes a shading structure, the spectrum acquisition system includes a visible light-near infrared spectrometer, the visible light-near infrared spectrometer and the shading structure are both arranged at the end of the robotic arm, and the shading structure is located directly in front of the collection of the visible light-near infrared spectrometer.

[0044] Furthermore, the navigation system includes a laser radar, a forward-looking depth camera and an inertial measurement unit, the mobile chassis includes a tracked chassis, and the fruit quality inspection robot also includes a device voltage conversion module, a device charging interface, a power display module and a chassis communication interface arranged on the tracked chassis.

[0045] In a third aspect, the present invention provides a terminal device comprising: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, a fruit quality detection method as described in the first aspect is implemented.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] It integrates visual recognition, spectral acquisition, image processing and deep learning technology. Specifically, by scanning the plant canopy, a canopy image is obtained, and then according to the canopy image, the target detection fruit is identified and the acquisition posture of the robotic arm is adjusted to realize the target recognition and positioning of the fruit; the appearance image of the target detection fruit is collected by the visual system to realize the detection of the external appearance characteristics of the fruit (such as pest and disease conditions, surface defects, etc.), and the fruit spectral data of the target detection fruit is collected by the spectral acquisition system to realize the internal quality detection of the fruit (such as sugar content, acidity, maturity, etc.), and the detection is carried out around the entire growth stage of the cherry tomato plant, with continuity; in addition, the fruit quality assessment is carried out based on the deep learning model to ensure that the quality detection task can be quickly responded to and completed in the orchard environment. In addition, the robot can also flexibly adjust the working mode according to different operating conditions to ensure stable operation in different environments. Therefore, the present invention can efficiently carry out accurate and automated monitoring of the growth stage of fruits in the orchard, improve the overall quality and market competitiveness of fruits, and promote the development of agricultural mechanization and automation. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a structural diagram of a fruit quality inspection robot provided by one embodiment of the present invention;

[0049] Figure 2 This is a structural schematic diagram of a fruit quality inspection robot provided by one embodiment of the present invention from another perspective;

[0050] Figure 3 This is a partial structural diagram of a fruit quality inspection robot provided by one embodiment of the present invention;

[0051] Figure 4 This is a flow chart of a fruit quality detection method provided by one embodiment of the present invention;

[0052] Figure 5 This is a schematic diagram of a process for a fruit quality inspection robot to perform a fruit quality inspection operation according to an embodiment of the present invention;

[0053] Figure 6 This is a flow chart of a robot arm-spectrometer data acquisition posture algorithm based on depth vision provided by one embodiment of the present invention;

[0054] Figure 7 1 is a flow chart of a multimodal fruit quality assessment algorithm for growth stages based on spectral and visual fusion provided by an embodiment of the present invention;

[0055] Figure 8 It is a structural diagram of a terminal device provided by one embodiment of the present invention.

[0056] in:

[0057] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments.

[0058] Description of reference numerals:

[0059] 1. Mobile chassis; 11. Chassis tracks; 12. Track drive wheels; 13. Track driven wheels; 21. Vision system; 22. Spectral acquisition system; 23. Robotic arm; 24. Rotation axis at the end of the robotic arm; 3. Navigation system; 31. LiDAR; 32. Forward-looking depth camera; 33. Inertial measurement unit; 4. Main control system; 41. Equipment voltage conversion module; 42. Equipment charging interface; 43. Power display module; 44. Chassis communication interface; 5. Shading structure. DETAILED DESCRIPTION

[0060] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0061] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the said features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0062] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; they may refer to mechanical connections, direct connections, or indirect connections through an intermediate medium; they may refer to internal communication between two components or the interaction between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0063] In the present invention, unless otherwise expressly specified or limited, a first feature being "above" or "below" a second feature may include the first and second features being in direct contact, or may include the first and second features being in contact not directly but through another feature between them. Furthermore, a first feature being "above," "above," and "above" a second feature may include the first feature being directly above or obliquely above the second feature, or may simply mean that the first feature is higher in level than the second feature. A first feature being "below," "below," and "below" a second feature may include the first feature being directly below or obliquely below the second feature, or may simply mean that the first feature is lower in level than the second feature.

[0064] See also Figure 1 , Figure 1 The figure is a structural diagram of a fruit quality inspection robot provided by one embodiment of the present invention.

[0065] A fruit quality inspection robot, described in embodiments of the present invention, is used for nondestructive quality testing of fruit during its growth stages. By combining navigation technology, visual recognition, spectral acquisition, and edge computing, it can autonomously navigate an orchard and accurately monitor the growth and quality characteristics of fruit, including cherry tomatoes.

[0066] Specifically, refer to Figure 1-3 The fruit quality inspection robot includes: a mobile chassis 1, a robotic arm 23, a navigation system 3, a visual system 21 and a spectrum acquisition system 22. The robotic arm 23 and the navigation system 3 are arranged on the mobile chassis 1, and the visual system 21 and the spectrum acquisition system 22 are arranged on the robotic arm 23.

[0067] Reference Figure 1-2 The mobile chassis 1 includes a crawler chassis, which includes chassis tracks 11, crawler drive wheels 12, and crawler driven wheels 13. The navigation system 3 includes a laser radar 31, a forward-looking depth camera 32, and an inertial measurement unit 33 (IMU).

[0068] The tracked chassis features differential speed control, ensuring the robot's stability and flexibility in complex orchard terrain, particularly on muddy or uneven surfaces. Path planning and intelligent obstacle avoidance algorithms ensure stable operation in complex orchard environments, while enabling precise positioning and autonomous navigation.

[0069] The robotic arm 23 includes a six-degree-of-freedom robotic arm. The six-degree-of-freedom robotic arm has high-precision motion control capabilities and can accurately locate the coordinates of the target detection fruit and perform tasks based on the position data provided by the vision system 21.

[0070] The vision system 21 includes a robotic arm depth camera. The robotic arm depth camera is mounted at the end of the robotic arm 23. To accurately monitor the fruit, the vision system 21 in this embodiment of the present invention uses the robotic arm depth camera to capture canopy images and fruit appearance images, and combines this with a deep learning algorithm to perform target detection and fruit location. Image preprocessing and denoising algorithms effectively remove the effects of ambient light and shadows, improving recognition accuracy. Using the optimized target detection algorithm, the robot can quickly and accurately identify the target fruit in complex environments, providing accurate location information for subsequent quality inspections.

[0071] In a specific embodiment, the front-view depth camera 32 and the robotic arm depth camera both include binocular stereo cameras.

[0072] The spectral acquisition system 22 includes a visible-near-infrared spectrometer (VIS-NIR spectrometer). This system collects spectral data from the fruit to analyze internal quality characteristics such as sugar content, maturity, and soluble solids content. The spectral data is processed using techniques such as standard normal transformation (SNV) and multivariate scatter correction (MSC) to reduce the impact of noise and ensure the accuracy of the test results.

[0073] The present invention combines deep learning algorithms to establish a fruit quality prediction model based on fruit appearance images and fruit spectral data, which can evaluate fruit quality in real time and provide accurate data support for orchard management.

[0074] Specifically, the robot's fruit quality inspection system consists of a VIS-NIR spectrometer, a robotic arm depth camera, a control and drive circuit, and a data transmission link. It is used to collect reflectance spectral data and appearance image information of the fruit. The spectrometer operates by illuminating the fruit surface with a light source of a specific wavelength, acquiring the reflectance spectrum and analyzing the fruit's quality based on its spectral characteristics. After preprocessing, the spectral data is processed in real time by an edge computing unit, and machine learning algorithms are used to predict and analyze internal quality parameters such as the fruit's sugar content and vitamin C content. After preprocessing the fruit's surface image information, a deep learning algorithm is used to analyze the surface condition. The fruit's quality is determined by combining these internal and external qualities. The robotic arm depth camera and spectrometer work in conjunction with a robotic arm 23, which can rotate around a rotation axis 24 at the end of the arm to adjust the position of the VIS-NIR spectrometer and depth camera mounted on the arm. This enables accurate non-destructive testing of the target fruit, ensuring that the inspection process does not affect the fruit's quality.

[0075] Reference Figure 3In a specific embodiment, the fruit quality inspection robot further includes a light-shielding structure 5. The visible light-near infrared spectrometer and the light-shielding structure 5 are both arranged at the end of the robotic arm 23, and the light-shielding structure 5 is located directly in front of the collection of the visible light-near infrared spectrometer.

[0076] In a specific embodiment, the light-shielding structure 5 includes light-shielding silicone rubber for the spectrum collection head.

[0077] Reference Figure 2 In a specific embodiment, the fruit quality inspection robot further includes a device voltage conversion module 41, a device charging interface 42, a power display module 43 and a chassis communication interface 44 arranged on the crawler chassis.

[0078] In addition, the fruit quality robot can replace the terminal device according to different operating conditions, extend the robot's applicability in the fruit growth cycle, and utilize the characteristics of different sensors to collect the required data more accurately.

[0079] A fruit quality detection method provided in an embodiment of the present invention is applied to the fruit quality detection robot of the above embodiment and can perform non-destructive quality detection of fruits during their growth stages.

[0080] See also Figure 4 , Figure 4 FIG1 is a flow chart of a method for detecting fruit quality according to an embodiment of the present invention. The method comprises the following steps:

[0081] S100 , based on navigation planning and orchard environmental data collected by the navigation system 3 , controlling the mobile chassis 1 to move toward a target detection area.

[0082] Reference Figure 5 When the fruit quality inspection robot needs to perform fruit (such as cherry tomatoes) quality inspection operations, the cherry tomato quality inspection program is automatically or manually started. The robot starts autonomously from the starting point, goes to the sampling area or the user-preset area, plans the path along the rows of the orchard, and moves to the target inspection area.

[0083] The navigation system 3 obtains data from sensors in real time and combines it with SLAM technology to dynamically adjust the path and autonomously avoid obstacles, ensuring that the robot can efficiently and autonomously drive in the orchard and complete its mission objectives.

[0084] S200 , after arriving at the target detection area, controlling the robotic arm 23 to adjust to a pre-detection state, and scanning the plant canopy through the visual system 21 to obtain a canopy image.

[0085] After the robot arrives at the target orchard area, the robotic arm 23 is adjusted to the pre-detection action, and the robotic arm depth camera of the visual system 21 scans the canopy structure of the fruit (such as cherry tomatoes) plant, locates the current fruit tree number in combination with the pre-stored orchard map, and uploads the canopy image information to the central server in real time.

[0086] S300 , identifying the target fruit according to the canopy image and adjusting the collection posture of the robotic arm 23 .

[0087] The robot detects the precise location of the fruit through target recognition in the canopy image and adjusts the position of the end effector of the robotic arm 23 in real time.

[0088] S400, controlling the visual system 21 to collect the appearance image of the target fruit;

[0089] S500: Control the spectrum acquisition system 22 to acquire the fruit spectrum data of the target fruit.

[0090] Taking cherry tomatoes as an example, the VIS-NIR spectrometer probe of the spectrum acquisition system 22 is aimed at the cherry tomato fruit, ensuring close contact between the spectrometer and the fruit. The system then starts scanning, collecting spectral data from the surface and interior of the cherry tomato fruit. The camera also simultaneously captures an image of the fruit's appearance (including color, shape, texture, etc.). The collected data is preprocessed in real time (e.g., denoising and normalization) by the edge computing module.

[0091] S600: Analyze the fruit appearance image based on the first deep learning model, identify the surface features of the target fruit, and perform quality assessment to obtain an external physical quality score of the target fruit.

[0092] S700: Analyze the fruit spectral data based on the second deep learning model to predict the internal quality score of the target fruit.

[0093] The robot analyzes the fruit appearance image data based on the first deep learning model (such as the YOLO target detection model) to identify characteristics such as fruit color, surface defects, and pests and diseases; it simultaneously analyzes the fruit spectral data and predicts internal quality indicators such as fruit sugar content and vitamin content through the second deep learning model (such as the PLS or SVM model).

[0094] S800: Obtain a quality detection result of the target fruit by combining the external physical quality score and the internal quality score of the target fruit.

[0095] Comprehensively analyze the internal components and external characteristics of fruits to achieve real-time non-destructive testing of fruit quality.

[0096] Reference Figure 5After the single plant detection is completed, the robot moves to the next target plant according to the path planned by the navigation system 3 and repeats the process of steps S200 to S800 until the detection procedure is completed.

[0097] After completing the inspection mission, the robot returns to the charging station and uploads the complete inspection data (including spectral data, image analysis results, and quality assessment reports) to the cloud server via wireless network. This data is then updated to the orchard database and a visual report is generated. After data upload is complete, the robot automatically docks to the charging station for charging. After charging is complete, the robot enters a low-power sleep mode and waits for the next mission command to wake it up.

[0098] A fruit quality inspection robot according to an embodiment of the present invention further includes a main control system 4, which is used to:

[0099] Based on the navigation plan and the orchard environmental data collected by the navigation system 3, the mobile chassis 1 is controlled to move toward the target detection area;

[0100] After reaching the target detection area, the robot arm 23 is controlled to adjust to the pre-detection state, and the visual system 21 is used to scan the plant canopy to obtain a canopy image;

[0101] According to the canopy image, the target fruit is identified and the collection posture of the robotic arm 23 is adjusted;

[0102] Controlling the visual system 21 to collect the appearance image of the target fruit;

[0103] Controlling the spectrum acquisition system 22 to acquire fruit spectrum data of the target detection fruit;

[0104] Analyzing the fruit appearance image based on the first deep learning model, identifying the surface features of the target fruit and performing a quality assessment to obtain an external physical quality score of the target fruit;

[0105] Analyze the fruit spectral data based on the second deep learning model to predict the internal quality score of the target fruit;

[0106] The external physical quality score of the target fruit and the internal quality score of the fruit are combined to obtain the quality detection result of the target fruit.

[0107] In this embodiment, the main control system 4 can execute all steps and functions of a fruit quality detection method provided in any embodiment, and the specific functions of the main control system 4 are not described in detail here.

[0108] In order to accurately obtain the quality of fruits at different growth stages, a fruit quality detection method in an embodiment of the present invention mainly consists of the following two algorithms: a robot arm-spectrometer data acquisition posture algorithm based on depth vision and a multimodal growth stage fruit quality assessment algorithm that integrates spectrum and vision.

[0109] Specifically, the robot arm-spectrometer data acquisition posture algorithm based on depth vision is as follows:

[0110] In a specific embodiment, step S300 identifies the target fruit and adjusts the acquisition posture of the robotic arm 23 based on the canopy image, including the following steps:

[0111] S310: Input the canopy image into a trained YOLO target detection model to identify the target detection fruit in the canopy image, and generate a bounding box of the target detection fruit and corresponding confidence information;

[0112] S320, calculating the center point of the bounding box of the target detection fruit to obtain the image coordinates of the target detection fruit in the canopy image;

[0113] S330, performing coordinate transformation on the coordinate system of the robotic arm 23 according to the image coordinates of the center point, the depth information of the canopy image, and the calibration parameters of the robotic arm 23, and generating a motion trajectory of the robotic arm 23;

[0114] S340 : Adjusting the collection posture of the robotic arm 23 according to the motion trajectory of the robotic arm 23 .

[0115] Spectral detection utilizes a light source to emit light of varying wavelengths, then detects the reflected spectral response after interaction with the sample to obtain intrinsic information about the substance. The spectrometer used in this invention covers a wavelength range of approximately 200 to 1700 nm. Light sources employing electrically pulsed LEDs and OLEDs achieve improved uniformity and reproducibility. Furthermore, Fourier transform signal filtering can be used to remove ambient noise. To eliminate ambient light interference and improve signal-to-noise ratio and measurement accuracy, detection is typically performed in a darkroom.

[0116] Since spectral detection requires a special dark box environment, in order to improve the accuracy of spectral data, it is necessary to keep close to the fruit during detection to create a dark box detection state. To this end, this embodiment proposes a robot arm-spectrometer data acquisition posture algorithm based on depth vision. This algorithm needs to be used together with the robot arm 23, the depth camera, the spectrometer at the end of the robot arm 23 and its light shielding structure 5. The positions of the various devices at the end of the robot arm 23 are as follows: Figure 3 shown.

[0117] Therefore, when conducting fruit quality inspection, the robotic arm 23 will automatically adjust the acquisition posture according to the canopy image data and fruit position obtained in real time, so that the spectrometer is accurately docked with the surface of the target fruit, thereby performing efficient and non-destructive spectral data collection.

[0118] Specifically, the fruit quality inspection robot is used as an automatic quality inspection system for fruit growth stages. The specific process of its use of a robot arm-spectrometer data acquisition posture algorithm based on deep vision is as follows: Figure 6 As shown:

[0119] S6-1 Image acquisition stage: The system uses the robotic arm depth camera to collect canopy images and corresponding depth data in real time, and preprocesses the collected images (such as denoising and image enhancement) to ensure that the subsequent target detection stage can obtain clearer and higher-quality image input.

[0120] S6-2 Target Detection Stage: The preprocessed images are input into the trained YOLO target detection model. The model can quickly and accurately detect the targets in the canopy image (i.e., target detection fruit), and generate the target's bounding box and corresponding confidence information to ensure that the position and size of each target are effectively identified.

[0121] S6-3 Target center point extraction stage: The system calculates the center point of the bounding box of each detected target to obtain the precise position of the object in the canopy image, and screens and verifies multiple targets to ensure that the selected target is the best object to be operated on.

[0122] S6-4 Coordinate conversion stage: The system uses the internal and external parameters of the robot arm's depth camera to convert the image coordinates and depth information of the target center point into corresponding three-dimensional space coordinates. Then, combined with the calibration parameters of the robot arm 23, these coordinates are corrected so that the converted coordinates are perfectly aligned with the coordinate system of the robot arm 23, laying the foundation for subsequent operations.

[0123] S6-5 Command sending and motion planning stage: The converted and corrected target coordinates are transmitted to the control system of the robot arm 23. The control system formulates a suitable motion trajectory based on these coordinates and instructs the robot arm 23 to start moving, ensuring that the robot arm 23 can move to the predetermined target position quickly and accurately.

[0124] S6-6 Real-time feedback and dynamic correction stage: In order to ensure that the VIS-NIR spectrometer can correctly align with the target (i.e., target detection fruit), the robot arm 23 collects new deep canopy images and target data in real time during the movement. Through continuous target detection and coordinate transformation to monitor the dynamic changes of the target position, the motion trajectory is fine-tuned according to the real-time data to ensure that the robot arm 23 always maintains the best alignment with the target during the movement. In the terminal posture adjustment and data acquisition stage, when the robot arm 23 approaches the target area, it will make the final fine posture adjustment to ensure that the terminal device (spectrometer) can fit the target closely. On this basis, the spectral data acquisition task is started. After the spectral acquisition is completed, the spectral data will be automatically compared with the historical data and the signal-to-noise ratio analysis will be performed to ensure the accuracy of the data acquisition.

[0125] The multimodal fruit quality assessment algorithm for growth stages based on spectral and visual fusion uses multimodal fusion of spectral and visual information to comprehensively analyze the internal components and external characteristics of fruits, achieving real-time non-destructive testing of fruits. It also combines prediction algorithms to predict growth conditions and large language models to provide water and fertilizer recommendations for subsequent planting stages. The algorithm process is as follows: Figure 7 shown.

[0126] Among them, the external characteristics of fruits mainly include the color, defects, diseases and pests on the surface of the fruit, while the internal characteristics refer to the internal components of the fruit, such as sugar content, vitamin C content and other quality parameters.

[0127] In a specific embodiment, the detection method further comprises the following steps:

[0128] S610: Preprocess the fruit appearance image using edge computing technology.

[0129] The fruit appearance images are pre-processed in real time (such as cropping, color correction, denoising, and normalization) through the robot's edge computing module.

[0130] In a specific embodiment, step S600 analyzes the fruit appearance image based on the first deep learning model, identifies the surface features of the target fruit, and performs a quality assessment to obtain an external physical quality score of the target fruit, including the following steps:

[0131] S620: Input the fruit appearance image into a trained YOLO target detection model to automatically identify and locate the target detection fruit in the fruit appearance image;

[0132] S630, using image segmentation technology to extract the surface features of the target fruit, and combining color analysis to calculate color distribution, to evaluate the maturity and color difference of the target fruit;

[0133] S640, extracting and quantitatively analyzing the defect features of the target fruit, and calculating the defect area and distribution;

[0134] S650: Obtain an external physical quality score of the target fruit according to the maturity and color difference, defect area and distribution of the target fruit.

[0135] In this embodiment, cherry tomatoes are taken as an example of target fruit for detection. The present invention first uses a high-resolution depth camera to obtain a fruit appearance image of cherry tomatoes, and performs a series of preprocessing operations on the fruit appearance image, such as cropping, color correction, and denoising, to improve the accuracy of subsequent analysis.

[0136] Subsequently, the preprocessed fruit appearance image is input into the pre-trained YOLO target detection model to achieve automatic recognition and precise positioning of the fruit area.

[0137] Next, image segmentation technology was used to extract surface features, and color analysis methods were used to calculate the distribution of primary colors to assess fruit maturity and color variation. Furthermore, feature extraction and quantitative analysis methods were used to calculate the area and distribution of surface defects such as cracks, spots, diseases, and insect infestations, thereby forming a comprehensive scoring index for evaluating the external physical quality of cherry tomatoes.

[0138] In a specific embodiment, the detection method further comprises the following steps:

[0139] S710, preprocessing the fruit spectral data using a standard normal transformation and a multivariate scatter correction method.

[0140] The spectrometer scans the fruit in the short ultraviolet to near-infrared wavelength range, collecting its reflectance spectral data. During data processing, this example uses the standard normal transformation (SNV) and multivariate scatter correction (MSC) methods to preprocess the raw fruit spectral data, eliminate noise interference, improve the spectral signal-to-noise ratio, and standardize the data.

[0141] In a specific embodiment, step S700 analyzes the fruit spectral data based on the second deep learning model to predict the internal quality score of the target fruit, including the following steps:

[0142] S720, identifying the fruit spectral data through a band selection and feature extraction algorithm to identify spectral characteristic peaks related to the internal quality of the fruit;

[0143] S730: Input the spectral feature peak value into a trained fruit internal quality prediction model to predict the fruit internal quality score of the target fruit.

[0144] Before steps S720 to S730, the quality characteristics of the fruit, such as sugar content, soluble solids, and maturity, are extracted through spectral data and deep learning technology to establish a non-destructive testing model. Specifically, through band selection and feature extraction algorithms, the spectral characteristic peaks related to the internal quality of the fruit are identified. These characteristic peaks represent important quality indicators such as the sugar content and vitamin C content of the fruit. Next, the spectral data is analyzed in combination with traditional spectral analysis methods such as principal component analysis (PCA), partial least squares regression (PLSR) and convolutional neural network (CNN) to construct a fruit internal quality prediction model. Through training and optimization, this model can accurately map spectral features to specific quality parameters to achieve non-destructive testing.

[0145] Therefore, in this embodiment, the fruit spectral data of the target fruit will be identified through band selection and feature extraction algorithms to identify the spectral characteristic peaks related to the internal quality of the fruit, and then the spectral characteristic peaks will be input into the trained fruit internal quality prediction model to predict the fruit internal quality score of the target fruit.

[0146] Taking cherry tomatoes as an example of target fruit, the sugar content and vitamin C model used in this embodiment has an accuracy rate greater than 98% and an R2 coefficient of determination greater than 0.92 after thousands of training cycles. Therefore, the fruit internal quality prediction model can accurately perform non-destructive testing on the internal characteristics of cherry tomatoes.

[0147] In a specific embodiment, the detection method further comprises the following steps:

[0148] S800. Based on the fruit appearance image and the fruit spectral data, a growth trend prediction model is constructed using a long short-term memory network and a temporal convolutional network; wherein the growth trend prediction model can use the historical growth data of the fruit to establish a nonlinear time series relationship to predict the future maturity time, final quality and potential disease risk of the fruit.

[0149] Reference Figure 7 On the basis of realizing real-time non-destructive detection of fruit quality, this embodiment further combines the prediction algorithm to predict the growth of the fruit.

[0150] Specifically, based on multi-time series spectra and visual data, this embodiment uses a method that combines long short-term memory networks (LSTMs) with temporal convolutional networks (TCNs) to construct a growth trend prediction model. This model can use historical growth data, including fruit color changes, sugar content accumulation trends, disease development, etc., to establish nonlinear time series relationships and predict key parameters of future growth stages, such as maturity time, final quality level, and potential disease risks. Through dynamic model updates, prediction parameters can be adjusted for different planting environments and management measures, thereby improving the adaptability and accuracy of the prediction.

[0151] In a specific embodiment, the detection method further comprises the following steps:

[0152] S900, introduces a large language model to provide water and fertilizer management recommendations for the subsequent fruit planting stage.

[0153] Reference Figure 7 This embodiment also incorporates a large language model (LLM) to provide water and fertilizer management recommendations for subsequent planting stages. Combining fruit growth predictions with environmental parameters like temperature and light intensity, the LLM generates personalized water and fertilizer management plans based on extensive agricultural data and expert knowledge.

[0154] For example, in the early stages of growth, if the fruit is predicted to expand rapidly and have low sugar accumulation, the large language model can recommend increasing potassium fertilizer to promote sugar synthesis. If the disease risk is high, the large language model can recommend appropriate prevention and control measures and green control programs. Through continuous learning and optimization, the large language model can dynamically adjust its recommendations, improving the precision and intelligence of planting management, thereby achieving efficient and sustainable fruit planting management.

[0155] In summary, the fruit quality detection method provided by the embodiment of the present invention integrates visual recognition, spectral acquisition, image processing and deep learning technology: specifically, by scanning the plant canopy to obtain a canopy image, and then according to the canopy image, identifying the target detection fruit and adjusting the acquisition posture of the robot arm 23 to achieve target recognition and positioning of the fruit; the fruit appearance image of the target detection fruit is collected by the visual system 21 to achieve external appearance feature detection of the fruit (such as pest and disease conditions, surface defects, etc.), and the fruit spectral data of the target detection fruit is collected by the spectral acquisition system 22 to achieve internal quality detection of the fruit (such as sugar content, acidity, maturity, etc.), and the detection is carried out around the entire growth stage of the cherry tomato plant, with sustainability; in addition, the fruit quality assessment is performed based on the deep learning model, ensuring that the quality detection task can be quickly responded to and completed in the orchard environment. In addition, the robot can also flexibly adjust the working mode according to different operating conditions to ensure stable operation in different environments. Therefore, the present invention can efficiently perform accurate and automated monitoring of the fruit growth stage in the orchard, improve the overall quality and market competitiveness of the fruit, and promote the development of agricultural mechanization and automation.

[0156] See also Figure 8 , Figure 8 FIG1 is a schematic diagram of the structure of a terminal device provided by one embodiment of the present invention. The terminal device includes:

[0157] The processor 100, the memory 200, and the computer program stored in the memory 200 and configured to be executed by the processor 100, when the processor 100 executes the computer program, implements a fruit quality detection method as described in the above embodiments.

[0158] The processor 100 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present invention.

[0159] The memory 200 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 200 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 200 and is called by the processor 100 to execute a fruit quality detection method according to an embodiment of the present invention.

[0160] Input / output interface 300, used to implement information input and output;

[0161] Communication interface 400, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0162] bus 500 , which transmits information between various components of the device (e.g., processor 100 , memory 200 , input / output interface 300 , and communication interface 400 );

[0163] The processor 100 , the memory 200 , the input / output interface 300 and the communication interface 400 are connected to each other in communication within the device via the bus 500 .

[0164] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed, the device containing the computer-readable storage medium is controlled to execute a fruit quality detection method according to the above embodiments.

[0165] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0166] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0167] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0168] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0169] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0170] The terms "first," "second," "third," "fourth," and the like (if any) in the description of the present invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in orders other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products, or apparatus.

[0171] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0172] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the above units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0173] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0175] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and other media that can store programs.

[0176] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A method for detecting fruit quality, characterized in that: Applied to a fruit quality inspection robot, the fruit quality inspection robot includes a mobile chassis, a mechanical arm, a navigation system, a visual system, and a spectrum acquisition system, the mechanical arm and the navigation system are arranged on the mobile chassis, and the visual system and the spectrum acquisition system are arranged on the mechanical arm; The detection method comprises: Based on the navigation plan and the orchard environmental data collected by the navigation system, controlling the mobile chassis to move to a target detection area; After reaching the target detection area, the robot arm is controlled to adjust to a pre-detection state, and the visual system is used to scan the plant canopy to obtain a canopy image; According to the canopy image, identifying the target and detecting the fruit and adjusting the collection posture of the robotic arm; Controlling the visual system to collect the appearance image of the target fruit; Controlling the spectrum acquisition system to acquire fruit spectrum data of the target detection fruit; Analyzing the fruit appearance image based on the first deep learning model, identifying the surface features of the target fruit and performing a quality assessment to obtain an external physical quality score of the target fruit; Analyze the fruit spectral data based on the second deep learning model to predict the internal quality score of the target fruit; The external physical quality score of the target fruit and the internal quality score of the fruit are combined to obtain the quality detection result of the target fruit.

2. A fruit quality detection method according to claim 1, characterized in that, The visual system includes a robotic arm depth camera, and the process of identifying targets, detecting fruits, and adjusting the collection posture of the robotic arm based on the canopy image includes: Input the canopy image into a trained YOLO object detection model to identify the target detection fruit in the canopy image and generate a bounding box of the target detection fruit and corresponding confidence information; Calculating a center point of a bounding box of the target detected fruit to obtain image coordinates of the target detected fruit in the canopy image; Performing coordinate transformation on the coordinate system of the robotic arm according to the image coordinates of the center point, the depth information of the canopy image, and the calibration parameters of the robotic arm, and generating a motion trajectory of the robotic arm; Adjust the collection posture of the robotic arm according to the motion trajectory of the robotic arm.

3. A fruit quality detection method according to claim 1, characterized in that: The step of analyzing the fruit appearance image based on the first deep learning model, identifying the surface features of the target fruit and performing quality assessment to obtain the external physical quality score of the target fruit includes: Inputting the fruit appearance image into a trained YOLO target detection model to automatically identify and locate the target detection fruit in the fruit appearance image; Utilizing image segmentation technology to extract surface features of the target fruit, and combining color analysis to statistically analyze color distribution, the maturity and color difference of the target fruit are evaluated; Extracting and quantitatively analyzing the defect features of the target fruit, and calculating the defect area and distribution; The external physical quality score of the target fruit is obtained according to the maturity and color difference, defect area and distribution of the target fruit.

4. A fruit quality detection method according to claim 1, characterized in that, Analyzing the fruit spectral data based on the second deep learning model to predict the internal quality score of the target fruit includes: Identifying the fruit spectral data through a band selection and feature extraction algorithm to identify spectral characteristic peaks related to the internal quality of the fruit; The spectral characteristic peak is input into a trained fruit internal quality prediction model to predict the fruit internal quality score of the target fruit.

5. A fruit quality detection method according to claim 1, characterized in that: The detection method further comprises: Preprocessing the fruit appearance image using edge computing technology; The fruit spectral data were preprocessed using standard normal transformation and multivariate scatter correction methods.

6. A fruit quality detection method according to claim 1, characterized in that: The detection method further comprises: Based on the fruit appearance image and the fruit spectral data, a growth trend prediction model is constructed using a long short-term memory network and a temporal convolutional network; wherein, the growth trend prediction model can use the historical growth data of the fruit to establish a nonlinear time series relationship, and predict the future maturity time, final quality and potential disease risk of the fruit.

7. A fruit quality inspection robot, characterized in that: A fruit quality detection method according to any one of claims 1 to 6 is used, wherein the fruit quality detection robot comprises: a mobile chassis, a robotic arm, a navigation system, a visual system, a spectrum acquisition system, and a main control system, wherein the robotic arm, the navigation system, and the main control system are arranged on the mobile chassis, and the visual system and the spectrum acquisition system are arranged on the robotic arm; and the main control system is used to: Based on the navigation plan and the orchard environmental data collected by the navigation system, controlling the mobile chassis to move to a target detection area; After reaching the target detection area, the robot arm is controlled to adjust to a pre-detection state, and the visual system is used to scan the plant canopy to obtain a canopy image; According to the canopy image, identifying the target and detecting the fruit and adjusting the collection posture of the robotic arm; Controlling the visual system to collect the appearance image of the target fruit; Controlling the spectrum acquisition system to acquire fruit spectrum data of the target detection fruit; Analyzing the fruit appearance image based on the first deep learning model, identifying the surface features of the target fruit and performing a quality assessment to obtain an external physical quality score of the target fruit; Analyze the fruit spectral data based on the second deep learning model to predict the internal quality score of the target fruit; The external physical quality score of the target fruit and the internal quality score of the fruit are combined to obtain the quality detection result of the target fruit.

8. The fruit quality inspection robot according to claim 7, characterized in that: The fruit quality inspection robot also includes a light-shielding structure, and the spectrum acquisition system includes a visible light-near infrared spectrometer. The visible light-near infrared spectrometer and the light-shielding structure are both arranged at the end of the robotic arm, and the light-shielding structure is located directly in front of the collection of the visible light-near infrared spectrometer.

9. The fruit quality inspection robot according to claim 7, characterized in that: The navigation system includes a laser radar, a forward-looking depth camera and an inertial measurement unit, the mobile chassis includes a tracked chassis, and the fruit quality inspection robot also includes a device voltage conversion module, a device charging interface, a power display module and a chassis communication interface arranged on the tracked chassis.

10. A terminal device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, a fruit quality detection method according to any one of claims 1 to 6 is implemented.

Images