An infrared guided night automatic fruit picking system

Abstract

The application relates to the field of agricultural automatic picking technology and discloses an infrared-guided full-automatic fruit picking system at night, which comprises an edge computing controller, a mechanical arm, an end effector, an infrared micro-bolometer, a water storage tank and a micro-spray electromagnetic valve. The controller acquires environmental temperature and humidity to regulate the constant temperature of the water storage tank; the saturated water vapor pressure difference is calculated, the mechanical arm is locked and water mist is sprayed when the standard is met; infrared image sequences are collected, false pixels caused by wind disturbance are removed through monotonicity determination of first-order backward difference; a fitting model is constructed for effective pixels to extract a temperature drop derivative, a binary mask is generated to divide fruit targets; the two-dimensional centroid of the targets is extracted and converted into three-dimensional coordinates in combination with a depth map to guide the mechanical arm to complete grabbing. The application utilizes active thermal excitation and temperature drop derivative characteristics, eliminates the interference of night weak light and outdoor wind field disturbance, and improves target recognition and picking accuracy.

Description

Technical Field

[0001] This invention relates to the field of agricultural automated harvesting technology, specifically to an infrared-guided fully automated nighttime fruit harvesting system. Background Technology

[0002] Currently, automated orchard harvesting equipment mainly relies on visible light vision sensors to obtain information about the environment and target objects, making it difficult for the equipment to perform identification and harvesting operations properly in environments without natural light at night. To achieve continuous operation at night, existing equipment typically uses passive infrared thermal imaging technology, which identifies fruit by sensing the natural temperature difference between the target and the background.

[0003] However, in nighttime environments, due to the lack of solar radiation, the natural temperature difference between the fruit and surrounding branches and leaves tends to balance, resulting in low contrast and unclear target edge contours in passive infrared thermal imaging. Simultaneously, wind interference is common in outdoor environments, causing random convective heat dissipation on the plant surface. This random heat dissipation process manifests as disordered temperature fluctuations and artifacts in infrared thermal images. These temperature fluctuations further disrupt the limited thermal characteristics of the fruit, causing conventional image thresholding or edge detection algorithms to misidentify background leaves as fruit targets.

[0004] Furthermore, due to the poor quality of infrared imaging at night, the target spatial coordinates output by the vision system often contain deviations. When the robotic arm plans its trajectory and moves within the fruit tree canopy based on these coordinates, if the planned path passes through a kinematically singular configuration region, the Jacobian matrix rank reduction can cause uncontrolled abrupt changes in the joint angular velocity of the robotic arm, easily leading to equipment collisions or damage to the target crop. Therefore, overcoming the interference of weak temperature differences at night and outdoor wind disturbances on visual recognition, and ensuring the stable operation of the robotic arm during harvesting, are urgent technical problems that need to be solved. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an infrared-guided fully automated fruit picking system for nighttime use. This system solves the problems of existing automated nighttime picking equipment, which suffers from difficulties in target identification due to weak temperature differences and wind disturbances at night caused by passive infrared imaging, and the tendency for the robotic arm to experience sudden speed changes at odd configurations due to coordinate deviations.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution: an infrared-guided fully automatic fruit picking system for nighttime use, comprising an edge computing controller with communication connection, a six-degree-of-freedom robotic arm, an end effector, an uncooled infrared microbolometer, a temperature and humidity sensor, a miniature water tank, a depth and distance sensor, and a micro-spray high-frequency solenoid valve; the miniature water tank is equipped with a temperature measuring element and a constant temperature heating component and is connected to the micro-spray high-frequency solenoid valve.

[0007] The edge computing controller acquires the absolute temperature and relative humidity of the environment through temperature and humidity sensors, and adjusts the constant temperature heating component to control the actual temperature inside the micro water storage tank in conjunction with the temperature measuring element; calculates the saturated water vapor pressure difference of the current environment, and when the saturated water vapor pressure difference of the current environment is not lower than the system's minimum effective evaporation threshold of 0.15 kPa to 0.3 kPa, controls the six-degree-of-freedom robotic arm to be stationary and locked, and drives the micro-spray high-frequency solenoid valve to spray water mist; controls the uncooled infrared microbolometer to collect infrared image sequences, performs monotonicity judgment on the pixel grayscale values ​​in the infrared image sequence to remove wind-disturbed pseudo pixels; generates a binary mask by differential differentiation of the infrared image sequence to divide the fruit target area; extracts the two-dimensional centroid coordinates of the fruit target area and converts them into three-dimensional spatial operation coordinates based on the depth ranging sensor, and controls the end effector to move to the three-dimensional spatial operation coordinates to grasp the fruit.

[0008] Furthermore, the edge computing controller acquires discrete sampled values ​​of ambient absolute temperature and ambient relative humidity through temperature and humidity sensors, applies a moving average filtering algorithm to the discrete sampled values ​​to output steady-state ambient absolute temperature and ambient relative humidity; it acquires the actual temperature inside the miniature water tank through a temperature measuring element, extracts the physical difference between the steady-state ambient absolute temperature and the actual temperature inside the miniature water tank to form the temperature deviation, and determines that the constant temperature feedforward adjustment is completed when the absolute difference between the steady-state ambient absolute temperature and the actual temperature inside the miniature water tank is less than the allowable temperature following error threshold of 0.1℃ to 0.3℃ and remains stable within a preset number of continuous control cycles.

[0009] Furthermore, the edge computing controller performs a non-zero check on the temperature deviation during the control cycle. Based on the current proportional state of the temperature deviation, the historical integral accumulation, and the derivative component, it performs closed-loop control to calculate the pulse width modulation duty cycle signal and inputs the pulse width modulation duty cycle signal to the constant temperature heating component to adjust the heating power.

[0010] Furthermore, the edge computing controller constructs an exponential function model based on the steady-state ambient absolute temperature to calculate the saturated water vapor pressure. The actual water vapor pressure is obtained by multiplying the saturated water vapor pressure by the steady-state ambient relative humidity percentage. The physical difference between the saturated water vapor pressure and the actual water vapor pressure is extracted to obtain the current ambient saturated water vapor pressure difference. The current ambient saturated water vapor pressure difference is dynamically compared with the system's minimum effective evaporation threshold. When the current ambient saturated water vapor pressure difference is lower than the system's minimum effective evaporation threshold for multiple consecutive calculation cycles, the drive pin output of the corresponding micro-spray high-frequency solenoid valve of the edge computing controller is blocked.

[0011] Furthermore, the edge computing controller sends a brake command to the six-degree-of-freedom robotic arm, reads the feedback position parameters of each joint motor in the six-degree-of-freedom robotic arm and obtains the feedback position change. When the feedback position change of multiple consecutive servo control cycles converges within the zero-position drift dead zone, the mechanical static locking action closed loop is determined to be completed. A nonlinear mapping model is constructed in which the control pulse width is negatively correlated with the saturated water vapor pressure difference of the current environment. The control pulse width output by the nonlinear mapping model is obtained and sent to drive the micro-spray high-frequency solenoid valve to perform spraying operations. After capturing the electrical closing event of the micro-spray high-frequency solenoid valve, a mechanical delay compensation timer is started. When the mechanical delay compensation timer overflows, a falling edge hard trigger signal is generated.

[0012] Furthermore, the edge computing controller receives the falling edge hard trigger signal and uses it as the time zero point. It controls the uncooled infrared microbolometer to continuously capture multiple frames of two-dimensional infrared images within a preset time window, forming an infrared image sequence. These images are then stacked in memory to form a spatiotemporal data cube. The controller calculates the first-order backward difference of the same spatial coordinate pixel between adjacent time frames along the time axis, constructing a monotonic mask matrix. When the first-order backward difference is no greater than the thermal noise tolerance of 1 to 3 gray levels, a logic high level is assigned to the coordinate position corresponding to the monotonic mask matrix. When the first-order backward difference is greater than the thermal noise tolerance, a logic low level is assigned to the coordinate position corresponding to the monotonic mask matrix. The generated monotonic mask matrix and the spatiotemporal data cube are used to perform a bitwise AND logical operation to remove wind-induced artifact pixels.

[0013] Furthermore, the edge computing controller constructs a first-order polynomial fitting model for the effective pixel coordinates in the spatiotemporal data cube after removing wind-induced pseudo-pixels. The model uses the time frame variable of the infrared image sequence as the independent variable and the corresponding pixel gray value sequence as the dependent variable. The slope parameter of the first-order polynomial fitting model is solved to obtain the cooling derivative of each spatial coordinate point within a preset time window. The discrete cooling derivative is reprojected into a two-dimensional spatial matrix to construct the original derivative spectrum. The global maximum derivative value, global minimum derivative value, and anti-overflow constant in the original derivative spectrum are used to perform extreme value normalization processing on the cooling derivative to generate a single-channel two-dimensional derivative spectrum feature map.

[0014] Furthermore, the edge computing controller inputs the single-channel two-dimensional derivative spectrum feature map into the two-dimensional probability feature map output by the fully convolutional neural network model. Pixels with a probability confidence greater than 0.6 in the two-dimensional probability feature map are converted into binary logic high levels. Morphological opening operations are performed on the image composed of binary logic high levels to eliminate isolated noise. The connected component area of ​​the independent patches after the opening operation is calculated. Patches with a connected component area lower than a set pixel threshold are forced to zero. A binary mask is output to divide the fruit target region. The pixel threshold is set to 10% to 15% of the expected standard pixel area of ​​the fruit.

[0015] Furthermore, the edge computing controller extracts the zero-order spatial moment representing the total pixel area and the first-order spatial moment representing the spatial distribution for the fruit target area in the binarized mask, and derives the two-dimensional centroid coordinates of the fruit target area through quotient calculation; based on the dual-target extrinsic parameters, it performs affine transformation and bilinear interpolation resampling on the depth map acquired and output by the depth ranging sensor, and generates a three-dimensional coordinate vector in the camera coordinate system by combining the two-dimensional centroid coordinates and the optical principal point offset coordinates of the uncooled infrared microbolometer, extracts the feedback position parameters of each joint motor of the six-degree-of-freedom robotic arm, performs homogeneous rigid body transformation by combining the preset hand-eye calibration matrix, and outputs the three-dimensional spatial operation coordinates in the robotic arm base coordinate system.

[0016] Furthermore, the edge computing controller reverse-engineers the target angles of each joint of the six-degree-of-freedom robotic arm and plans the motion trajectory based on the three-dimensional spatial operation coordinates. When the motion trajectory approaches the kinematic singular configuration, causing the Jacobian matrix to tend towards rank deficiency, the damped least squares method is triggered. The damping factor is dynamically superimposed on the main diagonal of the Jacobian matrix to smooth the joint output speed. Position loop control messages are sent to the servo drivers of the motors of each joint of the six-degree-of-freedom robotic arm along the motion trajectory. When the actual spatial coordinates of the end effector reach the set picking tolerance range and the feedback current of the gripper motor included in the end effector reaches the set torque threshold, the grasping action is determined to be successfully closed loop. The set torque threshold is calculated based on the biological and physical yield strength of the target crop and the deceleration transmission ratio of the end effector.

[0017] This invention provides an infrared-guided fully automated nighttime fruit harvesting system. It has the following beneficial effects: 1. This invention combines the constant temperature regulation of a miniature water tank with the calculation of the ambient saturated water vapor pressure difference. When the evaporation threshold is met, the robotic arm is controlled to remain stationary and spray water mist. The timer overflow signal synchronously triggers the image acquisition of an infrared microbolometer. This structure actively sprays water mist at a controlled temperature to induce physical evaporation and cooling, creating a temperature difference between the fruit and the background in a nighttime environment. This solves the problem of unclear imaging caused by the lack of natural light and significant temperature difference in passive thermal imaging at night. Furthermore, mechanical locking and signal synchronization eliminate the interference of motion on image acquisition.

[0018] 2. This invention calculates the first-order backward difference of the acquired continuous infrared image sequence to determine the monotonicity of the pixel gray value, removes pixels that do not have continuous cooling characteristics, and uses a first-order polynomial fitting to extract the cooling derivative of the effective pixel sequence to generate a derivative spectrum feature map. This image processing process uses the inevitable cooling trend of water evaporation after spraying as a screening condition, filters out pixel temperature fluctuation data caused by random disturbances of outdoor wind field, eliminates the interference of external wind field on infrared feature extraction, and makes the subsequent output of the fruit target area binarization mask more accurate.

[0019] 3. When calculating the three-dimensional operation coordinates and planning the motion trajectory of the robotic arm based on spatial moments and depth map data, this invention triggers the damped least squares method to superimpose damping factors on the main diagonal when the Jacobian matrix tends to be rank deficient. At the same time, the feedback current of the end gripper motor is used to determine the closed loop of the grasping action. The above control design prevents sudden changes in joint speed caused by unsolvable or multiple solutions when the robotic arm approaches a singular configuration, ensuring the safe operation of the equipment. Furthermore, the physical grasping state is confirmed by the torque threshold feedback based on the motor current, ensuring the stability and reliability of the mechanical system's picking action. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the hardware physical structure and communication topology of the present invention; Figure 2 This is the main flowchart of the harvesting method of the present invention; Figure 3 This is a block diagram showing the software architecture and functional modules of the present invention; Figure 4 Phase change cooling time-series curves of different physical surfaces after isothermal micro-spraying according to the present invention; Figure 5 The bar chart shows the comparative analysis of the recognition effect of this invention. Detailed Implementation

[0021] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Please see the appendix Figure 1 This invention provides a fully automated agricultural harvesting platform equipped with a nighttime infrared harvesting system. The platform's hardware includes an edge computing controller, a six-degree-of-freedom robotic arm, an end effector, an uncooled infrared microbolometer, a temperature and humidity sensor, a miniature water tank, and a micro-spray high-frequency solenoid valve.

[0023] The edge computing controller establishes communication connections via an industrial data bus with a six-degree-of-freedom robotic arm, an uncooled infrared microbolometer, a temperature and humidity sensor, and a micro-spray high-frequency solenoid valve. The end effector is mounted on the end flange of the six-degree-of-freedom robotic arm. The uncooled infrared microbolometer and temperature and humidity sensor are fixedly mounted on the end effector. A miniature water tank integrates a PTC constant-temperature heating element and a PT100 temperature sensing element. The miniature water tank is connected to the micro-spray high-frequency solenoid valve via fluid piping. The edge computing controller sends control commands to each hardware component and receives data collected by each component.

[0024] The nighttime infrared harvesting system is divided into multiple functional modules in its software architecture, each containing units that execute specific logic. Specifically, the system includes a micro-environment sensing and constant temperature control module, a saturated water vapor pressure difference calculation and operating condition blocking module, a static locking and thermal excitation control module, an image sequence acquisition and monotonicity verification module, a derivative spectrum calculation and target segmentation module, and a coordinate calculation and harvesting control module.

[0025] The microenvironment sensing and isothermal control module includes a parameter acquisition unit and a medium isothermal adjustment unit. The saturated water vapor pressure difference calculation and operating condition blocking module includes a pressure difference calculation unit and a blocking verification unit. The static locking and thermal excitation control module includes a joint locking unit and a pulse generation unit. The image sequence acquisition and monotonicity verification module includes a sequence acquisition unit and a monotonicity determination unit. The derivative spectrum calculation and target segmentation module includes a difference derivative unit and a mask segmentation unit. The coordinate calculation and acquisition control module includes a coordinate transformation unit and an acquisition execution unit.

[0026] See attached document Figure 2 This invention provides a nighttime infrared acquisition method based on multidimensional state constraints and temporal derivative spectrum, comprising the following steps: S10: The environmental temperature and humidity parameters of the working canopy are obtained by the parameter acquisition unit in the micro-environment sensing and constant temperature control module. The medium constant temperature adjustment unit adjusts the water temperature inside the micro water storage tank according to the ambient temperature so that the medium temperature in the pipeline is consistent with the ambient temperature. S20: The pressure difference calculation unit in the saturated water vapor pressure difference calculation and working condition blocking module calculates the current environmental saturated water vapor pressure difference based on the environmental temperature and humidity parameters. The blocking verification unit stops the current picking action when the saturated water vapor pressure difference is lower than the minimum effective evaporation threshold of the system, and enters the next node when the saturated water vapor pressure difference is higher than or equal to the minimum effective evaporation threshold of the system. S30 uses the joint locking unit in the static locking and thermal excitation control module to send joint braking commands to the six-degree-of-freedom robotic arm to keep it in a static locking state. After the robotic arm is locked, the pulse generation unit generates a control pulse signal based on the saturated water vapor pressure difference parameter to drive the micro-spray high-frequency solenoid valve to spray water mist forward into the working canopy. S40, after the micro-spray high-frequency solenoid valve is closed, the sequence acquisition unit in the image sequence acquisition and monotonicity verification module triggers the uncooled infrared micro-radiometry calorimeter to continuously acquire multiple frames of infrared image sequence within a preset time window. The monotonicity determination unit performs monotonicity determination on the gray value change of each pixel in the image sequence and removes pixel points that have physical displacement caused by external wind load. S50, using the derivative spectrum calculation and the differential calculation unit in the target segmentation module, performs first-order differential calculation on the image sequence that has passed the monotonicity check in the time dimension, and uses the mask segmentation unit to generate a binary mask based on the time derivative value of each pixel point, and divides the connected regions with the absolute value of the time derivative below the set segmentation threshold into the fruit target region; S60 uses the coordinate transformation unit in the coordinate calculation and picking control module to extract the two-dimensional centroid coordinates of the target area of ​​the fruit, converts the two-dimensional centroid coordinates into three-dimensional spatial operation coordinates in the six-degree-of-freedom robotic arm base coordinate system, uses the picking execution unit to release the six-degree-of-freedom robotic arm from the locked state, and controls the end effector to move to the three-dimensional spatial operation coordinates to perform the fruit grasping and separation action.

[0027] Please see the appendix Figure 3 In this embodiment, the microenvironment sensing and isothermal control module provided by the present invention is used to eliminate the initial temperature step interference during the active thermal excitation process. Based on the thermodynamic heat transfer mechanism, the heat change on the plant canopy surface mainly consists of two parts: sensible heat conduction and latent heat of phase change. To ensure that the characteristic attenuation signal captured by the subsequent infrared camera only reflects the latent heat consumption of water vaporization determined by the specific heat capacity of the target surface, it is necessary to block the sensible heat exchange between the spraying medium and the environment at the physical source. This module specifically includes the following operating steps: S110, the parameter acquisition unit collects the absolute temperature and relative humidity of the working environment. Temperature and humidity sensors are fixedly installed at the front end of the end effector. The edge computing controller reads the data register of the temperature and humidity sensors at a fixed sampling frequency through a bus interface to obtain the absolute temperature and relative humidity of the working canopy microenvironment. These two environmental parameters are chosen as the system's baseline input because they directly reflect the saturated vapor pressure difference of the microenvironment's micro-air masses, thus forming the core physical boundary constraining the water evaporation rate. To eliminate data jumps caused by nighttime gusts and sensor background thermal noise, the parameter acquisition unit applies a moving average filtering algorithm to smooth the continuously collected discrete temperature and humidity samples, outputting the steady-state absolute temperature and relative humidity. The specific code implementation for sensor data communication reading and the moving average filtering algorithm can be conventionally written by those skilled in the art based on the selected hardware platform; the specific algorithm and timing logic are well-known technologies in the field and will not be elaborated here.

[0028] S120, the medium constant temperature control unit acquires the current internal water temperature of the miniature water storage tank. The miniature water storage tank is equipped with a platinum resistance thermometer. The analog-to-digital converter within the edge computing controller reads the voltage divider signal across the temperature sensor and calculates the actual temperature inside the miniature water storage tank based on the temperature resistance mapping relationship of the platinum resistance thermometer. To ensure the data rigor of the control algorithm, the edge computing controller uses an internal hardware timer to hard-synchronize the sampling pulses of the two types of heterogeneous sensors, ensuring a strict timestamp alignment between the acquired ambient absolute temperature and the actual temperature inside the miniature water storage tank within the same control cycle.

[0029] S130, the medium constant temperature regulation unit drives the heating component to eliminate the initial temperature difference between the medium and the environment. The system sets the absolute ambient temperature as the target control quantity and uses the actual temperature inside the miniature water tank as the feedback quantity. To broaden the scope of patent protection and avoid unnecessary limitations caused by specific mathematical formulas, in this embodiment, the system obtains the temperature deviation of the current control cycle by calculating the physical difference between the absolute ambient temperature and the actual temperature inside the miniature water tank. Based on this temperature deviation, the edge computing controller runs a preset closed-loop control algorithm to calculate the output control quantity. This closed-loop control algorithm comprehensively considers the current proportional state of the temperature deviation, the historical integral accumulation, and the differential derivative component of the future trend, thereby calculating the pulse width modulation duty cycle signal. In the actual discrete system operation, the system performs a non-zero check on the control cycle to ensure that the time interval between two adjacent calculations is always greater than zero, so as to avoid the singular error of the denominator approaching 0 in the differential term calculation. The edge computing controller applies the generated pulse width modulation signal to the power drive circuit of the constant temperature heating component at the bottom of the miniature water tank. The constant temperature heating component dynamically adjusts the heating power according to the duty cycle, so that the actual temperature inside the miniature water storage tank steadily follows the absolute temperature of the environment.

[0030] To prevent temperature overshoot caused by thermal inertia from leading to system misjudgments, the medium isothermal control unit employs multi-dimensional logic for steady-state determination. Only when the absolute difference between the actual temperature inside the micro-water tank and the ambient absolute temperature is less than or equal to the set allowable temperature following error threshold, and this state remains stable within a preset number of continuous control cycles, does the medium isothermal control unit determine that isothermal feedforward regulation is complete and trigger subsequent operation procedures. The allowable temperature following error threshold is typically set between 0.1℃ and 0.3℃. In specific engineering implementations, this threshold is determined based on the noise equivalent temperature difference parameter of the uncooled infrared microbolometer. As long as this error is below the camera's thermal resolution sensitivity threshold, the system can be considered to have reached a physically equivalent isothermal state. Through this temperature following feedback regulation mechanism, the sprayed medium in the pipeline and the canopy environment reach an isothermal state, avoiding the sudden physical temperature change caused by sensible heat conduction when the water mist contacts the plant surface. This provides a clean physical signal source for subsequent extraction of the time-series derivative spectrum.

[0031] Please see the appendix Figure 3 In this embodiment, the saturated vapor pressure difference calculation and operating condition blocking module provided by the present invention is used to establish a mathematical constraint mechanism between the ambient air's water-holding capacity and the intensity of subsequent physical excitation. Based on the principles of meteorological thermodynamics, the extraction of latent heat of vaporization from plant surfaces is constrained by the meteorological conditions of the operating microenvironment. If spraying operations are performed under extremely high humidity conditions, the adhering medium cannot evaporate and dissipate quickly, thus forming a continuous liquid film on the canopy surface covering the original thermal radiation characteristics of the biological tissue. This module specifically includes the following operating steps: S210, the differential pressure calculation unit receives the ambient absolute temperature and relative humidity from the micro-environment sensing and constant temperature control module. The ambient absolute temperature directly determines the theoretical physical upper limit of water vapor that the current space can hold. Based on classical thermodynamic empirical formulas, the differential pressure calculation unit calculates the saturated water vapor pressure at the current ambient absolute temperature by constructing an exponential function model with the natural logarithm base as the base and the ambient absolute temperature as the independent variable. Considering the effective range boundary and communication reliability of industrial-grade sensors, the system pre-sets data cleaning and upper / lower limit clamping mechanisms for the ambient temperature input at the software level to avoid the risk of data overflow caused by abnormal extreme values ​​resulting from sensor communication failures in the above exponential function calculation.

[0032] S220, combining the actual water vapor saturation level in the working space, the pressure difference calculation unit calculates the actual water vapor pressure within the current micro-air mass by linearly multiplying the saturated water vapor pressure by the percentage of ambient relative humidity. Based on the calculation results, the system extracts the physical difference between the saturated water vapor pressure and the actual water vapor pressure to derive the core control parameter characterizing the evaporation potential of environmental moisture, namely the current ambient saturated water vapor pressure difference. This pressure difference parameter reflects the physical dryness of the microenvironment more accurately than the ambient relative humidity alone, and its value directly determines the driving force for the surrounding air to absorb the evaporation of water droplets attached to the surface of the target object.

[0033] After completing the calculation of core control parameters, the S230 system, to avoid wasted power consumption and uniformity of infrared signatures due to ineffective eruptions after rainfall or in high humidity conditions such as dense fog, intervenes with a blocking verification unit to perform proactive safety intervention. The blocking verification unit dynamically compares the calculated current ambient saturated vapor pressure difference with the pre-set minimum effective evaporation threshold of the system.

[0034] The minimum effective evaporation threshold of the system is set to range from 0.15 kPa to 0.3 kPa. This threshold represents the minimum evaporation-driving physical quantity required to generate a cooling signal sufficient to exceed the background thermal noise of the uncooled infrared microbolometer within a preset image acquisition window. In practice, this threshold is obtained through offline environmental calibration experiments before the device leaves the factory and is permanently stored in the non-volatile memory of the edge computing controller.

[0035] To enhance the system's robustness in unstructured outdoor weather conditions, the blocking verification unit employs fault-tolerant logic based on a sliding time window. When the saturated vapor pressure difference of the current environment is lower than the system's minimum effective evaporation threshold for multiple consecutive calculation cycles, the edge computing controller determines that the current micro-meteorological conditions do not meet the objective prerequisites for implementing latent heat extraction of phase change. At this time, the blocking verification unit blocks the drive pin output of the micro-spray high-frequency solenoid valve, suspends the current active thermal excitation process at the software scheduling level, and the system enters sleep mode or switches to a standby static infrared scanning mode that does not rely on media spraying. Conversely, when the saturated vapor pressure difference of the current environment is higher than or equal to the system's minimum effective evaporation threshold, the blocking verification unit determines that operation is allowed, and the system state machine subsequently allows it, enabling the program flow to smoothly transition to the next level of static locking and thermal excitation control node.

[0036] Please see the appendix Figure 3 In this embodiment, the static locking and thermal excitation control module provided by the present invention is used to eliminate the risk of visual observation drift caused by the displacement of the external mechanical body, and dynamically modulates the output intensity of the physical excitation source based on environmental calculation parameters. This module specifically includes the following operating steps: S310, the joint locking unit issues a braking command to the six-DOF robotic arm and confirms the observed pose. Based on computer vision theory, the subsequent feature extraction algorithm relies heavily on the time-series differentiation of the absolute grayscale values ​​of pixels under the same spatial coordinates. If there is micro-vibration or displacement of the mechanical body during image acquisition, the same pixel in consecutive frames of the infrared image sequence will correspond to different physical entities, thus violating the mathematical premise of time-difference differentiation. Before executing physical thermal excitation, the joint locking unit issues a brake command to the servo drivers of each joint of the six-DOF robotic arm via the industrial data bus, guiding the device into a steady-state observed pose. To ensure complete convergence of the motion mechanism, the edge computing controller continuously reads the feedback position parameters of the absolute encoders inside each joint motor. When the feedback position changes of multiple consecutive servo control cycles converge within the set zero-position drift dead zone, the joint locking unit determines that the mechanical static locking action closed loop is complete. The upper limit of the zero-drift dead zone depends on the product of the instantaneous field of view of the uncooled infrared microbolometer and the current working distance of the robotic arm. The physical purpose of this constraint is to ensure that the residual mechanical vibration amplitude does not exceed half a pixel's physical mapping size on the imaging focal plane, thus guaranteeing the spatial alignment of the temporal images from the source.

[0037] S320, the pulse generation unit calculates the control pulse width of the micro-spray high-frequency solenoid valve based on current meteorological parameters. The pulse generation unit obtains the current ambient saturated water vapor pressure difference output by the front-end module. Under different natural meteorological conditions, in order to maintain the overall heat dissipation of the target surface within the optimal light-sensing range of the infrared camera, the system constructs a nonlinear dynamic mapping model for the opening duration of the micro-spray high-frequency solenoid valve. In this embodiment, the control strategy inside the pulse generation unit is abstracted and set as follows: the control pulse width exhibits a negatively correlated nonlinear mapping law with the current ambient saturated water vapor pressure difference. This control logic manifests as the pulse width monotonically decreasing as the pressure difference value increases.

[0038] When the environment is dry and the evaporation driving force is strong, the system automatically reduces the jet pulse width to conserve liquid medium and prevent excessive cooling of the measured object surface. When the environment is relatively humid and the evaporation driving force is weak, the system increases the adhesion density of fine water droplets by extending the jet pulse width, thereby expanding the overall evaporative surface area of ​​the medium on the canopy surface and compensating for the attenuation of the physical evaporation rate of a single water droplet. In addition, at the underlying industrial-grade digital computing implementation, the pulse generation unit introduces a small anti-overflow bias parameter to eliminate the risk of microcontroller division-by-zero anomalies caused when the input differential pressure parameter is extremely close to zero.

[0039] S330, the edge computing controller drives the solenoid valve to perform spraying operations and triggers a hard synchronization signal based on the calculated parameters. Based on the control pulse width of the micro-spray high-frequency solenoid valve generated above, the advanced timer peripheral inside the edge computing controller generates a square wave level signal with a corresponding duty cycle. This signal, after passing through optocoupler isolation and power amplification circuitry, drives the micro-spray high-frequency solenoid valve to perform high-frequency opening and closing actions, spraying isothermal water mist towards the working canopy directly in front. When the level of the drive pin flips to the closed state, the edge computing controller captures this electrical shutdown event using an external interrupt. Considering the physical hysteresis of the mechanical return spring movement of the solenoid valve core, the disconnection of the level signal is not equivalent to the instantaneous cutoff of water flow in the pipeline. Based on this physical cause and effect, after capturing the electrical shutdown event, the system internally starts a preset mechanical delay compensation timer.

[0040] When the mechanical delay compensation timer overflows, it indicates that the physical droplet ejection has completely stopped. At this point, the edge computing controller sends a falling-edge hard trigger signal to the internal logic scheduling bus. This hard trigger signal serves as the zero-time start point for the next-level visual sequence observation process, ensuring strict time alignment between the initial state of the water mist evaporation physical heat dissipation process and the image acquisition time of the uncooled infrared microbolometer. This avoids capturing residual water mist artifacts in flight due to premature camera startup. The set duration of the aforementioned mechanical delay compensation timer is 1.2 to 1.5 times the nominal closing response time of the selected solenoid valve hardware.

[0041] Please see the appendix Figure 3 In this embodiment, the image sequence acquisition and monotonicity verification module provided by the present invention is used to filter out the physical displacement of branches and leaves caused by wind loads in an open natural environment from the source, ensuring that the image data entering the differential operation subsequently contains only pure thermodynamic phase transition dissipation information. This module specifically includes the following operating steps: S410, the sequence acquisition unit constructs a continuous infrared spatiotemporal data matrix based on a hardware trigger signal. The sequence acquisition unit receives a falling-edge hard trigger signal from the pre-mounted static locking and thermal excitation control module, and uses this signal as the zero point of image acquisition time. Based on this trigger reference, the edge computing controller drives the uncooled infrared microbolometer to continuously acquire multiple frames of two-dimensional infrared images at a fixed frame rate within a preset time window. To ensure the capture of the complete physical cooling process while avoiding the thermal regeneration phenomenon after water evaporation disrupting monotonicity, the preset time window length must strictly cover the effective phase change cooling cycle of the tiny water droplets attached to the canopy surface. This window duration is calculated based on the average Souter mean diameter of the particles ejected from the micro-water tank and the current environmental evaporation potential, and is typically set between 0.5 and 2.0 seconds. In practice, these discrete two-dimensional images are stacked in memory in chronological order, collectively constructing a three-dimensional spatiotemporal data cube. For the non-uniformity correction and blind pixel compensation algorithms involved in the imaging process of uncooled infrared microbolometers, those skilled in the art can make conventional settings according to the datasheet of the selected detector. The underlying signal readout and background correction are well-known technologies in the field and will not be elaborated here.

[0042] The S420 monotonicity determination unit, based on the thermodynamic phase transition mechanism, performs monotonicity checks on the grayscale changes of each pixel in the spatiotemporal data cube along the time dimension. In natural working scenarios, the evaporation of water from plant surfaces is a physical phase transition process that continuously absorbs latent heat. This process, mapped onto the infrared focal plane, manifests as a monotonically decreasing trend in the radiation grayscale value of the target pixel over time. When micro-environmental wind loads cause mechanical swaying and displacement of plant branches and leaves, specific pixel coordinates of the camera will alternately capture the higher-temperature plant foreground and the lower-temperature deep-space background, thus exhibiting a clear alternating positive and negative grayscale jump in the time series. Based on the essential differences in these physical phenomena, the monotonicity determination unit calculates the first-order backward difference of pixels with the same spatial coordinates between adjacent time frames along the time axis, and introduces a set thermal noise tolerance for constraint determination. To avoid underlying memory overflow anomalies, the starting point of the above time-series iteration is forcibly set to the second frame of the spatiotemporal data cube. Its core verification inequality is expressed as: ; In the formula, Represents cell coordinates At the current frame time The infrared radiation grayscale value; This indicates the pixel coordinates in the previous frame. The infrared radiation grayscale value; The thermal noise tolerance set for the system.

[0043] The introduction of a thermal noise tolerance in the system aims to ensure the physical integrity of the underlying algorithm. Under absolutely ideal thermodynamic conditions, the inter-frame difference in evaporative cooling should always be less than or equal to zero. However, industrial-grade infrared detectors inherently possess electronic thermal noise and zero-point drift that are difficult to eliminate. Even if the physical temperature of the measured object remains absolutely constant, the pixel output grayscale will still exhibit slight bidirectional fluctuations. Imposing a stringent absolute zero-value monotonicity constraint would lead to the incorrect removal of a large number of truly effective pixels due to the device's inherent noise. The range of the thermal noise tolerance set by the system is determined by converting the noise equivalent temperature difference parameter of the selected uncooled infrared microbolometer to the corresponding grayscale level, and is typically set to 1 to 3 grayscale levels. This tolerance allows for small positive increments in pixel density caused by electronic thermal noise within the overall cooling trend, but strictly blocks large grayscale spikes caused by physical displacement.

[0044] S430, the monotonicity determination unit generates a binarized spatiotemporal mask to eliminate wind-induced artifacts. To avoid biased judgments due to anomalies in a single image acquisition, the system does not rely on extreme value jumps at a single moment, but instead employs global continuous evaluation logic throughout the entire observation window. For any independent pixel coordinate in the spatiotemporal data cube, the monotonicity determination unit traverses all adjacent frame difference results on the time axis. If a pixel strictly satisfies the aforementioned check inequality with thermal noise tolerance in consecutive frame comparisons throughout the preset time window, the system determines that the pixel corresponds to a real physical surface in space that is stationary and undergoing latent heat evaporation, and assigns a logic high level at the coordinate position corresponding to the monotonicity mask matrix.

[0045] Conversely, if any difference result of a pixel in the time series derivation violates the above verification inequality, i.e., a positive grayscale jump exceeding the allowable range of the device's inherent noise floor occurs, the monotonicity determination unit directly identifies this point as a pseudo-pixel whose displacement is caused by wind load interference, and assigns it a logic low level in the monotonicity mask matrix. Based on this, a bitwise AND logic operation is performed between the generated two-dimensional monotonic mask matrix and the original spatiotemporal data cube, enabling the edge computing controller to achieve precise truncation of physical displacement data at the mathematical level. This process provides a strictly spatially aligned, high-purity phase-change cooling data source for subsequent derivative spectrum extraction, effectively improving the system's robustness in unstructured wind-disturbed environments in the field.

[0046] Please see the appendix Figure 3In this embodiment, the derivative spectrum calculation and target segmentation module provided by the present invention is used to transform the time-series thermodynamic dissipation process into a feature representation in two-dimensional space, and to achieve high-precision extraction of high-thermal-inertia targets such as crop fruits by relying on a lightweight convolutional network. Based on the physical premise that different biological tissues have different specific heat capacities, under the condition of absorbing the same latent heat of phase change, the cooling rate of water-saturated fruits is significantly lower than that of leaves and branches. This module captures this physical attenuation difference and peels off the target outline from the background, specifically including the following steps: S510, the derivative calculation unit extracts the cooling derivative of each pixel based on the spatiotemporal data cube. After the pre-module completes the removal of wind-disrupted pseudo-pixels, the derivative calculation unit performs data dimensionality reduction fitting operations on the remaining effective pixels along the time axis. To mitigate the slope error caused by transient background thermal fluctuations in a single frame image, the system does not use the first-to-last difference method between two adjacent frames, but instead introduces a global time window fitting algorithm based on the least squares criterion. For any effective pixel coordinate in the spatiotemporal data cube, the derivative calculation unit constructs a first-order polynomial fitting model using the image acquisition timestamp variable as the independent variable and the corresponding infrared radiation grayscale value sequence as the dependent variable. By solving for the slope parameter of this fitted line, the derivative calculation unit obtains the cooling derivative of each spatial coordinate point within a preset time window. The physical meaning of this derivative value characterizes the average phase change cooling rate of the physical surface corresponding to the pixel. In the implementation of underlying matrix operations, considering that the effective time series contains at least two frames of data and the timestamps are strictly monotonically increasing, the covariance matrix constructed by this fitting process naturally has full rank characteristics, thereby avoiding the singularity error that may occur in matrix inversion operations.

[0047] In S520, the feature mapping unit generates a two-dimensional derivative spectrum feature map and performs data preprocessing. After traversing all pixel coordinates, the feature mapping unit reprojects the discrete cooling derivatives onto a two-dimensional spatial matrix, constructing the original derivative spectrum characterizing the thermal inertia distribution of the scene. To meet the constraints of the input data distribution on subsequent deep learning models, the feature mapping unit performs extremum normalization on this original derivative spectrum. By constructing a linear mapping model, the derivative values ​​with physical dimensions are converted into dimensionless floating-point features. The mapping calculation method is as follows: ; In the formula, This is the normalized cooling derivative; The cooling derivative for the current pixel coordinates; This is the global minimum derivative value in the original derivative spectrum matrix; This is the global maximum derivative value in the original derivative spectrum matrix; This is a constant to prevent overflow.

[0048] The technical purpose of introducing an anti-overflow constant is that, under extreme isothermal scenarios where the environment is in thermal equilibrium and there is no external evaporation drive, the cooling derivatives of each element in the original derivative spectrum matrix tend to be consistent, resulting in a difference between the global maximum derivative value and the global minimum derivative value. Extremely close to zero. The overflow prevention constant ensures that the denominator of the underlying division operation always has a non-zero boundary, avoiding the division-by-zero overflow exception of the arithmetic logic unit. The overflow prevention constant is usually set to 10. -5 After the above mapping process, the infrared image sequence that originally depended on absolute temperature differences is transformed into a single-channel two-dimensional derivative spectrum feature map that only reflects the thermal inertia gradient of the target.

[0049] S530, the semantic segmentation unit extracts the target mask from a single-channel two-dimensional derivative spectrum feature map using a pre-trained deep neural network. Traditional fixed-threshold segmentation is highly susceptible to interference from ambient lighting and local hotspots. This embodiment constructs and deploys a fully convolutional neural network model based on an encoder-decoder architecture. The hierarchical structure of this model includes four downsampling feature extraction stages and four corresponding upsampling restoration stages, and establishes skip connections between the encoding and decoding layers with the same spatial resolution to fuse deep abstract semantic features with shallow fine-grained boundary features.

[0050] In the specific business logic, the feature mapping unit first scales the normalized single-channel two-dimensional derivative spectrum feature map to a fixed resolution (e.g., 512×512 pixels) to meet the dimensionality definition of the network input tensor. Subsequently, the semantic segmentation unit uses this single-channel tensor as network input, and the data flows through multiple layers of two-dimensional convolutional kernels, batch normalization layers, and nonlinear activation functions for layer-by-layer feature extraction. The sigmoid activation layer at the end of the network outputs a two-dimensional probability feature map with the same size as the input. Each floating-point value in this probability feature map represents the business probability confidence that the corresponding pixel belongs to the high-heat inertia fruit target.

[0051] To ensure that technicians can effectively reproduce the above model, its offline training process is fully disclosed here. Training samples are derived from single-channel two-dimensional derivative spectrum feature maps of various fruit trees collected under multiple temperature and humidity conditions after thermal excitation; label data consists of manually precisely drawn binary masks of fruit outlines. Since target segmentation is essentially a pixel-by-pixel binary classification task, the model training phase uses a binary cross-entropy loss function to measure the difference between the network's predicted probability and the actual physical state distribution. The network weights are iteratively updated using the backpropagation algorithm until the loss value on the validation set converges.

[0052] In S540, the contour post-processing unit outputs the final target geometric information based on multi-dimensional criteria. To avoid one-sided judgments relying solely on the single probability output of the neural network, the contour post-processing unit performs a comprehensive threshold decision on the two-dimensional probabilistic feature map. The system sets pixels with a probability confidence greater than 0.6 as candidate target points and binarizes them into a logic high level. Subsequently, the contour post-processing unit performs morphological opening operations on the binarized image to eliminate isolated noise points and uses a connected component analysis algorithm to calculate the geometric area of ​​each independent patch. The system forces small patches with connected component areas lower than a set pixel threshold to zero.

[0053] The aforementioned pixel threshold is not arbitrarily specified, but rather calculated based on the instantaneous field of view of the uncooled infrared microbolometer, the nominal working distance of the current robotic arm, and the physical average projected area of ​​the target fruit. This threshold represents the minimum reasonable pixel area that the effective fruit entity can occupy on the focal plane at a specific observation distance, with a typical value ranging from 10% to 15% of the expected standard pixel area of ​​the fruit. By combining pixel-level probability confidence and geometric morphological connectivity features with dual weighted logic, this module ultimately outputs a binary segmentation mask that is precisely aligned with the physical boundary of the fruit, providing a clean and reliable spatial coordinate reference for subsequent robotic arm servo control.

[0054] Please see the appendix Figure 3 In this embodiment, the coordinate calculation and picking control module provided by the present invention is used to convert the two-dimensional semantic features output by the front-end vision algorithm into deterministic motion commands in three-dimensional physical space, and drive the end effector to complete physical interaction. This module establishes a mathematical closed loop between visual pixels and the joint space of the robotic arm based on multi-sensor spatial registration and rigid body kinematic mapping, specifically including the following operating steps: S610, the 2D centroid calculation unit extracts the image coordinates of the target based on a binary segmentation mask. The 2D centroid calculation unit receives the binary segmentation mask output by the pre-module. Based on the connected component distribution of the target on the image plane, the system calculates the geometric centroid of the high-thermal-inertia target patch using an image spatial moment algorithm. Specifically, for independent connected regions in the mask, the 2D centroid calculation unit extracts the zero-order spatial moment representing the total pixel area and the first-order spatial moment representing the spatial distribution, respectively, and then derives the 2D pixel centroid coordinates of the target through quotient operations. In the underlying division operation logic, since the contour post-processing unit of the pre-module has performed strict minimum area threshold filtering on the connected components, the zero-order spatial moment is physically constrained to be an integer always greater than zero. This pre-constraint naturally constructs an arithmetic boundary, avoiding possible division-by-zero overflow anomalies during centroid calculation.

[0055] The S620 3D reconstruction unit generates the target's 3D coordinates in the camera coordinate system based on the pinhole camera model and forward cross-referenced ranging data. After acquiring the 2D coordinates, since the monocular infrared camera lacks depth information, the system calls upon a depth ranging sensor (such as a time-of-flight sensor) that has undergone rigorous spatial hardware registration with an uncooled infrared microbolometer to obtain the physical distance parameters of the target surface. To overcome the spatial mismatch problem caused by the inconsistency in field of view and hardware resolution between the infrared imaging detector and the depth sensor, the system pre-performs affine transformation and bilinear interpolation resampling on the original depth map based on dual-target extrinsic parameters before extracting the depth value, ensuring that the depth matrix and infrared pixels achieve strict pixel-by-pixel alignment.

[0056] Based on this, and combined with the pre-calibrated intrinsic parameter matrix of the infrared camera, the 3D reconstruction unit obtains the target's 3D coordinates in the camera coordinate system through multiplication and division algebraic calculations of inverse projection, using the target's 2D pixel centroid coordinates, the principal point offset coordinates of the camera's optical principal point, and the pixel equivalents in the horizontal and vertical directions, combined with the spatial orthogonal depth values ​​returned by the depth ranging sensor. The physical purpose of the above inverse projection calculation is to eliminate perspective distortion during optical lens imaging and restore the pixels attached to the 2D plane to true physical space vectors.

[0057] S630, the coordinate system transformation unit performs hand-eye matrix mapping to obtain the target's absolute spatial pose in the robot arm base coordinate system. Since the observation origin of the vision sensor changes with the robot arm's end effector posture, the system needs to convert the local coordinates in the camera coordinate system to global operational coordinates fixed to the robot arm base. The coordinate system transformation unit extracts encoder feedback data from each joint of the robot arm in the current locked state, combines it with the hand-eye calibration matrix imported during system initialization, and performs a homogeneous rigid body transformation: ; In the formula, Let be the three-dimensional coordinate vector of the target in the coordinate system of the robot arm base; This is the three-dimensional coordinate vector of the target in the camera coordinate system, formed by the previous solution. A 3×3 orthogonal rotation matrix representing the hand-eye rotation relationship; It is a 3×1 translation vector that characterizes the hand-eye translation relationship.

[0058] Based on the principles of rigid body kinematics, a 3×3 orthogonal rotation matrix characterizing the hand-eye rotation relationship is used. Belonging to a special orthogonal group, its determinant is always a non-zero constant. This mathematical property guarantees the non-singularity of the hand-eye mapping matrix, ensuring that the spatial transformation has a unique solution under any observation perspective. The parameters of the hand-eye calibration matrix are obtained by relying on the standard nine-point calibration method performed before the equipment leaves the factory and are persistently stored in the system settings table.

[0059] In S640, the inverse kinematics unit and the picking execution unit calculate the angles of each joint and drive the end effector to perform physical intervention. For the target's absolute spatial pose mapped to the base coordinate system, the inverse kinematics unit establishes a parametric model of the six-DOF manipulator and reverse-engineers the target angles of each joint required for the end effector to reach that three-dimensional coordinate system. To ensure operational safety, the system introduces a multi-dimensional weighted evaluation logic based on Jacobian matrix singular value monitoring at this stage. Specifically, the motion planning module not only checks whether the target pose is within the manipulator's working envelope space but also continuously calculates the condition number of the Jacobian matrix under the current pose.

[0060] When the robotic arm's planned trajectory approaches a kinematically singular configuration, causing the Jacobian matrix to tend towards rank deficiency, simple inverse kinematics calculations will generate extremely high joint angular velocities. To address the potential singularity anomalies in this matrix operation, the system's underlying layer automatically triggers damped least squares, dynamically superimposing damping factors on the main diagonal to forcibly smooth the joint output velocity, sacrificing minute end-effector positioning accuracy in exchange for the overall physical safety of the system.

[0061] After completing the safety trajectory planning, the harvesting execution unit sends position loop control messages to the servo drivers of each joint based on the interpolated motion trajectory. To avoid mechanical damage caused by single position control, the closing action of the end effector adopts a dual fusion judgment based on position error and motor current feedback. The internally set torque threshold is calculated based on the biological and physical yield strength of the target crop and the reduction transmission ratio of the end gripper mechanism. The core purpose of this parameter setting is to ensure the stability of the gripping while preventing irreversible crushing damage to the plant tissue due to excessive gripping output force. The system only determines that the gripping action has successfully closed the loop when the end spatial coordinates reach the set harvesting tolerance range and the feedback current of the end gripper motor reaches the set torque threshold representing physical contact. For the derivation of the forward and inverse kinematics matrices of the underlying six-degree-of-freedom robotic arm and the cubic spline trajectory interpolation algorithm, those skilled in the art can consult general robotics theory literature and related open source libraries, which are well-known technologies in this field and will not be elaborated here.

[0062] Specific application examples: Assuming the current time is 2:00 AM, the picking platform is operating in an apple orchard. At this time, there is no direct sunlight, and after several hours of natural heat dissipation from the orchard canopy, the physical temperature of the fruit, leaves, and branches has basically reached thermal equilibrium (i.e., a thermodynamic isothermal state).

[0063] Microenvironment sensing and constant temperature feedforward regulation: The temperature and humidity sensors at the end of the robotic arm measure the current absolute temperature of the canopy microenvironment as 18.5℃ and the relative humidity as 65%. The edge computing controller immediately activates the heating component of the micro water tank, and through closed-loop regulation, precisely heats and maintains the temperature of the pure water in the tank at 18.5℃. This ensures that no sensible heat exchange occurs when water droplets come into contact with the fruit trees.

[0064] Evaporation potential assessment and operating condition determination: Based on the above temperature and humidity parameters, the differential pressure calculation unit calculates that the current ambient saturated water vapor differential pressure is approximately 0.74 kPa. This value is much higher than the system's set minimum effective evaporation threshold of 0.2 kPa, indicating that the current air has sufficient water holding capacity, and the water mist can quickly evaporate and absorb heat after being sprayed out, allowing the system to proceed with operations.

[0065] Static locking and thermal excitation: The six-degree-of-freedom robotic arm enters the brake-locked state. Based on the saturated water vapor pressure difference of 0.74 kPa, the system inversely calculates the opening pulse width of the micro-spray high-frequency solenoid valve to be 1.2 seconds. The solenoid valve sprays isothermal water mist at 18.5℃ directly into the fruit tree canopy, and the water mist evenly adheres to the surface of the fruit and leaves.

[0066] Image acquisition and wind disturbance monotonicity removal (in conjunction with appendix) Figure 4 After the spraying ended, the infrared camera continuously acquired 45 frames of infrared images at a rate of 30 frames per second over the next 1.5 days. Combined with... Figure 4 , Figure 4 The horizontal axis represents the time sequence of image acquisition, in seconds; the vertical axis represents the equivalent radiation temperature of the physical surface captured by the uncooled infrared microbolometer, in °C. The horizontal dotted line running through the center of the figure represents the initial isothermal baseline of the environment. The system, through pre-set thermostat control, ensures that the temperature of the micro-spray medium is absolutely consistent with the ambient temperature, thus blocking heat conduction interference caused by instantaneous temperature differences at the physical source.

[0067] At this moment, a gentle breeze caused the leaves to sway. Figure 4 The image shows a dotted line with a cross-marked curve (fluctuating wildly up and down), representing the pixels of leaves disturbed by external gusts of wind. Because the leaves are swaying, the camera alternately captures images of the leaves and the deep-sky background at the same pixel location, causing drastic positive and negative jumps in temperature values, completely disrupting the monotonic decreasing law of water evaporation and cooling. These interfering pixels are directly intercepted by the monotonicity verification module. The core verification inequality based on this underlying interception is as follows: ; In the formula, Represents cell coordinates At the current frame time The infrared radiation grayscale value; This indicates the pixel coordinates in the previous frame. The infrared radiation grayscale value; This formula sets the thermal noise tolerance for the system. The system uses this formula to forcibly eliminate artifacts caused by wind disturbance.

[0068] Derivative spectrum calculation and target segmentation (with appendix) Figure 4 For the remaining stationary pixels after wind disturbance has been eliminated, the water droplets begin to evaporate and absorb heat. Figure 5 The curves are represented by two steadily decreasing curves: the thick black line that decreases slowly represents the phase transition cooling curve of the target fruit. Because the fruit is rich in water and has extremely high thermal inertia, its surface evaporation and cooling are extremely slow. This slope feature is extracted by the system and used as the core identification criterion. The black dashed line that decreases rapidly represents the phase transition cooling curve of the background leaves. Because the leaves are extremely thin and have very low thermal inertia, they cool down extremely quickly under the same conditions, thus forming a sharp numerical difference with the fruit.

[0069] After obtaining the cooling derivative through first-order polynomial fitting, the system performs extreme value normalization on the original derivative spectrum using the following mapping calculation method: ; In the formula, This is the normalized cooling derivative; The cooling derivative for the current pixel coordinates; This is the global minimum derivative value in the original derivative spectrum matrix; This is the global maximum derivative value in the original derivative spectrum matrix; To prevent overflow constants, this formula processes the apple and leaves, which were originally blended together under the infrared lens, to show a strong contrast on the two-dimensional derivative spectrum feature map, allowing the neural network to easily and perfectly segment the fruit region.

[0070] Coordinate calculation and harvesting execution: The system extracts the 2D centroid of the apple mask and, combined with a depth sensor, determines its physical distance to be 420 millimeters. The system extracts encoder feedback data from each joint of the robotic arm in the current locked state and executes the following homogeneous rigid body transformation formula: ; In the formula, Let be the three-dimensional coordinate vector of the target in the coordinate system of the robot arm base; This is the three-dimensional coordinate vector of the target in the camera coordinate system, formed by the previous solution. A 3×3 orthogonal rotation matrix representing the hand-eye rotation relationship; This is a 3×1 translation vector representing the hand-eye translation relationship. Through this calculation, the precise three-dimensional absolute spatial coordinates of the apple are obtained. Subsequently, the robotic arm unlocks and moves to the target point, the gripper closes and confirms the grasping force, ultimately successfully separating the fruit.

[0071] Experimental verification and effect comparison (with appendix) Figure 5 analyze): To verify the effectiveness and robustness of the present invention, three different technical solutions were designed for comparative experiments under the same nighttime orchard conditions. A total of two hundred apple target recognition and picking attempts were conducted.

[0072] Combination Figure 5 (Analysis of recognition performance) Analyze the performance. Figure 5 The horizontal axis lists three nighttime visual detection schemes used in the comparison: the traditional passive scheme, the conventional cold water single-shot imaging scheme, and the third scheme of this invention. The vertical axis represents the specific values ​​of each inspection and evaluation indicator, in percentage form. In the figure, the dark gray bars represent the target success detection rate (the higher the bar, the more targets were found, and none were missed); the light gray bars represent the wind disturbance false alarm rate (the lower the bar, the stronger the system's resistance to physical interference); the values ​​at the top of the bars are objective percentage data obtained through specific calculations.

[0073] In comparison scheme one (traditional passive infrared method), the thermal radiation intensity of the apple and the background leaves was almost identical at night, indicating thermal fusion. This is reflected in... Figure 5 In this study, the dark gray columnar particles were present in extremely low concentrations (detection rate was extremely low), which was far from meeting industrial requirements.

[0074] In comparison scheme two (conventional cold water spraying method), due to the direct spraying of cold water and the lack of wind disturbance resistance mechanism, single-frame threshold segmentation easily misidentifies leaves with thicker surface water films as fruits. This is reflected in... Figure 5 In the middle, although the dark gray bars representing the detection rate increased, the light gray bars representing the wind disturbance false alarm rate reached an extremely high level, indicating that the robotic arm frequently and incorrectly grabbed leaves, resulting in a large amount of invalid work.

[0075] Scheme 3 of this invention (isothermal spraying combined with derivative spectrum verification) demonstrates significant advantages. Isothermal feedforward regulation eliminates step interference, monotonicity verification removes blade swaying, and the derivative spectrum transforms differences into thermal inertia characteristics. (Observation) Figure 5 The third group of bars and their top values ​​visually demonstrate that the dark gray bars representing the detection rate reached the highest level across the entire field, while the light gray bars representing the false recognition rate were suppressed to a negligible low level. This fully verifies the high precision and robustness of the multidimensional state constraints and temporal derivative physical mechanism in agricultural infrared vision presented in this invention.

Claims

1. An infrared-guided fully automated nighttime fruit harvesting system, characterized in that, It includes an edge computing controller with communication connectivity, a six-degree-of-freedom robotic arm, an end effector, an uncooled infrared microbolometer, a temperature and humidity sensor, a miniature water tank, a depth and distance sensor, and a micro-spray high-frequency solenoid valve; The miniature water storage tank is equipped with a temperature measuring element and a constant temperature heating component, and is connected to the micro-spray high-frequency electromagnetic valve. The edge computing controller obtains the absolute temperature and relative humidity of the environment through the temperature and humidity sensor, and adjusts the constant temperature heating component in conjunction with the temperature measuring element to control the actual temperature inside the micro water storage tank. Calculate the saturated water vapor pressure difference of the current environment. When the saturated water vapor pressure difference of the current environment is not lower than the minimum effective evaporation threshold of the system (0.15 kPa to 0.3 kPa), control the six-degree-of-freedom robotic arm to remain stationary and lock it in place, and drive the micro-spray high-frequency solenoid valve to spray water mist. The uncooled infrared microbolometer is controlled to acquire infrared image sequences, and the pixel grayscale values ​​in the infrared image sequences are judged to be monotonic to remove wind-disturbed artifacts. The infrared image sequence is differentially differentiated to generate a binarized mask to divide the fruit target region; The two-dimensional centroid coordinates of the target area of ​​the fruit are extracted and converted into three-dimensional spatial operation coordinates based on the depth ranging sensor. The end effector is then controlled to move to the three-dimensional spatial operation coordinates to grasp the fruit.

2. The infrared-guided fully automated nighttime fruit harvesting system according to claim 1, characterized in that, The edge computing controller operates a micro-environment sensing and constant temperature control module, which includes a parameter acquisition unit and a medium constant temperature adjustment unit. The parameter acquisition unit acquires discrete sampled values ​​of the ambient absolute temperature and the ambient relative humidity through the temperature and humidity sensor, and applies a moving average filtering algorithm to the discrete sampled values ​​to output steady-state ambient absolute temperature and ambient relative humidity. The medium constant temperature regulation unit obtains the actual temperature inside the miniature water storage tank through the temperature measuring element, and extracts the physical difference between the steady-state ambient absolute temperature and the actual temperature inside the miniature water storage tank to form the temperature deviation. When the absolute difference between the steady-state ambient absolute temperature and the actual temperature inside the miniature water storage tank is less than the allowable temperature following error threshold and remains stable within a preset number of continuous control cycles, it is determined that the constant temperature feedforward regulation is completed. The allowable temperature following error threshold is 0.1℃ to 0.3℃.

3. The infrared-guided fully automated nighttime fruit harvesting system according to claim 2, characterized in that, The medium constant temperature regulation unit performs a non-zero check on the temperature deviation within the control cycle, and calculates the pulse width modulation duty cycle signal based on the current proportional state, historical integral accumulation and differential derivative components of the temperature deviation by performing closed-loop control, and inputs the pulse width modulation duty cycle signal to the constant temperature heating component to adjust the heating power.

4. The infrared-guided fully automated nighttime fruit harvesting system according to claim 3, characterized in that, The edge computing controller operates a saturated water vapor pressure difference calculation and operating condition blocking module, which includes a pressure difference calculation unit and a blocking verification unit. The pressure difference calculation unit constructs an exponential function model based on the steady-state ambient absolute temperature to calculate the saturated water vapor pressure. The saturated water vapor pressure is multiplied by the steady-state ambient relative humidity percentage to obtain the actual water vapor pressure. The physical difference between the saturated water vapor pressure and the actual water vapor pressure is extracted to obtain the saturated water vapor pressure difference of the current environment. The blocking verification unit dynamically compares the saturated water vapor pressure difference of the current environment with the minimum effective evaporation threshold of the system. When the saturated water vapor pressure difference of the current environment is lower than the minimum effective evaporation threshold of the system for multiple consecutive calculation cycles, the drive pin output of the micro-spray high-frequency solenoid valve corresponding to the edge computing controller is blocked.

5. The infrared-guided fully automated nighttime fruit harvesting system according to claim 4, characterized in that, The edge computing controller operates a static locking and thermal excitation control module, which includes a joint locking unit and a pulse generation unit. The joint locking unit sends a brake command to the six-degree-of-freedom robotic arm, reads the feedback position parameters of each joint motor in the six-degree-of-freedom robotic arm and obtains the feedback position change. When the feedback position change of multiple consecutive servo control cycles converges within the zero drift dead zone, the mechanical static locking action is determined to be completed. The pulse generation unit constructs a nonlinear mapping model in which the control pulse width is negatively correlated with the saturated water vapor pressure difference of the current environment. It obtains the control pulse width output by the nonlinear mapping model and sends it down to drive the micro-spray high-frequency solenoid valve to perform spraying operations. After capturing the electrical shutdown event of the micro-spray high-frequency solenoid valve, it starts a mechanical delay compensation timer and generates a falling edge hard trigger signal when the mechanical delay compensation timer overflows.

6. The infrared-guided fully automated nighttime fruit harvesting system according to claim 5, characterized in that, The edge computing controller operates an image sequence acquisition and monotonicity verification module, which includes a sequence acquisition unit and a monotonicity determination unit. The sequence acquisition unit receives the falling edge hard trigger signal and uses the falling edge hard trigger signal as the time zero point to control the uncooled infrared microbolometer to continuously acquire multiple frames of two-dimensional infrared images that make up the infrared image sequence within a preset time window, and stacks them in memory to build a spatiotemporal data cube. The monotonicity determination unit calculates the first-order backward difference of the same spatial coordinate pixel between adjacent time frames one by one along the time axis, and constructs a monotonicity mask matrix. When the first-order backward difference is not greater than the thermal noise tolerance, a logic high level is assigned to the coordinate position corresponding to the monotonicity mask matrix. When the first-order backward difference is greater than the thermal noise tolerance, a logic low level is assigned to the coordinate position corresponding to the monotonicity mask matrix. The generated monotonicity mask matrix and the spatiotemporal data cube are used to perform a bitwise AND logical operation to remove wind-induced artifact pixels. The thermal noise tolerance is set to 1 to 3 gray levels.

7. The infrared-guided fully automated nighttime fruit harvesting system according to claim 6, characterized in that, The edge computing controller operates a derivative spectrum calculation and target segmentation module, which includes a difference derivative unit and a mask segmentation unit. The differential unit constructs a first-order polynomial fitting model for the effective pixel coordinates in the spatiotemporal data cube after removing wind-induced false pixels. It uses the time frame variable of the infrared image sequence as the independent variable and the corresponding pixel gray value sequence as the dependent variable. It then solves the slope parameter of the first-order polynomial fitting model to obtain the cooling derivative of each spatial coordinate point within the preset time window.

8. The infrared-guided fully automated nighttime fruit harvesting system according to claim 7, characterized in that, The mask segmentation unit reprojects the discrete cooling derivatives into a two-dimensional spatial matrix to construct the original derivative spectrum. It then uses the global maximum derivative value, global minimum derivative value, and anti-overflow constant in the original derivative spectrum to perform extreme value normalization processing on the cooling derivatives, generating a single-channel two-dimensional derivative spectrum feature map. The mask segmentation unit inputs the single-channel two-dimensional derivative spectrum feature map into the two-dimensional probability feature map output by the fully convolutional neural network model. Pixels with a probability confidence greater than 0.6 in the two-dimensional probability feature map are converted into binary logic high levels. Morphological opening operations are performed on the image composed of the binary logic high levels to eliminate isolated noise. The connected component area of ​​the independent patches after the opening operation is calculated. Patches with connected component areas lower than a set pixel threshold are forcibly set to zero. The binary mask is output to segment the target fruit region. The set pixel threshold is 10% to 15% of the expected standard pixel area of ​​the fruit.

9. The infrared-guided fully automated nighttime fruit harvesting system according to claim 8, characterized in that, The edge computing controller operates a coordinate calculation and harvesting control module, which includes a coordinate transformation unit and a harvesting execution unit. The coordinate transformation unit extracts the zero-order spatial moment representing the total pixel area and the first-order spatial moment representing the spatial distribution for the fruit target region in the binary mask, and derives the two-dimensional centroid coordinates of the fruit target region through quotient operation. The coordinate transformation unit performs affine transformation and bilinear interpolation resampling on the depth map acquired and output by the depth ranging sensor based on dual-target extrinsic parameters. It generates a three-dimensional coordinate vector in the camera coordinate system by combining the two-dimensional centroid coordinates and the optical principal point offset coordinates of the uncooled infrared microbolometer. It extracts the feedback position parameters of each joint motor of the six-degree-of-freedom robotic arm, performs homogeneous rigid body transformation by combining the preset hand-eye calibration matrix, and outputs the three-dimensional spatial operation coordinates in the robotic arm base coordinate system.

10. The infrared-guided fully automated nighttime fruit harvesting system according to claim 9, characterized in that, The picking execution unit reverse-engineers the target angles of each joint of the six-degree-of-freedom robotic arm and plans the motion trajectory based on the three-dimensional spatial operation coordinates. When the motion trajectory approaches the kinematic singular configuration, causing the Jacobian matrix to tend to be rank-deficient, the damped least squares method is triggered. The damping factor is dynamically superimposed on the main diagonal of the Jacobian matrix to smooth the joint output speed. The picking execution unit sends position loop control messages to the servo drivers of the joint motors of the six-degree-of-freedom robotic arm along the motion trajectory. When the actual spatial coordinates of the end effector reach the set picking tolerance range and the feedback current of the gripper motor included in the end effector reaches the set torque threshold, the grasping action is determined to be successfully closed loop. The set torque threshold is calculated based on the biological and physical yield strength of the target crop and the reduction transmission ratio of the end effector.

Images