Collaborative control method and system for non-destructive harvesting of fragrant pears with anti-occlusion visual guidance
By registering and fusing multi-source image data, combined with a pest and disease detection model, the problem of low accuracy in detecting pests and diseases in fragrant pears was solved, enabling non-destructive harvesting.
Patent Information
- Application Number
- CN202611384803.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-08
- Publication Date
- 2026-11-10
AI Technical Summary
Existing intelligent detection methods for diseases and pests in fragrant pears suffer from poor feature fusion processing due to the different responses of different types of diseases and pests to color, texture, spectrum, temperature, and structural characteristics, making it impossible to achieve lossless harvesting.
By acquiring multi-source image data and positioning data, performing registration processing, extracting multi-dimensional features and fusing them, and using a pest and disease detection model for detection, a non-destructive harvesting collaborative control strategy is generated.
It improves the accuracy of pest and disease detection, enables non-destructive harvesting, and reduces mechanical damage to fruits and cross-infection of diseases.
Smart Images

Figure CN123071835A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of collaborative control technology for harvesting, specifically to a collaborative control method and system for non-destructive harvesting of fragrant pears with anti-occlusion visual guidance. Background Technology
[0002] Pear trees are susceptible to various pests and diseases during their growth, such as pear psyllids, aphids, spider mites, fruit borers, pear rust, black spot, ring rot, and leaf spot. Early symptoms of these pests and diseases typically include changes in leaf color, spots, curling, localized necrosis, abnormal canopy temperature, decreased plant vitality, and abnormal branch structure. Therefore, it is necessary to monitor pear trees for pests and diseases.
[0003] Existing intelligent detection methods for diseases and pests in fragrant pears are ineffective in feature fusion processing due to the different responses of different types of diseases and pests to color, texture, spectrum, temperature, and structural features. This results in low accuracy of disease and pest detection results, making it impossible to achieve lossless processing when harvesting fragrant pears. Summary of the Invention
[0004] To address the shortcomings mentioned in the background art, the present invention aims to provide a collaborative control method and system for non-destructive harvesting of fragrant pears with anti-occlusion visual guidance, which solves the problem that the existing feature fusion processing effect is poor when different types of pests and diseases respond differently to different features.
[0005] Firstly, the objective of this invention can be achieved through the following technical solution: a collaborative control method for non-destructive harvesting of fragrant pears with anti-occlusion visual guidance, the method comprising the following steps: Acquire multi-source image data and positioning data, and perform registration processing on the multi-source image data based on the positioning data to obtain registered multi-source image data. The multi-source image data includes visible light images, multispectral images, thermal infrared images, and depth images, and the positioning data includes three-dimensional coordinate data and attitude data. The target canopy region is delineated based on the registered multi-source image data. Multidimensional features are extracted from the target canopy region to obtain a multidimensional feature set. The pest and disease sensitivity of each dimension feature in the multidimensional feature set is fused with the corresponding dimension feature to obtain a multidimensional fused feature. The multidimensional feature set includes color features, texture features, spectral response features, temperature anomaly features and canopy structure features. Pest and disease sensitivity is used to characterize the ability of the corresponding dimension feature to distinguish different types of pests and diseases. Multi-dimensional fusion features are input into a pre-established pest and disease detection model, and pest and disease detection results are output, including pest and disease categories and the location of pest and disease occurrence.
[0006] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the registration processing of multi-source image data based on positioning data includes initial registration and precise registration.
[0007] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the initial registration is to construct a single pear tree coordinate system based on three-dimensional coordinate data and pose data, so as to establish a spatial correspondence between image data from different sources and the single pear tree coordinate system; The precise registration process: Based on the multi-source image data, the initially registered multi-source image data is referenced, common features are extracted in the overlapping areas of the multi-source images, and the rotation and translation parameters between the initially registered multi-source image data are optimized with the goal of minimizing the reprojection error, and the registered multi-source image data is output.
[0008] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the process of extracting multidimensional features from the target canopy region, comprising: For visible light images, extract color and texture features; for multispectral images, extract spectral response features; for thermal infrared images, extract temperature anomaly features; and for depth images, extract canopy structure features.
[0009] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the calculation formula for the susceptibility of the disease and pest characteristics of each dimension is as follows: In the formula, S k,y Let be the sensitivity of the k-th dimension feature to the y-th type of pests and diseases. The characteristic mean of the healthy sample, The characteristic mean of the diseased samples, The characteristic standard deviation of the healthy sample, The characteristic standard deviation of the diseased samples. This is a preset constant.
[0010] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the multi-dimensional feature set is represented as: Among them, F color For color characteristics, F texture For texture features, F spec For spectral response characteristics, F temp As a characteristic of temperature anomalies, F struct Features of the canopy structure; The susceptibility to pests and diseases based on multidimensional characteristics is represented as follows: Among them, S color S represents the sensitivity of pests and diseases based on color characteristics. texture For the sensitivity of texture features to pests and diseases, S spec S represents the spectral response characteristics of pest and disease susceptibility. temp S represents the susceptibility of pests and diseases to abnormal temperature characteristics. struct The susceptibility of the canopy structure to pests and diseases.
[0011] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the process of fusing the susceptibility of each dimension feature within the multidimensional feature set with the corresponding dimension feature, including: The susceptibility to pests and diseases for each dimension is normalized to obtain the fusion weights for each dimension, as shown in the following formula: In the formula, S represents the fusion weight of the k-th dimension features. k For the k-th dimension feature, the sensitivity to pests and diseases. S is the preset adjustment coefficient. j For the disease and pest sensitivity of the j-th index dimension feature; The multidimensional fused features are obtained by weighting and summing the features of each dimension in the multidimensional feature set based on the fusion weights of each dimension.
[0012] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the pest and disease detection model includes a feature input layer, a feature encoding layer, a self-attention interaction layer, and a multi-task output layer. In this process, the multi-dimensional fusion features are obtained by unifying and normalizing the dimensions of the multi-dimensional fusion features through the feature input layer. The processed multi-dimensional fusion features are mapped to a higher-dimensional semantic space through a feature encoding layer. Deeper pest and disease information within the processed multi-dimensional fusion features is extracted through nonlinear transformation to obtain the encoded features. The encoded features are transformed through a self-attention interaction layer to obtain high-level features, which include query vector, key vector and value vector. High-level features are processed and output through a multi-task output layer. The classification output head of the multi-task output layer outputs the pest and disease category, and the location output head outputs the location of the pest and disease occurrence.
[0013] Secondly, in order to achieve the above objectives, this invention discloses a collaborative control system for non-destructive harvesting of fragrant pears with anti-occlusion visual guidance, comprising: The data acquisition module is used to acquire multi-source image data and positioning data, and to perform registration processing on the multi-source image data based on the positioning data to obtain registered multi-source image data. The multi-source image data includes visible light images, multispectral images, thermal infrared images and depth images, and the positioning data includes three-dimensional coordinate data and attitude data. The data processing module is used to delineate the target canopy region based on the registered multi-source image data, extract multi-dimensional features from the target canopy region to obtain a multi-dimensional feature set, and fuse the pest and disease sensitivity of each dimension feature in the multi-dimensional feature set with the corresponding dimension feature to obtain a multi-dimensional fused feature. The multi-dimensional feature set includes color features, texture features, spectral response features, temperature anomaly features and canopy structure features. The detection output module is used to input multi-dimensional fused features into a pre-established pest and disease detection model and output pest and disease detection results, which include pest and disease categories and the location of pest and disease occurrence.
[0014] The beneficial effects of this invention are: This invention obtains registered multi-source image data by registering the acquired positioning data with multi-source image data. Then, based on the registered multi-source image data, a target canopy region is delineated, and multi-dimensional features are extracted from the target canopy region to obtain a multi-dimensional feature set. Next, the pest and disease sensitivity of each dimension feature in the multi-dimensional feature set is fused with the corresponding dimension feature to obtain a multi-dimensional fused feature. Finally, the multi-dimensional fused feature is input into a pre-established pest and disease detection model to output the pest and disease detection result. This invention realizes the detection of multi-dimensional features of fragrant pears combined with responses to different types of pests and diseases, thereby improving the accuracy of the detection results. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the system structure of the present invention. Detailed Implementation
[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0017] Example 1: like Figure 1 As shown, a collaborative control method for non-destructive harvesting of fragrant pears with anti-occlusion visual guidance is proposed, which includes the following steps: S101: Acquire multi-source image data and positioning data, perform registration processing on the multi-source image data based on the positioning data, and obtain registered multi-source image data, wherein the multi-source image data includes visible light images, multispectral images, thermal infrared images and depth images, and the positioning data includes three-dimensional coordinate data and attitude data; Specifically, the process of acquiring multi-source image data includes the following steps: First, digital modeling is carried out on the pears in the target pear orchard to create a digital file for each tree, which includes plot number, row number, tree number, individual tree geographical coordinates, tree age and variety information. A unique tree number is assigned to each pear tree so that some information can be obtained through the digital file of each pear tree during subsequent data collection, and matching information can be achieved based on the tree number of the pear tree. Rigorous intrinsic parameter calibration and multi-sensor extrinsic parameter calibration are performed on the visible light camera, multispectral camera, thermal infrared camera, depth camera or lidar mounted on the acquisition platform to obtain the focal length, principal point, distortion coefficient and relative pose transformation matrix of each sensor. At the same time, radiometric calibration of the multispectral camera is performed using a standard white board and temperature calibration of the thermal infrared camera is performed using a blackbody radiation source to eliminate environmental radiation errors in subsequent data processing and provide a precise hardware parameter basis for spatial registration of multi-source images.
[0018] The process of rigorously calibrating the intrinsic parameters and extrinsic parameters of visible light cameras, multispectral cameras, thermal infrared cameras, depth cameras, or lidar mounted on the acquisition platform includes: In the pear pest and disease detection scheme based on multi-source image fusion features, the core purpose of camera calibration is to establish the geometric and imaging relationships between visible light cameras, multispectral cameras, thermal infrared cameras, depth cameras, or lidar. This ensures that images acquired by different sensors can accurately correspond to the same canopy area of the same pear tree, providing a reliable foundation for subsequent multi-source image registration, canopy area extraction, and pest and disease feature fusion. Specifically, calibration requires preparing a calibration site and calibration targets. Commonly used targets include checkerboard calibration boards, dotted calibration boards, ChArUco calibration boards, ArUco coded boards, or high-contrast flat targets. For on-site calibration in pear orchards, calibration boards with distinct corner points or geometric boundaries can be selected and placed on the ground, on a support, or in front of the tree canopy, allowing them to be observed by multiple sensors simultaneously. The calibration process should be conducted under conditions of uniform lighting, flat targets, and no significant obstructions to avoid corner point recognition errors caused by strong reflections, shadows, or bent targets.
[0019] For visible light camera calibration, single-camera intrinsic parameter calibration is typically performed first. The specific process involves controlling the visible light camera to capture multiple calibration images of the calibration board from different angles, distances, and orientations, for example, fifteen to thirty images. The system performs corner detection on the calibration images to obtain the pixel coordinates of the calibration board's corner points in the image coordinate system; simultaneously, based on the physical dimensions of the calibration board itself, it establishes the three-dimensional coordinates of the calibration board in the world coordinate system. Then, using the camera imaging model, the world coordinate points are transformed to the camera coordinate system via rotation matrices and translation vectors, and finally projected onto the image pixel coordinate system using the camera intrinsic parameter matrix. The camera intrinsic parameter matrix typically includes horizontal focal length, vertical focal length, principal point x-coordinate, and principal point y-coordinate; distortion parameters typically include radial distortion coefficients and tangential distortion coefficients. The calibration process optimizes the visible light camera's intrinsic and distortion parameters by minimizing the reprojection error between the actual detected pixels and the theoretically projected pixels. After calibration, subsequently acquired visible light images can be distorted and geometrically corrected based on these intrinsic and distortion parameters.
[0020] The calibration process for multispectral cameras is similar to that of visible light cameras. However, due to the lower spatial resolution and weaker texture information of multispectral images, it is necessary to use calibration targets with higher contrast, or to utilize a visible light camera for calibration assistance. In practice, materials with significant differences in reflectivity across different wavelengths can be placed on the calibration board, such as black and white checkerboard patterns, gray and white combination boards, or targets with near-infrared reflectivity. After the multispectral camera images the calibration board, the system extracts the target corner points or region centers and calculates their intrinsic parameters and distortion parameters using the multispectral camera's imaging model. Since multispectral cameras also require reflectivity conversion, radiometric calibration is performed in addition to geometric calibration. Radiometric calibration is typically performed before and after acquisition. Using standard white, gray, or black boards with known reflectivity as references, the raw grayscale values or digitally quantized values output by the multispectral camera are converted into reflectivity values, ensuring the comparability of spectral indices such as NDVI, NDRE, SAVI, and PRI acquired at different times and under different lighting conditions.
[0021] The calibration of thermal infrared cameras mainly includes two aspects: geometric calibration and temperature calibration. Geometric calibration aims to determine the spatial correspondence between thermal infrared images and visible light or multispectral images. Since thermal infrared images have limited texture, direct corner identification is difficult. Therefore, heated targets, temperature difference targets, metal reflectors, or calibration plates with heated areas can be used to create clear boundaries for the targets in the thermal infrared images. Alternatively, the calibration plate can be identified first using a visible light camera, and then the thermal infrared camera calibration can be indirectly completed through multi-sensor extrinsic parameter relationships. Temperature calibration aims to ensure the accuracy and reliability of the temperature values output from the thermal infrared images. Typically, a blackbody radiation source or a reference object with a known temperature is used for two-point or multi-point calibration. During the calibration process, the thermal infrared camera images different temperature reference sources, and the system establishes a mapping relationship between the original grayscale values and the actual temperature, thereby obtaining accurate canopy temperature data in subsequent analysis of pear canopy temperature anomalies.
[0022] The key to calibrating depth cameras or LiDAR lies in establishing the spatial registration relationship between depth data and visible light or multispectral images. If a depth camera is used, its intrinsic parameters and depth accuracy can be calibrated first, followed by extrinsic parameter calibration with a visible light camera. Specifically, both the depth and visible light cameras simultaneously photograph the same calibration board. The visible light camera identifies the corner points of the calibration board, while the depth camera acquires the depth values or point cloud data of the corresponding area on the calibration board. The system then calculates the rotation matrix and translation vector from the depth camera coordinate system to the visible light camera coordinate system based on the projection relationship between the 3D point cloud and 2D pixels. If a LiDAR is used, planar features or corner features can be extracted from the LiDAR point cloud using the calibration board plane, wall, ground, or artificially set geometric targets. These features are then matched with corresponding features in the visible light image to solve the rigid transformation relationship from the LiDAR coordinate system to the camera coordinate system. This allows the subsequent point cloud data to be projected onto the visible light or multispectral images for the extraction of canopy height, canopy volume, point cloud density, and canopy structural features.
[0023] In multi-sensor joint calibration, a reference sensor or reference coordinate system is typically selected. For example, a high-resolution visible light camera can be used as the reference, and data from multispectral cameras, thermal infrared cameras, depth cameras, or LiDAR can be uniformly transformed to the visible light camera coordinate system. Alternatively, the device's own coordinate system or the gimbal coordinate system can be used as the reference, and the pose of each sensor relative to the reference coordinate system can be calculated separately. Specifically, during calibration, the pose of each sensor when observing the same calibration board is first acquired, and then the extrinsic parameters between any two sensors are calculated through coordinate transformation relationships. If the pose of a sensor relative to the world calibration board is a transformation matrix composed of rotation and translation vectors, then the extrinsic parameters between two sensors can be obtained by combining the inverse of the transformation from one sensor to the world coordinate system and the transformation from the other sensor to the world coordinate system. In this way, a stable spatial mapping relationship can be established between visible light, multispectral, thermal infrared, and depth data, providing a foundation for subsequently projecting different images uniformly onto a single-tree canopy coordinate system.
[0024] After the camera is calibrated, the actual data collection phase begins. Fixed data collection is conducted in the morning or afternoon when the light is stable, the wind speed is low, and there is no rain, in order to avoid interference with image quality caused by strong backlight, large areas of shadow, and violent leaf shaking. A drone-type data collection platform is used to fly above the pear tree in both orthogonal and oblique directions to obtain high-resolution visible light images, multispectral images, thermal infrared temperature distribution maps, and depth images for three-dimensional reconstruction of the top of the pear canopy. This allows for a macroscopic understanding of the overall growth of the orchard and the characteristics of the top of the canopy. Furthermore, the ground inspection robot's data collection platform moves at a constant speed along the rows of fruit trees in the pear orchard. Using a lifting gimbal, it collects high-resolution images of the sides, lower and middle branches and leaves, and hidden canopy of a single pear tree from multiple angles within a preset distance from the tree canopy. These images include visible light images, multispectral images, thermal infrared temperature distribution maps, and depth images for three-dimensional reconstruction. Specifically, the multiple angles in this embodiment can be selected as eye-level, upward-looking, and oblique-looking. The process of acquiring positioning data is triggered synchronously by combining the camera acquisition timestamp. The coordinate positioning is associated with the image data acquired by the camera for each frame. The three-dimensional coordinate data of the pear is obtained through the GNSS module and the attitude data of the pear is obtained through the MU inertial measurement unit. The three-dimensional coordinate data is used to represent the spatial position of the acquisition device or sensor in the orchard geographic coordinate system, and the attitude data is used to represent the attitude direction of the acquisition device in space. The process of registering multi-source image data based on positioning data includes initial registration and precise registration, specifically including the following steps: The initial registration process includes: A coordinate system for a single pear tree is constructed based on 3D coordinate data and pose data to establish a spatial correspondence between image data from different sources and the coordinate system for a single pear tree. In one implementation, the process of constructing a coordinate system for a single pear tree based on three-dimensional coordinate data and attitude data includes: Establish the pose relationship of each sensor in a unified geographic coordinate system based on 3D coordinates and attitude data.
[0025] Specifically, this includes: determining the position and orientation of visible light images, multispectral images, thermal infrared images, and depth images relative to a unified geographic coordinate system based on three-dimensional coordinates and attitude data and pre-calibrated camera parameters, thereby determining the initial spatial transformation relationship between the images, and initially projecting multi-source images from different sources onto the same single-tree canopy coordinate system to complete the initial registration; Specifically, the process of determining the location includes: for any sensor, its imaging process can be represented as the world coordinate point being rotated and translated into the camera coordinate system, and then projected onto the image pixel coordinate system through an intrinsic parameter matrix. Based on the positioning data, the position of the camera's optical center in the orchard's geographic coordinate system is obtained by calculating the rotation matrix of the positioning acquisition device's body coordinate system relative to the geographic coordinate system and the product of the positioning acquisition device and the camera's mounting, plus the three-dimensional coordinates. The precise registration process includes: Based on multi-source image data, the initially registered multi-source image data is referenced, and common features are extracted in the overlapping areas of the multi-source images. Common features include canopy edges, branch outlines, corners, lesion boundaries, or temperature anomaly boundaries. The rotation and translation parameters between the initially registered multi-source image data are optimized with the goal of minimizing the reprojection error. This enables the initially registered multi-source image data to achieve pixel-level or region-level alignment within the canopy area of a single tree. Finally, a registered image that can be used for subsequent feature extraction and fusion recognition is obtained, and the registered multi-source image data is output. In this embodiment, the optimization objective is to minimize the reprojection error, continuously adjusting the rotation, translation, or homography parameters between images. Specifically, if the error is large, it indicates that the current transformation relationship is inaccurate, and the system will gradually correct the parameters through optimization algorithms; if the error gradually decreases and tends to stabilize, it indicates that the spatial correspondence between the multi-source images is gradually aligned. When the reprojection error meets a preset threshold, optimization stops, and the final rotation, translation, or homography parameters are used as the fine registration result. Through this process, visible light, multispectral, thermal infrared, and depth images or point cloud projection images can achieve more accurate spatial alignment within the same single-tree canopy area. S102: Delineate the target canopy region based on the registered multi-source image data, extract multi-dimensional features from the target canopy region to obtain a multi-dimensional feature set, and fuse the pest and disease sensitivity of each dimension feature in the multi-dimensional feature set with the corresponding dimension feature to obtain a multi-dimensional fused feature. The multi-dimensional feature set includes color features, texture features, spectral response features, temperature anomaly features and canopy structure features. The pest and disease sensitivity is used to characterize the ability of the corresponding dimension feature to distinguish different types of pests and diseases. The target canopy region is delineated based on the registered multi-source image data, which is based on the selection of the area to be detected for diseases and pests in pears. The process of extracting multidimensional features from the target canopy region includes: For images within the target canopy area, color features are first extracted from the visible light image to characterize phenotypic changes such as leaf yellowing, browning, and lesions. Secondly, texture features are extracted from the visible light image to characterize local texture anomalies such as leaf roughness, necrosis, insect damage, or notches. Next, spectral response features, such as NDVI, NDRE, SAVI, and PRI, are calculated from the multispectral image to characterize changes in chlorophyll content and vegetation vitality. Then, temperature anomaly features are extracted from the thermal infrared image to characterize localized canopy warming, water stress, or stomatal anomalies. Finally, canopy structural features are extracted from the depth image to characterize canopy sparseness, holes, leaf drop, or structural damage. The extracted color features, texture features, spectral response features, and temperature anomaly features—i.e., canopy structural features—form a multidimensional feature set, serving as the basis for subsequent adaptive fusion and pest / disease identification. The calculation process for susceptibility to pests and diseases based on various dimensions includes: For a candidate pest or disease category y, the distribution of healthy and diseased samples on the k-th dimension feature is statistically analyzed to determine the sensitivity of that dimension feature relative to this type of pest or disease. The specific calculation formula is as follows: In the formula, S k,y Let be the sensitivity of the k-th dimension feature to the y-th type of pests and diseases. The characteristic mean of the healthy sample, The characteristic mean of the diseased samples, The characteristic standard deviation of the healthy sample, The characteristic standard deviation of the diseased samples. This is a preset constant used to prevent the letters in the formula from being 0; The specific meaning of this formula is that the greater the difference in the mean of features between diseased samples and healthy samples, the higher the sensitivity; the greater the fluctuation of the sample itself, the lower the sensitivity. The characteristic mean of healthy samples and the characteristic mean of diseased samples are obtained by summing the dimensional characteristic values of healthy samples or diseased samples and dividing them by the number of healthy samples or diseased samples after determining the number of healthy samples or diseased samples. The characteristic standard deviation of healthy samples and the characteristic standard deviation of diseased samples are both obtained by summing the squared differences between the characteristic values of healthy or diseased samples and the characteristic mean of healthy or diseased samples, and then dividing by the number of healthy or diseased samples. The process of fusing the susceptibility to pests and diseases of each dimension feature within a multidimensional feature set with the corresponding dimension feature includes: The multi-dimensional feature set is represented as: Among them, F color For color characteristics, F texture For texture features, F spec For spectral response characteristics, F temp As a characteristic of temperature anomalies, F struct Features of the canopy structure; The susceptibility to pests and diseases based on multi-dimensional characteristics is expressed as: Among them, S color S represents the sensitivity of pests and diseases based on color characteristics. texture For the sensitivity of texture features to pests and diseases, S spec S represents the spectral response characteristics of pest and disease susceptibility. temp S represents the susceptibility of pests and diseases to abnormal temperature characteristics. structSensitivity to pests and diseases based on canopy structure characteristics; The susceptibility to pests and diseases for each dimension is normalized to obtain the fusion weights for each dimension, as shown in the following formula: In the formula, S represents the fusion weight of the k-th dimension features. k For the k-th dimension feature, the sensitivity to pests and diseases. S is the preset adjustment coefficient. j For the disease and pest sensitivity of the j-th index dimension feature; The denominator of the formula is the sum of the dimensional features of index j. The preset adjustment coefficient is mainly used to control the smoothness of the distribution of weights of each dimension during feature fusion. When the environmental interference is small and the sensitivity and confidence of a certain feature are extremely high, the value is automatically reduced to amplify the weight advantage of the sensitive feature and make the weight distribution sharper. When the sensitivity of each feature is similar or the orchard environment is complex and changeable, the value is increased to make the weight distribution more uniform. The multi-dimensional fused features are obtained by weighting and summing the features based on the fusion weights of each dimension. S103: Input the multi-dimensional fusion features into the pre-established pest and disease detection model and output the pest and disease detection results, which include the pest and disease category and the location of the pest and disease occurrence.
[0026] The pest and disease detection model is constructed using an attention-based Transformer network, such as a lightweight Transformer encoder or a Vision Transformer structure. This application chooses a lightweight Transformer encoder because the fused features of pests and diseases contain multi-modal information, and the Transformer can effectively model the correlation between different feature dimensions. The pest and disease detection model as a whole may include a feature input layer, a feature encoding layer, a self-attention interaction layer, and a multi-task output layer. The process of inputting multi-dimensional fused features into a pre-established pest and disease detection model includes: By unifying and normalizing the dimensions of the multi-dimensional fused features through the feature input layer, the processed multi-dimensional fused features are obtained. The processed multi-dimensional fusion features are mapped to a higher-dimensional semantic space through a feature encoding layer. Deeper pest and disease information within the processed multi-dimensional fusion features is extracted through nonlinear transformation to obtain the encoded features. The encoded features are transformed through a self-attention interaction layer to obtain high-level features, which include query vectors, key vectors, and value vectors. This layer is mainly used to explore the correlations between features of different dimensions and determine which information among color, texture, spectrum, temperature, and structural features is more important for current pest and disease identification. For example, when spectral anomalies and temperature anomalies occur simultaneously, the model can enhance the correlation between them; when a certain dimension feature is greatly affected by environmental interference, the model can reduce its influence weight, thereby improving the stability of the identification results. High-level features are processed and output through a multi-task output layer. The classification output head of the multi-task output layer outputs the pest and disease category, and the positioning output head outputs the location of the pest and disease, such as bounding box, region number or canopy orientation. Specifically, in this embodiment, the training process of the pest and disease detection model includes: By constructing a sample set containing source images, canopy regions, fused features, and pest and disease detection result labels, the sample set is divided into a training set, a validation set, and a test set. The training set is input into the pest and disease detection model, which outputs the detection category and detection location. Then, the loss between the detection result and the true label is calculated, where the loss function includes classification cross-entropy loss, localization regression loss, and ranking classification loss. The model parameters of the pest and disease detection model are then continuously updated through backpropagation and an optimizer, allowing the model to gradually learn the response patterns of different pests and diseases under multi-source features. After reaching the preset training epochs, training is complete. The pest and disease detection model is evaluated using the validation set and the test set, ultimately obtaining a well-trained pest and disease detection model that can be used for detecting pests and diseases in pears. In this embodiment, the evaluation metrics of the pest and disease detection model may include recognition accuracy, recall, F1 score, localization IoU, and ranking accuracy.
[0027] Specifically, the reasons for the improved recognition accuracy of this invention are as follows: Early symptoms of diseases and pests in Korla pears are usually weak, and a single visible light image is insufficient to fully reflect the physiological changes inside the leaves. This method uses multi-source fusion of visible light, multispectral, thermal infrared, depth images, or point cloud data to complement color anomalies, decreased spectral viability, temperature anomalies, and changes in canopy structure, thereby improving the ability to identify early diseases and pests.
[0028] Secondly, the orchard background is complex, with soil, shadows, weeds, branches, etc., easily causing misidentification. This solution first extracts the target canopy area and performs feature calculations only within the effective canopy area, which can reduce background interference, increase the proportion of effective features, and thus improve the recognition accuracy.
[0029] Furthermore, different pests and diseases respond differently to color, texture, spectral response, temperature anomalies, and canopy structure features. This approach calculates the pest and disease sensitivity of features across each dimension and determines the fusion weights based on this sensitivity. This gives higher weights to features that are more sensitive to the current pest or disease, suppressing the influence of noisy or weakly correlated features, thereby improving the model's ability to distinguish target pests and diseases.
[0030] Then, based on the pest and disease detection results, combined with the registered multi-source image data, anti-occlusion visual guidance is performed to generate a non-destructive harvesting collaborative control strategy to control the harvesting execution mechanism to complete the non-destructive harvesting of fragrant pears.
[0031] Specifically, due to the dense canopy of the fragrant pear, the branches and leaves severely obstruct the view, and pests and diseases (such as black spot, ring rot, and fruit borer) can cause local tissue fragility, abnormal morphology, or the presence of pathogenic spores in the fruit. Traditional single-visual harvesting methods are highly susceptible to mechanical damage to the fruit or cross-infection of diseases. This embodiment achieves anti-shading guidance and non-destructive collaborative control by deeply integrating pest and disease detection results with multi-source spatial data. The specific process includes: (1) Three-dimensional reconstruction of shading-resistant fruits and assessment of their suitability for harvesting: Based on the registered depth image and visible light image, a three-dimensional canopy point cloud model of a single fragrant pear tree is constructed. The depth data is used to penetrate or identify the spatial location of fragrant pear fruits that are covered by branches and leaves. At the same time, the output pest and disease detection results (including pest and disease types, occurrence locations and lesion boundaries) are mapped to the three-dimensional canopy point cloud model to generate a three-dimensional fruit model with pest and disease distribution labels. Based on the three-dimensional model of the fruit with the disease and pest distribution label, the suitability for picking is assessed: if the disease and pest category is a serious infectious disease or the area of the lesion exceeds a preset threshold, it is marked as isolated picking or removal; if it is a healthy fruit or only has slight surface defects, it is marked as non-destructive picking, and the three-dimensional coordinates, posture and three-dimensional spatial distribution information of the lesion of the target fruit are extracted.
[0032] (2) Anti-occlusion path and non-destructive grasping pose planning based on the distribution of pests and diseases: Anti-occlusion path planning: Taking the three-dimensional coordinates of the target fruit as the endpoint, and combining the distribution of physical occlusions such as branches and dense leaves in the three-dimensional canopy point cloud model, a three-dimensional path planning algorithm (such as RRT or A algorithm) is used to generate a collision-free approach path for the robotic arm, so as to achieve visual-guided anti-occlusion obstacle avoidance. Non-destructive grasping posture planning: Based on the location of pests and diseases and the vulnerability of the fruit, the grasping posture of the end effector is planned. For example, when the pest and disease detection model outputs that the lesion is located on the left side of the fruit, the collaborative control algorithm will drive the robotic arm to adjust its posture so that the flexible end effector can approach the fruit from the right side (healthy side) in a wrapping manner, ensuring that the contact point of the gripper or suction cup avoids the lesion area, preventing the lesion from cracking, juice from flowing out, or fungal spores from spreading due to compression.
[0033] (3) Multimodal collaborative harvesting control execution: As the harvesting mechanism approaches and contacts the target fruit, multimodal collaborative control is activated: Visual collaborative guidance: Real-time updates of fruit pose deviation using registered multi-source images, and closed-loop visual servo correction. Force and temperature coordinated feedback: Combining the abnormal temperature characteristics of the fruit surface extracted from thermal infrared images (pests and diseases often cause local temperature anomalies and tissue softening), and feedback from the torque sensor on the end effector, the gripping force is dynamically adjusted. If the target area is detected to be softened due to pests and diseases, the coordinated control system automatically reduces the gripping force threshold in that area, achieving truly non-destructive gripping; Ultimately, the pears are harvested and placed using the coordinated actions of the robotic arm and the end effector.
[0034] Example 2: Figure 2 As shown, the anti-occlusion visual guidance collaborative control system for non-destructive harvesting of fragrant pears includes: The data acquisition module 11 is used to acquire multi-source image data and positioning data, and to perform registration processing on the multi-source image data based on the positioning data to obtain registered multi-source image data. The multi-source image data includes visible light images, multispectral images, thermal infrared images and depth images, and the positioning data includes three-dimensional coordinate data and attitude data. Data processing module 12 is used to delineate the target canopy region based on the registered multi-source image data, extract multi-dimensional features from the target canopy region to obtain a multi-dimensional feature set, and fuse the disease and pest sensitivity of each dimension feature in the multi-dimensional feature set with the corresponding dimension feature to obtain a multi-dimensional fused feature. The multi-dimensional feature set includes color features, texture features, spectral response features, temperature anomaly features and canopy structure features. The detection output module 13 is used to input multi-dimensional fusion features into a pre-established pest and disease detection model and output pest and disease detection results, which include pest and disease categories and the location of pest and disease occurrence.
[0035] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., serving as the computing and control core of the terminal, and is used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the aforementioned method.
[0036] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0037] The above formulas are all numerical calculations after removing dimensions. The formulas are obtained by software simulation based on a large amount of data and are closest to the real situation. The preset parameters and preset thresholds in the formulas are set by those skilled in the art according to the actual situation or obtained by simulation based on a large amount of data.
[0038] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0039] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.
Claims
1. A collaborative control method for non-destructive harvesting of fragrant pears with anti-occlusion visual guidance, characterized in that, The method includes the following steps: Acquire multi-source image data and positioning data, and perform registration processing on the multi-source image data based on the positioning data to obtain registered multi-source image data. The multi-source image data includes visible light images, multispectral images, thermal infrared images, and depth images, and the positioning data includes three-dimensional coordinate data and attitude data. The target canopy region is delineated based on the registered multi-source image data. Multidimensional features are extracted from the target canopy region to obtain a multidimensional feature set. The pest and disease sensitivity of each dimension feature in the multidimensional feature set is fused with the corresponding dimension feature to obtain a multidimensional fused feature. The multidimensional feature set includes color features, texture features, spectral response features, temperature anomaly features and canopy structure features. Pest and disease sensitivity is used to characterize the ability of the corresponding dimension feature to distinguish different types of pests and diseases. Multi-dimensional fusion features are input into a pre-established pest and disease detection model, and pest and disease detection results are output, including pest and disease categories and the location of pest and disease occurrence.
2. The anti-occlusion visual guidance collaborative control method for non-destructive harvesting of fragrant pears according to claim 1, characterized in that, The registration process for multi-source image data based on positioning data includes initial registration and precise registration.
3. The anti-occlusion visual guidance collaborative control method for non-destructive harvesting of fragrant pears according to claim 2, characterized in that, The initial registration is to construct a single pear tree coordinate system based on three-dimensional coordinate data and attitude data, so as to establish a spatial correspondence between image data from different sources and the single pear tree coordinate system; The precise registration process: Based on the multi-source image data, the initially registered multi-source image data is referenced, common features are extracted in the overlapping areas of the multi-source images, and the rotation and translation parameters between the initially registered multi-source image data are optimized with the goal of minimizing the reprojection error, and the registered multi-source image data is output.
4. The anti-occlusion visual guidance collaborative control method for non-destructive harvesting of fragrant pears according to claim 1, characterized in that, The process of extracting multidimensional features from the target canopy region includes: For visible light images, extract color and texture features; for multispectral images, extract spectral response features; for thermal infrared images, extract temperature anomaly features; and for depth images, extract canopy structure features.
5. The anti-occlusion visual guidance collaborative control method for non-destructive harvesting of fragrant pears according to claim 1, characterized in that, The formulas for calculating the susceptibility to pests and diseases for each dimension of the characteristics are as follows: In the formula, S k,y Let be the sensitivity of the k-th dimension feature to the y-th type of pests and diseases. The characteristic mean of the healthy sample, The characteristic mean of the diseased samples, The characteristic standard deviation of the healthy sample, The characteristic standard deviation of the diseased samples. This is a preset constant.
6. The anti-occlusion visual guidance collaborative control method for non-destructive harvesting of fragrant pears according to claim 1, characterized in that, The multi-dimensional feature set is represented as follows: Among them, F color For color characteristics, F texture For texture features, F spec For spectral response characteristics, F temp F is a characteristic of temperature anomalies. struct Features of the canopy structure; The susceptibility to pests and diseases based on multidimensional characteristics is expressed as follows: Among them, S color S represents the sensitivity of pests and diseases based on color characteristics. texture For the sensitivity of texture features to pests and diseases, S spec S represents the spectral response characteristics of pest and disease susceptibility. temp S represents the susceptibility of pests and diseases to abnormal temperature characteristics. struct The susceptibility to pests and diseases is a characteristic of the canopy structure.
7. The anti-occlusion visual guidance collaborative control method for non-destructive harvesting of fragrant pears according to claim 6, characterized in that, The process of fusing the susceptibility to pests and diseases of each dimension feature within the multidimensional feature set with the corresponding dimension feature includes: The susceptibility to pests and diseases for each dimension is normalized to obtain the fusion weights for each dimension, as shown in the following formula: In the formula, S represents the fusion weight of the k-th dimension features. k For the k-th dimension feature, the sensitivity to pests and diseases. S is the preset adjustment coefficient. j For the disease and pest sensitivity of the j-th index dimension feature; The multidimensional fused features are obtained by weighting and summing the features of each dimension in the multidimensional feature set based on the fusion weights of each dimension.
8. The anti-occlusion visual guidance collaborative control method for non-destructive harvesting of fragrant pears according to claim 1, characterized in that, The pest and disease detection model includes a feature input layer, a feature encoding layer, a self-attention interaction layer, and a multi-task output layer. In this process, the multi-dimensional fusion features are obtained by unifying and normalizing the dimensions of the multi-dimensional fusion features through the feature input layer. The processed multi-dimensional fusion features are mapped to a higher-dimensional semantic space through a feature encoding layer. Deeper pest and disease information within the processed multi-dimensional fusion features is extracted through nonlinear transformation to obtain the encoded features. The encoded features are transformed through a self-attention interaction layer to obtain high-level features, which include query vector, key vector and value vector. High-level features are processed and output through a multi-task output layer. The classification output head of the multi-task output layer outputs the pest and disease category, and the location output head outputs the location of the pest and disease occurrence.
9. A collaborative control system for non-destructive harvesting of fragrant pears with anti-occlusion visual guidance, comprising the collaborative control method for non-destructive harvesting of fragrant pears with anti-occlusion visual guidance as described in any one of claims 1 to 8, characterized in that, include: The data acquisition module is used to acquire multi-source image data and positioning data, and to perform registration processing on the multi-source image data based on the positioning data to obtain registered multi-source image data. The multi-source image data includes visible light images, multispectral images, thermal infrared images and depth images, and the positioning data includes three-dimensional coordinate data and attitude data. The data processing module is used to delineate the target canopy region based on the registered multi-source image data, extract multi-dimensional features from the target canopy region to obtain a multi-dimensional feature set, and fuse the pest and disease sensitivity of each dimension feature in the multi-dimensional feature set with the corresponding dimension feature to obtain a multi-dimensional fused feature. The multi-dimensional feature set includes color features, texture features, spectral response features, temperature anomaly features and canopy structure features. The detection output module is used to input multi-dimensional fused features into a pre-established pest and disease detection model and output pest and disease detection results, which include pest and disease categories and the location of pest and disease occurrence.
10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, The memory stores a computer program that can run on a processor. When the processor loads and executes the computer program, it employs the anti-occlusion visual guidance collaborative control method for non-destructive harvesting of fragrant pears as described in any one of claims 1 to 8.
