Pruning analysis method and system based on deep learning

Abstract

The invention discloses a pruning analysis method and system based on deep learning, and the method comprises the following steps: S1, collecting images of fruit tree branches through an image collection device, uploading the image data to a cloud end, and constructing three-dimensional positions between the fruit tree branches at the cloud end; s2, compensating image offset caused in the shooting process, and correcting image blurring; s3, extracting key features in the images according to the obtained fruit tree branch images, comparing the key features with historical data, and matching the key features with the historical data to obtain historical data with the same conditions; S4, predicting branch development conditions after different pruning modes, predicting subsequent fruit generation conditions, and determining a pruning strategy. According to the pruning analysis method and system based on deep learning provided by the invention, the problems existing in the fruit tree branches in the image can be found, and the corresponding pruning mode is given, so that the pruning difficulty is reduced, and according to the pruning mode given by the system, people lacking experience can also achieve the pruning effect of professional fruit farmers.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and specifically relates to a pruning analysis method and system based on deep learning. Background Art

[0002] Currently, fruit tree pruning primarily relies on manual labor performed by experienced fruit growers. This traditional method presents numerous limitations in practical application. As the fruit cultivation industry develops towards scale and intensification, the drawbacks of traditional pruning methods are becoming increasingly prominent, necessitating an urgent need for technological innovation to address the bottlenecks facing the industry's development. The current state of the industry, however, faces a severe shortage of skilled pruning professionals due to the long training cycle and high technical barriers to entry. According to surveys, a qualified fruit tree pruner typically requires several years of practical experience to master the essentials of pruning various types of fruit trees. This talent shortage is particularly acute in areas with concentrated fruit plantings, severely impacting the progress of pruning operations. Regarding economic benefits, professional fruit farmers face rising labor costs, significantly compressing their operating profit margins.

[0003] Moreover, due to the limited number of professional pruning personnel, large-scale orchards often face the dilemma of being unable to complete pruning on schedule. From the perspective of agronomic requirements, scientific pruning is crucial to the growth of fruit trees. Reasonable pruning can optimize the tree structure, improve ventilation and light conditions, promote flower bud differentiation, and improve fruit quality and yield. At the same time, timely removal of diseased and insect-infested branches, weak branches, and overgrown branches can effectively reduce the incidence of pests and diseases and reduce the use of pesticides. If pruning is improper or the pruning time is delayed, it will at best affect the yield of the year, and at worst lead to the decline of tree vigor, causing long-term economic losses. In the face of these realistic challenges, the market urgently needs an intelligent pruning solution to alleviate the pressure of professional talent shortage, reduce production costs, and ensure the standardization and timeliness of pruning operations, providing technical support for the sustainable development of the modern fruit industry. Therefore, there is a need for an intelligent pruning analysis method to reduce the difficulty of pruning so that inexperienced people can also achieve the pruning effect of professional fruit farmers. The present invention solves this technical problem. Summary of the Invention

[0004] The present invention provides a pruning analysis method and system based on deep learning, which can collect images of fruit tree branches and analyze the images to find problems with the fruit tree branches in the images and provide corresponding pruning methods, thereby reducing the difficulty of pruning. According to the pruning method provided by the system, even inexperienced people can achieve the same pruning effect as professional fruit farmers.

[0005] A pruning analysis method based on deep learning includes the following steps:

[0006] S1. Capture images of fruit tree branches using an image acquisition device, upload the image data to the cloud, and construct the three-dimensional positions of the fruit tree branches in the cloud;

[0007] S2, compensate for the image deviation caused during the shooting process and correct the image blur;

[0008] S3. Extract key features from the obtained fruit tree branch image, compare the key features with historical data, and match them with historical data with the same situation;

[0009] S4. Predict the branch development after different pruning methods, predict the subsequent fruit production, and determine the pruning strategy;

[0010] S5. Determine the influencing factors related to the fruit tree based on the pruning method and the predicted fruit production. Design a reward function based on the influencing factors to guide model learning.

[0011] S6. The cloud transmits information related to the pruning strategy back to the display device, and the user performs pruning work based on the information obtained;

[0012] S7. Optimize the model structure to increase the model inference speed, reduce its memory usage, and improve the model's generalization ability.

[0013] Furthermore, the step S1 specifically includes the following steps:

[0014] S11. The user collects images of fruit tree branches using a binocular camera to obtain image 1 and image 2, and uploads image 1 and image 2 to the cloud;

[0015] S12, aligning the epipolar lines of the first image and the second image so that corresponding points in the images are located on the same horizontal scan line, to simplify the calculation of the disparity;

[0016] S13, calculate the intrinsic and extrinsic parameters of the camera, and calculate the horizontal displacement of corresponding points in image 1 and image 2, i.e., parallax. Based on this parallax, the distance between branches is calculated by Euclidean distance, and a depth map is generated based on the distance between branches to mark the spatial coordinates of each fruit in the image;

[0017] S14, converting the pixels in the image into a three-dimensional point cloud, restoring the three-dimensional structure of the fruit tree, and distinguishing overlapping branches and fruits in the image;

[0018] S15. Identify the physiological state of fruit tree flower buds through near-infrared spectroscopy for subsequent prediction of fruit growth location.

[0019] Furthermore, the step S2 specifically includes the following steps:

[0020] S21, the image acquisition device includes a binocular camera, a depth sensor and an inertial sensor, and data from the depth sensor and the inertial sensor are simultaneously acquired during the process of acquiring images of fruit tree branches;

[0021] S22, unifying the image data, the data collected by the depth sensor, and the data collected by the inertial sensor to the same timestamp;

[0022] S23. The original image is subjected to interference caused by uneven lighting, shaking branches and leaves, and camera noise through Gaussian blur filtering to eliminate image noise, enhance the contrast of the diseased spots on the branches of the fruit trees, highlight the target features, and correct image deformation caused by lens distortion or tilted viewing angle.

[0023] Furthermore, the step S22 specifically includes the following steps:

[0024] S221, the image data at time T is expressed as I(T), Indicates the timestamp collected by the binocular camera;

[0025] S222, when <T< ,but:

[0026] ;

[0027] S223 : Calculate the data collected by the depth sensor and the data collected by the inertial sensor in the above manner, so that the image data, depth data, and inertial data are aligned at the same time T.

[0028] Furthermore, the step S3 specifically includes the following steps:

[0029] S31, inputting fruit tree variety genetic data, weather history records, soil moisture data, and pruning history data into a cloud database;

[0030] S32. Extract key features of the fruit tree in the image, including the structure of the main and side branches, the color of the lesions, and the degree of branch bending;

[0031] S33. Based on the extracted key features, the key features are compared with pruning records of fruit trees of the same variety and age in the cloud database to obtain information on the location and quantity of subsequent results corresponding to the pruning records.

[0032] Furthermore, the step S4 specifically includes the following steps:

[0033] S41. Build a branch development prediction model. This technology stack uses LSTM and an attention mechanism to input the extracted key features of fruit tree branches into the model, along with future weather conditions. Based on this input data and combined with historical data consistent with the current situation, the model predicts and outputs information on the length, thickness, probability of flower bud differentiation, and potential disease risk of new branches.

[0034] S42. Build a fruit generation model. This technology stack uses a generative adversarial network. The generator generates a virtual fruit distribution based on the spatial topology of the branches. The discriminator compares this with historical data from the real orchard to verify the rationality of the virtual fruit distribution. It then outputs a probability cloud of the fruit's three-dimensional coordinates and predicts fruit data based on historical data.

[0035] S43. Establish a pruning strategy optimization model based on the deep reinforcement learning framework and adopt a near-segment strategy optimization algorithm to describe the real-time characteristics of fruit trees, including the morphological characteristics, physiological characteristics, and environmental characteristics of fruit tree branches, determine feasible pruning methods, and determine the pruning position, incision length, stump length, and pruning priority based on the predicted situation after pruning.

[0036] Furthermore, the step S5 specifically includes the following steps:

[0037] S51. Design a multi-dimensional quantitative evaluation system, including yield factors, quality factors, damage factors, and energy consumption factors related to fruit trees;

[0038] S52. The yield factor is calculated as follows: predicted number of flower buds × historical fruit setting rate × fruit tree variety characteristic coefficient;

[0039] S53, the quality factor is calculated as follows: expected fruit uniformity + percentage of colored area - disease probability;

[0040] S54, the damage factor is calculated as follows: total area of ​​the trimmed wound × risk factor of incision infection;

[0041] S55. Taking the mechanical work required for pruning as the energy consumption factor;

[0042] S56. Design the reward function through the above factors to guide model learning.

[0043] Furthermore, the step S56 specifically includes the following steps:

[0044] S561. Assign corresponding weights to each factor 、 、 、 ,in, + + + =1;

[0045] S562. The reward function R is calculated using the following formula:

[0046] ;

[0047] in, represents the yield factor, represents the quality factor, represents the damage factor, represents the energy consumption factor; 、 、 、 They respectively represent the maximum value of the corresponding factors in the historical data.

[0048] Furthermore, step S6 specifically includes the following steps:

[0049] S61. After the cloud completes data processing and analysis, it compresses the data and transmits it back to the display device.

[0050] S62. The display device decodes the data and renders it after decoding. The rendered data is projected into the human eye through optical waveguide technology to achieve visual fusion of the human eye.

[0051] S63. The optical waveguide is divided into an input area, a pupil expansion area and an output area. The light is deflected through the input area to make it fully reflected in the waveguide, the effective display area is expanded through the pupil expansion area, and the light is gradually guided toward the direction of the human eye through the output area.

[0052] Furthermore, the step S7 specifically includes the following steps:

[0053] S71. In channel pruning of a convolutional neural network, calculate the L1 norm of the weight matrix of each channel;

[0054] S72, removing channels whose L1 norm is less than a set value from the model to achieve model compression;

[0055] S73. In the batch normalization layer, obtain the scaling factor gamma of each channel, sort them according to the size of the gamma value, remove the channels corresponding to the gamma value smaller than the set value, and adjust the number of input channels of the subsequent layer.

[0056] A deep learning-based pruning analysis system, based on the above-mentioned deep learning-based pruning analysis method, includes an image acquisition device and a display device connected to the image acquisition device, the display device uses AR glasses, the image display device includes a binocular camera, a depth sensor and an inertial sensor connected to the AR glasses, and the AR glasses are connected to the cloud server via a network.

[0057] The technical effects of the present invention are as follows:

[0058] (1) The present invention collects images of fruit tree branches through image acquisition equipment, analyzes the image data through the cloud, understands the problems existing in the fruit tree branches, and provides corresponding pruning strategies. The pruning strategies can be displayed through a display device, so that fruit farmers can understand how to prune specifically, making the fruit tree pruning method more intelligent, reducing the difficulty of pruning, and allowing inexperienced people to achieve the pruning effect of professional fruit farmers;

[0059] (2) The present invention can unify the timestamps of the image data collected by the binocular camera and the data of the depth sensor and inertial sensor, align the color of the RGB image with the distance information of the depth sensor, and use the motion data of the inertial sensor to compensate for the image offset caused by the user's movement, thereby correcting the image blur, which is conducive to the cloud to perform accurate analysis based on the corrected image;

[0060] (3) The present invention can obtain the depth map of the image based on the image captured by the binocular camera and combined with the Mahalanobis distance, thereby restoring the three-dimensional structure of the fruit tree, which is conducive to accurately analyzing the shape and number of branches, as well as the size and number of fruits, so as to make the subsequent prediction results more accurate;

[0061] (4) The present invention can compare the branches in the image with the branches in the historical data that have the same conditions, and query the subsequent corresponding results of the historical data, so as to achieve the prediction of the current branches, and can determine the pruning priority according to the results of different pruning methods in the historical data;

[0062] (5) The present invention can remove channels that contribute less to the model and have less impact on subsequent layers, making the model lightweight while having less impact on the output results. This reduces the complexity of data calculation, improves the model's reasoning speed, and reduces the model's memory usage. It can also make the model more concise and improve the model's generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is the workflow diagram of the present invention.

[0064] Figure 2It is a structural block diagram of the present invention. DETAILED DESCRIPTION

[0065] The technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments and drawings.

[0066] See also Figure 1 , a pruning analysis method based on deep learning, comprising the following steps:

[0067] S1. Capture images of fruit tree branches using an image acquisition device, upload the image data to the cloud, and construct the three-dimensional positions of the fruit tree branches in the cloud;

[0068] S2, compensate for the image deviation caused during the shooting process and correct the image blur;

[0069] S3. Extract key features from the obtained fruit tree branch image, compare the key features with historical data, and match them with historical data with the same situation;

[0070] S4. Predict the branch development after different pruning methods, predict the subsequent fruit production, and determine the pruning strategy;

[0071] S5. Determine the influencing factors related to the fruit tree based on the pruning method and the predicted fruit production. Design a reward function based on the influencing factors to guide model learning.

[0072] S6. The cloud transmits information related to the pruning strategy back to the display device, and the user performs pruning work based on the information obtained;

[0073] S7. Optimize the model structure to increase the model inference speed, reduce its memory usage, and improve the model's generalization ability.

[0074] Furthermore, step S1 specifically includes the following steps:

[0075] S11. The user uses a binocular camera to capture images of fruit tree branches, obtaining Image 1 and Image 2, and uploads Image 1 and Image 2 to the cloud. Using lightweight object detection models such as YOLOv8n and NanoDet, the user performs real-time analysis of the video stream output by the camera to identify objects such as fruit tree branches and fruits.

[0076] S12, aligning the epipolar lines of the first image and the second image so that corresponding points in the images are located on the same horizontal scan line, to simplify the calculation of the disparity;

[0077] S13. Using the Zhang Zhengyou calibration method, calculate the camera's intrinsic and extrinsic parameters, and calculate the horizontal displacement of corresponding points in image 1 and image 2, i.e., the disparity. Based on this disparity, calculate the distance between branches using the Euclidean distance, and generate a depth map based on the distance between branches to mark the spatial coordinates of each fruit in the image. This can be specifically implemented in combination with the Bouguet algorithm.

[0078] S14. Convert the pixels in the image into a 3D point cloud to restore the three-dimensional structure of the fruit tree and distinguish overlapping branches and fruits in the image. Specifically, Mask R-CNN can be used to achieve high-precision detection, output instance segmentation masks, and distinguish overlapping fruits.

[0079] S15. Identify the physiological state of fruit tree flower buds through near-infrared spectroscopy for subsequent prediction of fruit growth location.

[0080] The reflectance of healthy flower buds shows characteristic peaks in specific bands, while the spectral curve of degraded flower buds is flat or fluctuates abnormally. The obtained data will be uploaded to the cloud and processed by the cloud to predict where (flower buds) will bear fruit (specifically, a large number of data sets on the Internet can be compared, and the location of fruit growth can be compared with the location of flower buds and recorded in the log), which in turn affects the decision to prune branches.

[0081] The advantages of using binocular cameras are that the left and right cameras can simulate the stereoscopic vision of the human eye, capture the depth information of the fruit trees, construct the three-dimensional positional relationship between the branches and fruits of the fruit trees, and analyze the distribution density of the fruit trees and the load capacity of the branches. Specifically, the analysis can be done in the following ways: based on the obtained depth map, the number of fruits per unit volume is statistically predicted, and high-yield areas are marked with color gradients. The density heat map is constructed by using red to represent dense and blue to represent sparse. The indicators of high-yield and low-yield areas can be divided manually.

[0082] The working principle of a binocular camera: Two cameras capture the same scene from different angles. The distance to the target object is calculated by comparing the parallax between the left and right images. For example, the closer the branch is to the camera, the greater the parallax between the left and right images. A Mahalanobis distance algorithm is used to generate a depth map based on the parallax, marking the spatial coordinates of each predicted fruit. This part is well known to those skilled in the art and will not be elaborated here.

[0083] Furthermore, step S2 specifically includes the following steps:

[0084] S21, the image acquisition device includes a binocular camera, a depth sensor, and an inertial sensor. During the process of acquiring images of fruit tree branches, data from the depth sensor and the inertial sensor are simultaneously acquired;

[0085] S22, unifying the image data, the data collected by the depth sensor, and the data collected by the inertial sensor to the same timestamp;

[0086] S23 uses a dedicated AI chip (NPU) to accelerate inference, multi-sensor collaborative acquisition, and a low-power communication module to perform preliminary analysis and optimization of the collected tree data to reduce data transmission volume, improve real-time performance, and ensure privacy and security. It uses Gaussian blur filtering to eliminate image noise caused by uneven lighting, shaking branches and leaves, and camera noise in the original image, enhance the contrast of diseased spots on fruit tree branches, highlight target features, and correct image deformation caused by lens distortion or tilted viewing angle.

[0087] The binocular camera is used for image acquisition. The sensors include a depth sensor for spatial ranging and an inertial sensor for sensing user posture. Regarding the communication module, the low-power Bluetooth (BLE) / ZigBee communication module is used to perform preliminary analysis and optimization of tree data, reduce transmission volume, eliminate image interference, and enhance features.

[0088] Furthermore, step S22 specifically includes the following steps:

[0089] S221, the image data at time T is expressed as I(T), Indicates the timestamp collected by the binocular camera;

[0090] S222, when <T< ,but:

[0091] ;

[0092] S223 : Calculate the data collected by the depth sensor and the data collected by the inertial sensor in the above manner, so that the image data, depth data, and inertial data are aligned at the same time T.

[0093] Furthermore, step S3 specifically includes the following steps:

[0094] S31, inputting fruit tree variety genetic data, weather history records, soil moisture data, and pruning history data into a cloud database;

[0095] For historical data, you can use the LabelImg tool to annotate the fruit bounding boxes (Bounding Box) and generate a VOC / COCO format dataset (labeled fields: fruit category, maturity). The recommended sample size is ≥ 2000.

[0096] S32. Extract key features of the fruit tree in the image, including the structure of the main and side branches, the color of the lesions, and the degree of branch bending;

[0097] S33. Based on the extracted key features, the key features are compared with pruning records of fruit trees of the same variety and age in the cloud database to obtain information on the location and quantity of subsequent results corresponding to the pruning records.

[0098] On this basis, the future development trend of the branches can be predicted based on their biological characteristics, that is, growth model deduction, and then environmental factors can be coupled and the prediction parameters can be adjusted in combination with local climate and soil data.

[0099] Specifically, a 3D model of a fruit tree, reconstructed using AR glasses, is used to analyze the spatial structure of current branches. Variety genetic data, linked to the database of the Academy of Agricultural Sciences, is used to determine growth parameters. Historical meteorological records, linked to the National Meteorological Administration's API, are used to predict the impact of sunlight and precipitation on flower bud formation. Soil moisture data collected using IoT sensors is used to assess the soil's nutrient supply capacity for the tree. Agricultural operation logs from the orchard management system are used to correlate pruning intensity with yield. Key tree characteristics, such as morphological characteristics such as branch angle, diameter, bud density, and fork level; physiological characteristics such as lignification, lenticel density, and bud scale status; and environmental characteristics such as effective accumulated temperature, diurnal temperature curve, and UV intensity, are extracted through the cloud. Potential latent buds are simulated on bare branches based on the variety's budding rate and injected with historical disease data to test changes in disease resistance after pruning.

[0100] Furthermore, step S4 specifically includes the following steps:

[0101] S41. Build a branch development prediction model. This technology stack uses LSTM and an attention mechanism to input the extracted key features of fruit tree branches into the model. It also inputs the weather conditions for the next 180 days into the model. Based on this input data and combined with historical data consistent with the current situation, the model predicts and outputs information on the length, thickness, probability of flower bud differentiation, and potential disease risk of new branches.

[0102] The technology stack in this embodiment uses LSTM and an attention mechanism. Taking key features of fruit tree branches (such as length, number of forks, and growth rate, which change over time) as input, LSTM learns the long-term dependencies of these features and captures the impact of historical data (key features) on future growth. For example, by recording the growth of several fruit trees with different key features in the same environment, the impact of these key features on future growth can be summarized based on the different growth patterns.

[0103] An attention mechanism is introduced to dynamically focus on time segments or feature dimensions that are more critical to future development, such as bifurcation characteristics at a specific growth stage, and the meteorological conditions (temperature, light, humidity, etc.) for a period of time in the future are simultaneously input into the model. The model combines the developmental patterns of similar historical situations to predict the future growth status of branches (length, morphology, bifurcation trend, etc.).

[0104] S42. Build a fruit generation model. This technology stack uses a generative adversarial network. The generator generates a virtual fruit distribution based on branch spatial topology (extracting branch bifurcations, directions, and spatial positions, and constructing a topological structure with bifurcations / endpoints as nodes and branch segments as edges). The discriminator then compares this with historical data from the real orchard to verify the rationality of the virtual fruit distribution. It then outputs a probability cloud of the fruit's three-dimensional coordinates and uses this historical data to predict fruit data.

[0105] If a branch is predicted to have too many fruits, the cloud will indicate that the branch needs to be thinned or reinforced.

[0106] S43. Establish a pruning strategy optimization model based on the deep reinforcement learning framework. First, collect the morphological, physiological and environmental characteristics of fruit tree branches to build the state, define the action space such as pruning method and position, design the reward function based on yield, tree vigor, etc., and use the near-segment strategy optimization algorithm to describe the real-time characteristics of fruit trees, including the morphological, physiological and environmental characteristics of fruit tree branches, determine the feasible pruning method, and determine the pruning position, incision length, stump length and pruning priority based on the predicted situation after pruning.

[0107] Furthermore, step S5 specifically includes the following steps:

[0108] S51. Design a multi-dimensional quantitative evaluation system, including yield factors, quality factors, damage factors, and energy consumption factors related to fruit trees;

[0109] S52. The yield factor is calculated as follows: predicted number of flower buds × historical fruit setting rate × fruit tree variety characteristic coefficient;

[0110] S53. The quality factor is calculated as follows: expected fruit uniformity + percentage of pigmented area - probability of disease, where pigmentation refers to the color distribution area of ​​the fruit skin due to pigment accumulation;

[0111] S54. The damage factor is calculated as follows: total pruning wound area × incision infection risk coefficient; the incision infection risk coefficient is obtained through: historical data modeling + real-time parameter calculation, specifically: collecting the orchard's historical environment (temperature, humidity, microorganisms), fruit tree wound characteristics (area, location) and infection results, and training the prediction model; after the training is completed, the current wound parameters and environmental data of the fruit trees in actual conditions are input into the model, and the model outputs the corresponding risk coefficient.

[0112] S55. Use the mechanical work required for pruning as an energy consumption factor. This is determined through mechanical testing and path modeling: First, test the shear fracture force of different fruit tree branches (diameters and varieties). Combined with the planned cut length and tool travel, calculate the mechanical work per branch using the formula "force × displacement." Then, add up the work required for all pruned branches and incorporate a tool energy efficiency factor correction (e.g., mechanical efficiency).

[0113] S56. Design the reward function through the above factors to guide model learning.

[0114] Furthermore, step S56 specifically includes the following steps:

[0115] S561. Assign corresponding weights to each factor 、 、 、 ,in, + + + =1;

[0116] S562. The reward function R is calculated using the following formula:

[0117] ;

[0118] in, represents the yield factor, represents the quality factor, represents the damage factor, represents the energy consumption factor; 、 、 、 They respectively represent the maximum value of the corresponding factors in the historical data.

[0119] Preferably, in this embodiment, an L-system algorithm is used to simulate the regeneration process of branches after pruning, and the parameters include the apical dominance strength of the tree, the latent bud germination threshold and the nutrient distribution weight, and random simulated extreme weather and virtual pest and disease infestation events and other environmental disturbances are injected.

[0120] L-system Algorithm Principle: The L-system (Lin's system) is a parallel rewriting system used to describe plant growth. When simulating branch regeneration, a series of rules can be used to iteratively generate plant structures. It is used here to simulate branch regeneration after pruning because it can effectively depict changes in plant topology.

[0121] The cloud-based system automatically adjusts reward weights based on the orchard's geographic location, integrating regional adaptation and economic factors. For example, it increases disease penalties in rainy areas and water use efficiency rewards in arid regions. By integrating with market price fluctuation data, the system dynamically optimizes the proportion of high-quality fruit output. Climate, soil, and other conditions vary significantly across regions. Automatically adjusting reward weights based on the orchard's location ensures the model is more responsive to local conditions. In rainy areas, where excessive humidity can easily lead to disease, increased disease penalties encourage the system to optimize management strategies to reduce disease occurrence. In arid regions, where water resources are scarce, increased water use efficiency rewards incentivize water-saving irrigation and other measures to improve water use efficiency. Integration with market price fluctuation data allows orchard management strategies to adapt to market fluctuations. When the market price of a particular fruit increases, the system dynamically optimizes the proportion of high-quality fruit output by adjusting planting and management parameters (such as pruning intensity and fertilization strategies) to increase the yield of high-quality fruit, thereby improving the orchard's economic benefits.

[0122] Furthermore, step S6 specifically includes the following steps:

[0123] After processing and analyzing the data in the cloud, S61 compresses the data and transmits it back to the display device. The data transmission protocol uses the 5G NR low-latency and high-reliability mode, with end-to-end latency less than 50ms. The anti-interference mechanism uses low-density parity-check code for channel coding and automatic retransmission requests to ensure that critical instructions are 100% delivered.

[0124] S62, the display device decodes the data and renders it after decoding. The rendered data is projected into the human eye through optical waveguide technology, achieving human visual fusion. Local decoding uses the display driver chip to parse the AR data packet and call pre-stored 3D library icons such as pruning symbols and pest and disease markers. Graphics are rendered in real time based on OpenGL ES 3.2, and the viewing angle is dynamically adjusted according to the IMU data.

[0125] S63. The optical waveguide is divided into an input area, a pupil expansion area and a coupling-out area. The light is deflected through the input area to make it fully reflected in the waveguide, the effective display area is expanded through the pupil expansion area, and the light is gradually guided toward the direction of the human eye through the coupling-out area.

[0126] The coupling-in area in this embodiment uses a nano-grating with a sawtooth structure of 300nm period to deflect the light emitted by the micro-display by 70-85°, so that it is totally reflected in the waveguide. The pupil expansion area expands the effective display area by 3 times through two-dimensional beam expansion, so that the light undergoes 12-15 total reflections inside the lens. The coupling-out area gradually guides the light to the direction of the human eye, forming a 40° diagonal field of view and using a gradient period grating of 200-400nm. Finally, the human eye vision is integrated, and the left and right eyes display different images. The horizontal offset is about 6cm for binocular parallax matching to automatically synthesize a stereoscopic visual effect in the brain. The focal length virtual image is adjusted to 2m optically to better match the typical observation distance of the tree, while also avoiding visual fatigue and eliminating the need to frequently switch between near and far focus.

[0127] Furthermore, step S7 specifically includes the following steps:

[0128] S71. In channel pruning of a convolutional neural network, calculate the L1 norm of the weight matrix of each channel. In this embodiment, YOLOv8 is used for object detection and image processing of fruit tree branches and fruits. It can identify and extract features such as branch morphology (position, length, bifurcation) and fruit distribution (number, position), providing input data for branch development prediction and pruning strategy optimization.

[0129] S72, removing channels whose L1 norm is less than a set value from the model to achieve model compression;

[0130] S73. In the batch normalization layer, obtain the scaling factor gamma of each channel, sort them according to the size of the gamma value, remove the channels corresponding to the gamma value smaller than the set value, and adjust the number of input channels of the subsequent layer.

[0131] During channel pruning in convolutional neural networks, the L1 norm of each channel's weight matrix is ​​calculated. This L1 norm measures the importance of a channel to the model's output. A channel with a small L1 norm means the sum of the absolute values ​​of its weight elements is small, potentially contributing less to the model. Therefore, channels with small L1 norms can be considered unimportant and removed from the model, achieving model compression.

[0132] In batch normalization (BN) layers, the scaling factor gamma can be used to measure channel importance. Smaller gamma values ​​indicate a channel's lesser impact on subsequent layers. Therefore, channels with smaller gamma values ​​can be considered unimportant and pruned. Implementation steps: Obtain the scaling factor for the BN layer: Obtain the gamma scaling factor for each channel from the BN layer. Sort and select channels for pruning: Sort channels by gamma value and select channels with smaller gamma values ​​for pruning.

[0133] Pruning operation: remove the channels to be pruned from the weights of the convolutional layer and adjust the number of input channels of the subsequent layers accordingly.

[0134] Therefore, dynamically adjusting the L1 norm and gamma value to control the number of image pruning channels becomes a means to quickly identify and extract target features. Combining the two yields a comprehensive channel importance score (the weight can be adjusted using the hyperparameter alpha), which is achieved through the following methods:

[0135] text{score}(c) = alpha cdot text{score}_{text{L1}}(c) + (1-alpha)cdot text{score}_{gamma}(c))\.

[0136] When (alpha=1), it only depends on the L1 norm, and when (alpha=0), it only depends on (gamma). In practice, (alpha) can be optimized through experiments.

[0137] The specific core algorithm logic is as follows:

[0138] 1. Core Objectives:

[0139] Initial pruning: retain ≤10% of channels (remove >90% of channels) to force the model to be lightweight;

[0140] Feedback mechanism: If the current pruned model does not detect the target, gradually reduce the pruning ratio (i.e. retain more channels);

[0141] Termination condition: The difference in target detection results after two consecutive prunings is ≤ 1 (such as the change in the number of detection boxes).

[0142] 2. Pruning ratio adjustment strategy

[0143] Aggressive pruning phase (initial phase): keep_ratio starts at 10% and increases by 5% each time no target is detected (e.g., 10% → 15%) to quickly find the minimum number of valid channels.

[0144] Fine-tuning stage (after the target is detected): fine-tune the keep_ratio in 1% steps to prevent the number of channels from oscillating, for example:

[0145] If the number of detected targets increases suddenly, it means that the last pruning was excessive, and the retention ratio is retracted by 1%;

[0146] If the target number is stable, gradually prune slightly (reduce the retention ratio by 1% each time).

[0147] 3. Model stability assurance

[0148] Fine-tuning after pruning: Fine-tune the model in small batches (e.g., 10 epochs) after each pruning to prevent a sudden drop in performance.

[0149] Cross-layer channel matching: When pruning, ensure that the input / output channels of the convolutional layer and subsequent layers (such as shortcuts and fully connected layers) are consistent to avoid dimensionality errors.

[0150] 4. Termination condition extension

[0151] Dual conditions: The difference between two consecutive target numbers must be ≤ 1, and the change in the retention ratio must be ≤ 1% (for example, |keep_ratio_prev - keep_ratio_current| ≤ 0.01). This prevents early termination due to fluctuations in the target number.

[0152] In object detection, we prioritize the closest targets by distance and mark their scores. The scores are used to measure the confidence of the target detection results (selecting high-confidence targets and filtering out false detections). Sorting combines distance and scores to prioritize close, high-confidence targets, assisting users in making efficient decisions (for example, prioritizing close, well-defined branches during pruning). Therefore, achieving this requires combining three core steps: target positioning (3D coordinate calculation), distance sorting, and visual rendering. The following is a detailed implementation:

[0153] 1. Core Technology Process

[0154] 1. Get the 3D spatial coordinates of the target (key prerequisite) Depth camera (such as RGBD camera):

[0155] Get the 3D coordinates (x, y, z) of the target (branch or fruit) directly from the depth map and calculate the Euclidean distance from the target to the camera (user's perspective):

[0156] distance = sqrt{x^2 + y^2 + z^2}.

[0157] 2. Post-processing of target detection results (for distance sorting and screening);

[0158] Output format (taking YOLOv8 as an example, the output is [x1, y1, x2, y2, score, class]);

[0159] Sort by distance (closest targets first).

[0160] 3. Visual rendering (the purpose is to display the nearest targets one by one)

[0161] Core logic:

[0162] Only the first target after sorting (the most recent one with the highest score) is displayed each time, or they are displayed one by one in sequence (a display strategy needs to be defined, such as "only show the most recent one" or "show the first N").

[0163] Use OpenCV to draw marking boxes and information to intuitively mark the target location and simultaneously display data such as confidence and distance to help users quickly identify the trimmed object.

[0164] A deep learning-based pruning analysis system, based on the above-mentioned deep learning-based pruning analysis method, includes an image acquisition device and a display device connected to the image acquisition device. The display device uses AR glasses. The image display device includes a binocular camera, a depth sensor and an inertial sensor connected to the AR glasses. The AR glasses are connected to a cloud server via a network.

[0165] The functions of each component are as follows:

[0166] Binocular camera: identifies the shape and position of branches and fruits;

[0167] Depth sensor: senses the spatial distance and outline of branches and fruits;

[0168] Inertial sensors: built into AR glasses to monitor the device's posture and movement;

[0169] AR glasses are responsible for transmitting data to the cloud server through the communication module and displaying the results fed back by the cloud server, which can enhance the display effect;

[0170] Cloud servers carry model training, historical data storage and call;

[0171] Preferably, this embodiment also provides smart pruning gloves: the smart pruning gloves have built-in mechanical sensors (for measuring pruning cutting force) and displacement sensors (for measuring tool travel). Users can wear smart pruning gloves during the pruning process, which helps to obtain the above data, thereby being used for subsequent calculations of mechanical work and other data.

[0172] The connection method of each component is as follows:

[0173] Binocular camera, depth sensor → directly connected to AR glasses via USB interface;

[0174] Smart trimming gloves → Wireless communication with AR glasses via a low-latency Bluetooth module (such as BLE 5.0) (reducing cable interference);

[0175] AR glasses ↔ cloud server → transmit model parameters and perception data through 5G / WiFi 6 network.

[0176] The above embodiments are only preferred embodiments of the present invention. Those skilled in the art can derive other embodiments from the above embodiments without creative work. Therefore, this application protects not only the above embodiments, but also the scope consistent with the principles and features of this application.

Claims

1. A pruning analysis method based on deep learning, characterized in that: The following steps are involved: S1. Capture images of fruit tree branches using an image acquisition device, upload the image data to the cloud, and construct the three-dimensional positions of the fruit tree branches in the cloud; S2, compensate for the image deviation caused during the shooting process and correct the image blur; S3. Extract key features from the obtained fruit tree branch image, compare the key features with historical data, and match them with historical data with the same situation; S4. Predict the branch development after different pruning methods, predict the subsequent fruit production, and determine the pruning strategy; S5. Determine the influencing factors related to the fruit tree based on the pruning method and the predicted fruit production. Design a reward function based on the influencing factors to guide model learning. S6. The cloud transmits information related to the pruning strategy back to the display device, and the user performs pruning work based on the information obtained; S7. Optimize the model structure to increase the model inference speed, reduce its memory usage, and improve the model's generalization ability; The step S7 specifically includes the following steps: S71. In channel pruning of a convolutional neural network, calculate the L1 norm of the weight matrix of each channel; S72, removing channels whose L1 norm is less than a set value from the model to achieve model compression; S73. In the batch normalization layer, obtain the scaling factor gamma of each channel, sort them according to the size of the gamma value, remove the channels corresponding to the gamma value smaller than the set value, and adjust the number of input channels of the subsequent layer.

2. The deep learning-based pruning analysis method according to claim 1, characterized in that: The step S1 specifically includes the following steps: S11. The user collects images of fruit tree branches using a binocular camera to obtain image 1 and image 2, and uploads image 1 and image 2 to the cloud; S12, aligning the epipolar lines of the first image and the second image so that corresponding points in the images are located on the same horizontal scan line, to simplify the calculation of the disparity; S13, calculate the intrinsic and extrinsic parameters of the camera, and calculate the horizontal displacement of corresponding points in image 1 and image 2, i.e., parallax. Based on this parallax, the distance between branches is calculated by Euclidean distance, and a depth map is generated based on the distance between branches to mark the spatial coordinates of each fruit in the image; S14, converting the pixels in the image into a three-dimensional point cloud, restoring the three-dimensional structure of the fruit tree, and distinguishing overlapping branches and fruits in the image; S15. Identify the physiological state of fruit tree flower buds through near-infrared spectroscopy for subsequent prediction of fruit growth location.

3. The deep learning-based pruning analysis method according to claim 1, characterized in that: The step S2 specifically includes the following steps: S21, the image acquisition device includes a binocular camera, a depth sensor and an inertial sensor, and data from the depth sensor and the inertial sensor are simultaneously acquired during the process of acquiring images of fruit tree branches; S22, unifying the image data, the data collected by the depth sensor, and the data collected by the inertial sensor to the same timestamp; S23. The original image is subjected to interference caused by uneven lighting, shaking branches and leaves, and camera noise through Gaussian blur filtering to eliminate image noise, enhance the contrast of the diseased spots on the branches of the fruit trees, highlight the target features, and correct image deformation caused by lens distortion or tilted viewing angle.

4. The deep learning-based pruning analysis method according to claim 3, characterized in that: The step S22 specifically includes the following steps: S221, the image data at time T is expressed as I(T), Indicates the timestamp collected by the binocular camera; S222, when <T< ,but: ； S223 : Calculate the data collected by the depth sensor and the data collected by the inertial sensor in the above manner, so that the image data, depth data, and inertial data are aligned at the same time T.

5. The deep learning-based pruning analysis method according to claim 1, characterized in that: The step S3 specifically includes the following steps: S31, inputting fruit tree variety genetic data, weather history records, soil moisture data, and pruning history data into a cloud database; S32. Extract key features of the fruit tree in the image, including the structure of the main and side branches, the color of the lesions, and the degree of branch bending; S33. Based on the extracted key features, the key features are compared with pruning records of fruit trees of the same variety and age in the cloud database to obtain information on the location and quantity of subsequent results corresponding to the pruning records.

6. The deep learning-based pruning analysis method according to claim 5, characterized in that: The step S4 specifically includes the following steps: S41. Establish a branch development prediction model, input the extracted key features of the fruit tree branches into the model, and input the meteorological conditions in the future into the model. Based on the input data and combined with historical data consistent with the current situation, the model predicts and outputs information on the length, thickness, flower bud differentiation probability, and potential disease risk value of new branches; S42. Build a fruit generation model. This technology stack uses a generative adversarial network. The generator generates a virtual fruit distribution based on the spatial topology of the branches. The discriminator compares this with historical data from the real orchard to verify the rationality of the virtual fruit distribution. It then outputs a probability cloud of the fruit's three-dimensional coordinates and predicts fruit data based on historical data. S43. Establish a pruning strategy optimization model based on the deep reinforcement learning framework and adopt a near-segment strategy optimization algorithm to describe the real-time characteristics of fruit trees, including the morphological characteristics, physiological characteristics, and environmental characteristics of fruit tree branches, determine feasible pruning methods, and determine the pruning position, incision length, stump length, and pruning priority based on the predicted situation after pruning.

7. The deep learning-based pruning analysis method according to claim 1, characterized in that: The step S5 specifically includes the following steps: S51. Design a multi-dimensional quantitative evaluation system, including yield factors, quality factors, damage factors, and energy consumption factors related to fruit trees; S52. The yield factor is calculated as follows: predicted number of flower buds × historical fruit setting rate × fruit tree variety characteristic coefficient; S53, the quality factor is calculated as follows: expected fruit uniformity + percentage of colored area - disease probability; S54, the damage factor is calculated as follows: total area of ​​the trimmed wound × risk factor of incision infection; S55. Taking the mechanical work required for pruning as the energy consumption factor; S56. Design the reward function through the above factors to guide model learning.

8. The deep learning-based pruning analysis method according to claim 7, characterized in that: The step S56 specifically includes the following steps: S561. Assign corresponding weights to each factor 、 、 、 ,in, + + + =1; S562. The reward function R is calculated using the following formula: ； in, represents the yield factor, represents the quality factor, represents the damage factor, represents the energy consumption factor; 、 、 、 They respectively represent the maximum value of the corresponding factors in the historical data.

9. The deep learning-based pruning analysis method according to claim 7, characterized in that: The step S6 specifically includes the following steps: S61. After the cloud completes data processing and analysis, it compresses the data and transmits it back to the display device. S62. The display device decodes the data and renders it after decoding. The rendered data is projected into the human eye through optical waveguide technology to achieve visual fusion of the human eye. S63. The optical waveguide is divided into an input area, a pupil expansion area and an output area. The light is deflected through the input area to make it fully reflected in the waveguide, the effective display area is expanded through the pupil expansion area, and the light is gradually guided toward the direction of the human eye through the output area.

10. A deep learning-based pruning analysis system, based on the deep learning-based pruning analysis method according to claim 1, characterized in that: It includes an image acquisition device and a display device connected to the image acquisition device. The display device adopts AR glasses. The image display device includes a binocular camera, a depth sensor and an inertial sensor connected to the AR glasses. The AR glasses are connected to the cloud server through a network.

Images