Deep learning-based crop quality comprehensive evaluation method and system

Abstract

The invention relates to the technical field of crop quality comprehensive evaluation, and discloses a crop quality comprehensive evaluation method and system based on deep learning, and the method comprises the following steps: collecting and preprocessing multi-modal data, deploying an unmanned plane carrying a multispectral camera and a ground high-definition camera, the method comprises the following steps: synchronously collecting multi-dimensional data such as a canopy structure, leaf texture and fruit morphology, constructing a three-flow deep neural network architecture, detecting crops in a specified area from a plant integrity detection flow, a pest and disease damage detection flow and a growth robustness detection flow, and constructing a comprehensive evaluation model. And mapping a detection result in the three-flow deep neural network into the comprehensive evaluation model to generate a comprehensive evaluation report. According to the crop quality comprehensive evaluation method and system based on deep learning, crops in a designated area are detected from a plant integrity detection flow and a pest and disease damage detection flow by constructing a three-flow deep neural network architecture, the detection has real-time performance and accuracy, and therefore it is guaranteed that the evaluation result is more accurate.

Description

TECHNICAL FIELD

[0001] The application relates to a crop quality comprehensive evaluation method and system based on deep learning. BACKGROUND

[0002] The crop quality comprehensive evaluation is a method system for quantitatively analyzing multi-dimensional indexes, and systematically evaluating the economic value, nutritional characteristics and agronomic adaptability of crops. The core goal is to realize the market mechanism of high quality and high price, and guide the improvement of varieties and the optimization of planting techniques.

[0003] The traditional crop quality evaluation relies on manual visual inspection and laboratory analysis, and has the problems of low efficiency, strong subjectivity and inability to realize real-time monitoring. In the prior art, the disease and pest detection method based on a single image sensor ignores the influence of other indexes on crop evaluation, so that the evaluation result is single and has no guiding significance. In addition, as a representative of deep learning, the CNN model is widely used, but the detection accuracy of early diseases and small target disease spots is insufficient, and the multi-task collaborative optimization capability is limited. Therefore, the application provides a crop quality comprehensive evaluation method and system based on deep learning to solve the above problems. SUMMARY

[0004] (I) Technical problems to be solved

[0005] In view of the defects of the prior art, the application provides a crop quality comprehensive evaluation method and system based on deep learning, which has the advantages of higher detection result accuracy and more accurate evaluation result. The traditional crop quality evaluation relies on manual visual inspection and laboratory analysis, and has the problems of low efficiency, strong subjectivity and inability to realize real-time monitoring. In the prior art, the disease and pest detection method based on a single image sensor ignores the influence of other indexes on crop evaluation, so that the evaluation result is single and has no guiding significance. In addition, as a representative of deep learning, the CNN model is widely used, but the detection accuracy of early diseases and small target disease spots is insufficient, and the multi-task collaborative optimization capability is limited.

[0006] (II) Technical solutions

[0007] In order to realize the above-mentioned higher detection result accuracy and more accurate evaluation result, the application provides the following technical scheme: a crop quality comprehensive evaluation method based on deep learning, comprising the following steps:

[0008] Step 1: Multimodal data acquisition and preprocessing: Deploy drones equipped with multispectral cameras (400-1000nm) and ground-based high-definition cameras (≥20 megapixels) to simultaneously collect multi-dimensional data such as canopy structure, leaf texture, and fruit morphology. Time series photography is used to record the entire process of disease incubation and outbreak. Combined with an expert annotation system, a labeled dataset containing common diseases and pests is established.

[0009] Step 2: Build a three-stream deep neural network architecture to detect crops in a specified area from the plant integrity detection stream, pest and disease detection stream, and growth and health detection stream;

[0010] Step 3: Construct a comprehensive evaluation model and determine the evaluation indicators;

[0011] Step 4: Map the detection results in the three-stream deep neural network to the comprehensive evaluation model to generate a comprehensive evaluation report.

[0012] Preferably, the three-stream deep neural network architecture in step 2 specifically includes the following steps:

[0013] Step 201: Acquire plant point cloud data using an RGB-D camera or a LiDAR, perform multi-view point cloud registration using an ICP algorithm, and reconstruct a three-dimensional plant model;

[0014] Step 202: Segment the leaf / stem region based on the improved U-Net++ network and calculate geometric parameters such as leaf defect rate and stem curvature;

[0015] Step 203: Use YOLOX as the backbone network and integrate the cross-stage feature pyramid (FPN-PAFPN) to enhance the small object recognition capability;

[0016] Step 204: Strengthen the feature response of the lesion area through the attention mechanism (CBAM module);

[0017] Step 205: construct a bidirectional LSTM network to process multi-temporal remote sensing data and capture the dynamic change trend of plant height and leaf area index;

[0018] Step 206: extract near-infrared spectral features through ResNet50, invert chlorophyll content (SPAD value) and nitrogen accumulation, and perform potential energy evaluation;

[0019] Step 207: Design a feature concatenation layer at the end of the three-stream network to achieve cross-modal feature fusion through 1×1 convolution.

[0020] Preferably, in step 3, the construction of the comprehensive evaluation model includes the following steps:

[0021] Step 301: multimodal feature normalization and alignment, normalizing the three-stream detection results separately;

[0022] Step 302: Multi-stream feature space calibration is achieved using improved STN (Spatial Transformer Networks);

[0023] Step 303: A multi-head attention mechanism is constructed to generate a cross-stream feature weight matrix;

[0024] Step 304: A multi-level index system is constructed, including health index, pest and disease growth index, and growth index;

[0025] Step 305: Refer to the reference agricultural expert knowledge base to set the threshold

[0026] Step 306: Generate a visual report and output the evaluation results.

[0027] Preferably, the step 202 of improving the U-Net++ network to segment the leaf / stem area specifically includes the following steps:

[0028] Step 2021: Use an RGB-D camera to obtain three-dimensional point cloud data of the plant, and combine a multispectral sensor to capture vegetation indices (such as NDVI, PSRI) for distinguishing leaf and stem areas;

[0029] Step 2022: Enhance the data by flipping and shaking, and label the data;

[0030] Step 2023: The backbone network uses a ResNet50 pre-trained model to extract multi-scale features, embeds a channel attention module (CBAM) to enhance stem edge response, and introduces a multi-scale input module after the fourth layer downsampling to fuse different resolution feature maps (512x512 to 64x64);

[0031] Step 2024: Use Dense Skip Connection to fuse shallow details and deep semantic information, reduce feature loss, and add a residual deformable convolution module (RDCM) before each upsampling to improve the adaptability to curved stems;

[0032] Preferably, the combined weighted cross-entropy loss (WCE) and Dice loss of the improved U-Net++ network balances the small target detection requirements of the leaf loss area, and the formula is:

[0033] Loss = 0.7·Dice + 0.3·WCE

[0034] Wherein, the Dice coefficient = leaf / stem area

[0035] Preferably, the evaluation index in step 3 includes the completeness of the plant, the degree of crop pests and diseases, and the length of growth.

[0036] Preferably, the crop quality comprehensive evaluation method further comprises triggering a safety warning when the detection scores of 10 consecutive crops in the region do not meet the standard.

[0037] Preferably, the crop quality comprehensive evaluation system comprises a main control module, an image acquisition module, a data storage module, a three-flow deep neural network module, a comprehensive evaluation model module, a result generation module, and a safety warning module.

[0038] Preferably, the image acquisition module acquires images of the specified area through deployment of unmanned aerial vehicles and ground shooting devices.

[0039] The data storage module stores the acquired images and related crop historical data to generate a data set.

[0040] The three-flow deep neural network module has three different branches to detect the shape, disease and pest, and growth length of crops, and provides data support for the comprehensive evaluation module.

[0041] The comprehensive evaluation model module performs comprehensive evaluation of crops based on different indicators through imported data and outputs evaluation results to generate a report.

[0042] (Three) beneficial effects

[0043] Compared with the prior art, the present application provides a crop quality comprehensive evaluation method and system based on deep learning, which has the following beneficial effects:

[0044] 1. The crop quality comprehensive evaluation method and system based on deep learning detects the crops in the specified area from the plant integrity detection flow, disease and pest detection flow, and growth and strength detection flow through the construction of a three-flow deep neural network architecture, which is real-time and accurate, thereby ensuring more accurate evaluation results.

[0045] 2. The crop quality comprehensive evaluation method and system based on deep learning maps the three-flow deep neural network architecture to the crop quality comprehensive evaluation, thereby realizing more comprehensive crop evaluation and more representative results. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 The crop quality comprehensive evaluation method flowchart of the present application;

[0047] Figure 2 The three-flow deep neural network architecture diagram of the present application;

[0048] Figure 3 The crop quality comprehensive evaluation system diagram of the present application. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0050] See also Figure 1-3 A method for comprehensive evaluation of crop quality based on deep learning, characterized by comprising the following steps:

[0051] Step 1: Multimodal data acquisition and preprocessing: Deploy drones equipped with multispectral cameras (400-1000nm) and ground-based high-definition cameras (≥20 megapixels) to simultaneously collect multi-dimensional data such as canopy structure, leaf texture, and fruit morphology. Time series photography is used to record the entire process of disease incubation and outbreak. Combined with an expert annotation system, a labeled dataset containing common diseases and pests is established.

[0052] Step 2: Build a three-stream deep neural network architecture to detect crops in a specified area from the plant integrity detection stream, pest and disease detection stream, and growth and health detection stream;

[0053] Step 3: Construct a comprehensive evaluation model and determine the evaluation indicators;

[0054] Step 4: Map the detection results in the three-stream deep neural network to the comprehensive evaluation model to generate a comprehensive evaluation report.

[0055] In step 1, the three-stream deep neural network architecture in step 2 specifically includes the following steps:

[0056] Step 201: Acquire plant point cloud data using an RGB-D camera or a LiDAR, perform multi-view point cloud registration using an ICP algorithm, and reconstruct a three-dimensional plant model;

[0057] Step 202: Segment the leaf / stem region based on the improved U-Net++ network and calculate geometric parameters such as leaf defect rate and stem curvature;

[0058] Step 203: Use YOLOX as the backbone network and integrate the cross-stage feature pyramid (FPN-PAFPN) to enhance the small object recognition capability;

[0059] Step 204: Strengthen the feature response of the lesion area through the attention mechanism (CBAM module);

[0060] Step 205: construct a bidirectional LSTM network to process multi-temporal remote sensing data and capture the dynamic change trend of plant height and leaf area index;

[0061] Step 206: Extract near-infrared spectral features through ResNet50, inverse the chlorophyll content (SPAD value) and nitrogen accumulation, and perform potential energy assessment;

[0062] Step 207: Design feature concatenation layer at the end of the three-stream network, realize cross-modal feature fusion through 1x1 convolution.

[0063] In this embodiment, through the three-stream social network, the core role, the multi-modal feature is complementary

[0064] Plant integrity detection stream: Quantify the integrity of plant morphology and structure (such as leaf damage rate, stem bending degree), reflect the mechanical damage and growth state of crops;

[0065] Disease and pest detection stream: Identify pathogen infection area (such as rust patch, aphid aggregation area), provide disease threat level assessment;

[0066] Growth robustness detection stream: Analyze chlorophyll content (NDVI index), stem diameter and other parameters, and predict yield potential;

[0067] Three-stream cooperation can cover the full-dimensional features of crop phenotype-pathology-physiology.

[0068] It also includes dynamic weight optimization, which automatically adjusts the contribution weight of each stream according to environmental parameters (temperature / humidity) through the gating attention mechanism:

[0069] Break through the limitation of single modal detection, traditional methods only rely on visible light image (such as RGB camera) are easily disturbed by light, three-stream fusion combined with near-infrared (NIR) and multispectral data can improve the feature robustness46

[0070] Solve the problem that a single sensor cannot capture morphological changes (such as leaf curling) and physiological indicators (such as photosynthetic rate) at the same time.

[0071] In Figure 2 In step 3, the construction of the comprehensive evaluation model includes the following steps:

[0072] Step 301: Multi-modal feature standardization and alignment, normalize the three-stream detection results respectively;

[0073] Step 302: Use improved STN (Spatial Transformer Networks) to realize multi-stream feature space calibration;

[0074] Step 303: Construct multi-head attention mechanism to generate cross-stream feature weight matrix;

[0075] Step 304: Multi-level index system construction, including health index, disease and pest growth index, and growth index;

[0076] Step 305: Refer to the agricultural expert knowledge base and set the threshold

[0077] Step 306: Generate a visualization report and output the evaluation results.

[0078] In this embodiment, cross-modal feature fusion is used to integrate three streams of data: plant integrity (morphology), probability of pests and diseases (pathology), and growth strength (physiology) to construct a full-dimensional evaluation system covering crop phenotype, pathology, and physiology.

[0079] defevaluate_status(H,R_disease):

[0080] ifH>0.8andR_disease<0.3:

[0081] return "health status"

[0082] elifH>0.5orR_disease<0.6:

[0083] return"Sub-health status"

[0084] else:

[0085] return "Intervention Required" #Define threshold based on expert knowledge base: ml-citation{ref="1,4"data="citationList"}

[0086] exist Figure 2 In the step 202, the improved U-Net++ network segmentation of the leaf / stem region specifically includes the following steps:

[0087] Step 221: Use an RGB-D camera to acquire three-dimensional point cloud data of the plant, and combine it with a multispectral sensor to capture vegetation indices (such as NDVI and PSRI) to distinguish between leaf and stem areas;

[0088] Step 2022: Enhance the data by flipping and dithering, and label the data;

[0089] Step 223: The backbone network uses the ResNet50 pre-trained model to extract multi-scale features, embeds a channel attention module (CBAM) to enhance stem edge response, introduces a multi-scale input module after downsampling in the fourth layer, and fuses feature maps of different resolutions (512×512 to 64×64);

[0090] Step 2024: Dense Skip Connection is used to fuse shallow details with deep semantic information to reduce feature loss. A residual deformable convolution module (RDCM) is added before each upsampling to improve adaptability to bent stems.

[0091] exist Figure 2 In the above, the combined weighted cross entropy loss (WCE) and Dice loss of the improved U-Net++ network balances the small target detection requirements in the leaf defect area as follows:

[0092] Loss = 0.7 Dice + 0.3 WCE

[0093] Where Dice coefficient = leaf / stem area

[0094] The evaluation indicators in step 3 include plant integrity, crop pest and disease severity, and growth.

[0095] The comprehensive crop quality evaluation method also includes triggering a safety warning when the inspection scores of 10 consecutive crops in the area do not meet the standards.

[0096] exist Figure 3 The crop quality comprehensive evaluation system includes a main control module, an image acquisition module, a data storage module, a three-stream deep neural network module, a comprehensive evaluation model module, a result generation module, and a safety warning module.

[0097] The image acquisition module collects images of a designated area by deploying ground shooting equipment on drones;

[0098] The data storage module stores collected images and related crop historical data for generating a data set. The three-stream deep neural network module, with three different branches, detects crop appearance, pests and diseases, and growth, providing data support for the comprehensive evaluation module.

[0099] The comprehensive evaluation model module comprehensively evaluates crops based on different indicators using the imported data, outputs the evaluation results, and generates a report.

[0100] In summary, this deep learning-based comprehensive crop quality evaluation method and system, by constructing a three-stream deep neural network architecture, detects crops in a specified area from the plant integrity detection stream, pest and disease detection stream, and growth strength detection stream. The detection is real-time and accurate, thus ensuring more accurate evaluation results.

[0101] Moreover, by mapping the three-stream deep neural network architecture to the comprehensive evaluation of crop quality, a more comprehensive crop evaluation can be achieved and the results are more representative.

[0102] It has to be noted that, in the present document, the terms "first", "second", etc. merely serve to identify a subject or action, without necessarily requiring or implying any such actual relationship or order between such subjects or actions. Moreover, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0103] While embodiments of the application have been shown and described, it is to be understood that the application is not limited to the details of the embodiments described, since numerous further modifications and changes can be apparent to one skilled in the art without departing from the spirit and scope of the application, which is defined by the appended claims and their equivalents.

Claims

1. A comprehensive crop quality evaluation method based on deep learning, characterized by: The steps include: Step 1: Multimodal data acquisition and preprocessing: Deploy drones equipped with multispectral cameras (400-1000nm) and ground-based high-definition cameras (≥20 megapixels) to simultaneously collect multi-dimensional data such as canopy structure, leaf texture, and fruit morphology. Time series photography is used to record the entire process of disease incubation and outbreak. Combined with an expert annotation system, a labeled dataset containing common diseases and pests is established. Step 2: Build a three-stream deep neural network architecture to detect crops in a specified area from the plant integrity detection stream, pest and disease detection stream, and growth and health detection stream; Step 3: Construct a comprehensive evaluation model and determine the evaluation indicators; Step 4: Map the detection results in the three-stream deep neural network to the comprehensive evaluation model to generate a comprehensive evaluation report.

2. The method for comprehensive crop quality evaluation based on deep learning according to claim 1, characterized in that: The three-stream deep neural network architecture in step 2 specifically includes the following steps: Step 201: Acquire plant point cloud data using an RGB-D camera or a LiDAR, perform multi-view point cloud registration using an ICP algorithm, and reconstruct a three-dimensional plant model; Step 202: Segment the leaf / stem region based on the improved U-Net++ network and calculate geometric parameters such as leaf defect rate and stem curvature; Step 203: Use YOLOX as the backbone network and integrate the cross-stage feature pyramid (FPN-PAFPN) to enhance the small object recognition capability; Step 204: Strengthen the feature response of the lesion area through the attention mechanism (CBAM module); Step 205: construct a bidirectional LSTM network to process multi-temporal remote sensing data and capture the dynamic change trend of plant height and leaf area index; Step 206: extract near-infrared spectral features through ResNet50, invert chlorophyll content (SPAD value) and nitrogen accumulation, and perform potential energy evaluation; Step 207: Design a feature concatenation layer at the end of the three-stream network to achieve cross-modal feature fusion through 1×1 convolution.

3. The method for comprehensive crop quality evaluation based on deep learning according to claim 1, characterized in that: In step 3, the construction of the comprehensive evaluation model includes the following steps: Step 301: multimodal feature normalization and alignment, normalizing the three-stream detection results separately; Step 302: using an improved STN (Spatial Transformer Networks) to implement multi-stream feature space calibration; Step 303: Build a multi-head attention mechanism to generate a cross-stream feature weight matrix; Step 304: constructing a multi-level indicator system, including health index, pest growth index, and growth index; Step 305: Refer to the agricultural expert knowledge base and set the threshold Step 306: Generate a visualization report and output the evaluation results.

4. The method for comprehensive crop quality evaluation based on deep learning according to claim 1, characterized in that: The step 202 of segmenting the leaf / stem region using the improved U-Net++ network specifically includes the following steps: Step 221: Use an RGB-D camera to acquire three-dimensional point cloud data of the plant, and combine it with a multispectral sensor to capture vegetation indices (such as NDVI and PSRI) to distinguish between leaf and stem areas; Step 2022: Enhance the data by flipping and dithering, and label the data; Step 223: The backbone network uses the ResNet50 pre-trained model to extract multi-scale features, embeds a channel attention module (CBAM) to enhance stem edge response, introduces a multi-scale input module after downsampling in the fourth layer, and fuses feature maps of different resolutions (512×512 to 64×64); Step 2024: Dense Skip Connection is used to fuse shallow details with deep semantic information to reduce feature loss. A residual deformable convolution module (RDCM) is added before each upsampling to improve adaptability to bent stems.

5. The method for comprehensive crop quality evaluation based on deep learning according to claim 1, characterized in that: The combination of weighted cross entropy loss (WCE) and Dice loss of the improved U-Net++ network balances the small target detection requirements in the leaf defect area. The formula is: Loss = 0.7 Dice + 0.3 WCE Where Dice coefficient = leaf / stem area.

6. The method for comprehensive crop quality evaluation based on deep learning according to claim 1, characterized in that: The evaluation indicators in step 3 include plant integrity, crop pest and disease severity, and growth.

7. The method for comprehensive crop quality evaluation based on deep learning according to claim 1, characterized in that: The comprehensive crop quality evaluation method also includes triggering a safety warning when the inspection scores of 10 consecutive crops in the area do not meet the standards.

8. A deep learning-based comprehensive crop quality evaluation system comprising the comprehensive crop quality evaluation method and evaluation system according to any one of claims 1 to 7, characterized in that: The crop quality comprehensive evaluation system includes a main control module, an image acquisition module, a data storage module, a three-stream deep neural network module, a comprehensive evaluation model module, a result generation module, and a safety warning module.

9. The deep learning-based comprehensive crop quality evaluation system according to claim 8, characterized in that: The image acquisition module collects images of a designated area by deploying ground shooting equipment on drones; The data storage module stores the collected images and related crop history data for generating a data set; The three-stream deep neural network module, with three different branches, detects crop appearance, pests and diseases, and growth, providing data support for the comprehensive evaluation module. The comprehensive evaluation model module comprehensively evaluates crops based on different indicators using the imported data, outputs the evaluation results, and generates a report.

Images