Method for estimating plant biomass based on map multi-modal feature extraction and fusion

By employing an automatic extraction and fusion method of multimodal features from plant maps, combined with Grounding DINO and SAM models, the high cost and low accuracy of plant biomass estimation in existing technologies are resolved. This enables rapid, non-destructive, and accurate biomass monitoring, making it suitable for high-throughput plant phenotypic analysis.

CN121074680APending Publication Date: 2025-12-05NANJING FORESTRY UNIV
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202511308512.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-09-15
Publication Date
2025-12-05

AI Technical Summary

Technical Problem

Existing technologies for plant biomass estimation suffer from problems such as high destructiveness, time and labor consumption, difficulty in dynamic monitoring, insufficient model generalization ability, high cost of high-resolution images, and limited coverage, making it difficult to achieve high-precision and non-destructive plant biomass monitoring.

Method used

We adopted an automatic extraction and fusion method based on spectral multimodal features, and used a dual-model collaborative segmentation framework of Grounding DINO and Segment Anything Model (SAM) to combine RGB and multispectral images. We then used an improved SHAP model to perform feature dimensionality reduction and biomass prediction, and established a high-precision biomass prediction model.

Benefits of technology

It enables rapid, non-destructive, and accurate acquisition of plant biomass phenotypes, reduces manual annotation costs and computation time, and improves the universality and accuracy of the model, making it suitable for high-throughput, non-destructive, and precise plant phenotypic analysis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121074680A_ABST
    Figure CN121074680A_ABST
Patent Text Reader

Abstract

The invention discloses a method for estimating plant biomass based on map multi-modal feature extraction and fusion. The method comprises the following steps: acquiring a plant RGB image and a multi-spectral image; inputting the preprocessed RGB image and NIR wave band image into a double-model cooperation segmentation framework to realize image segmentation; binary image features, color features, texture features, reflectivity and the like are calculated, feature splicing is carried out, and high-dimensional features are constructed; carrying out dimension reduction on the high-dimensional features; and training a deep neural network through the effective features and the biomass to realize biomass estimation. According to the method, a zero sample learning-based double-model cooperation segmentation framework is utilized to realize accurate segmentation of a single plant on the premise that a large number of training sets are not needed; multi-modal feature information is extracted based on the segmented single plant image, an improved SHAP model is introduced to reduce the feature space dimension, and the inversion precision and the operation efficiency are improved while the information effectiveness is ensured; through a high-precision deep neural network model, rapid, lossless and accurate biomass acquisition is realized.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of plant phenotype parameter acquisition, and particularly relates to a method for estimating plant biomass based on atlas multi-modal feature automatic extraction and fusion. BACKGROUND

[0002] Biomass is the product of photosynthesis in the process of plant growth and development, which can be described as the actual survival of organic matter (dry weight) and the dry weight of living organisms per unit area at a certain moment. Plant biomass is one of the most representative parameters in yield functional phenotype characteristics, and as a core index for evaluating plant growth status, yield potential and carbon storage of ecological system, it has important significance for precision agriculture, ecological monitoring and carbon neutralization research. Traditional biomass estimation methods include sampling method and model prediction method, which have obvious limitations. Among them, the sampling method is destructive, requires a large number of repeated experiments, is time-consuming and labor-intensive, and is difficult to monitor dynamically. The model prediction method predicts the yield formation of plants by observing the field data of plants and comprehensively considering the influence of weather, environment, water and fertilizer on the dynamic growth of plants, which is limited by the difficulty of parameter acquisition and environmental heterogeneity, and has insufficient generalization ability. Although the indirect estimation method based on easily measured phenotype parameters (such as plant height, crown width, and ground stem) alleviates this problem to some extent, due to the great differences in plant varieties and habitats, the universality and precision of traditional models still cannot meet the needs of modern agriculture and forestry at large scale and high dynamics.

[0003] In recent years, the rapid development of machine vision and spectral technology has provided a new idea for non-destructive monitoring of plant biomass, but it is still limited by data resolution, time resolution, etc. High-resolution image data is costly and has limited coverage, and cannot guarantee continuous observation of plants within a short interval of time, making it difficult to capture rapid changes in biomass. Using optical sensors to collect image data to evaluate plant phenotypes can quickly obtain spatial variation information of plants, and through frequent or regular monitoring, the dynamic changes of plants can be effectively captured, which has gradually become a new means of modern precision agriculture. Different imaging sensors can cover and obtain plant data, and can obtain image information of multiple scales (such as RGB and spectrum) in real time, which can overcome the data saturation problem of single sensor, and visible light sensors have the advantages of high resolution and intuitive data. Multi-spectral different channels have large differences in ground objects, and can realize non-destructive acquisition of phenotypes. Both kinds of imaging sensors are widely used in the field of plant phenotypes. Based on computer science technology, the relationship between spectral information and phenotypes is associated, and an empirical method is used to establish an estimation model of phenotypes, which can realize the visualization of plant phenotype distribution on a large scale, and provides a fast means for promoting non-destructive analysis of plant phenotypes.

[0004] The acquisition of plant phenotype information shows great potential in the context of the rapid development of high-throughput imaging technology. However, there are two key challenges in accurately extracting plant phenotypes through imaging technology: first, due to the limitations of high-resolution image acquisition, the high cost and limited spatial coverage make it difficult to accumulate multi-temporal and continuous observation data of plant individuals throughout their growth cycle, resulting in insufficient samples for model training, which affects the accuracy and stability of plant canopy segmentation; second, models have strong dependence on features and data, and traditional machine learning regression methods often struggle to achieve ideal inversion accuracy under high-dimensional feature conditions; although the fusion of multispectral and visible light sensors to some extent alleviates the saturation problem caused by a single data source, how to efficiently integrate multi-modal information and build a high-precision prediction model remains a key problem to be solved.

[0005] Currently, the emergence of small sample, unsupervised and lightweight image segmentation models in the field of computer vision provides new opportunities to solve the above problems. The emergence of two new models, Grounding DINO and Segment Anything Model (SAM), significantly reduces the dependence of image segmentation on labeled data. Among them, Grounding DINO is a zero-shot object detector that can automatically locate target objects in images based on text prompts, but relying solely on detection models cannot meet the demand for high-precision mask extraction; SAM, as a general zero-shot segmentation model, supports multiple interaction modes such as language prompts, positive and negative point prompts, and bounding box prompts, and has the ability to achieve high-quality image segmentation without training samples. However, the SAM model still has significant limitations in practical applications - it does not have the ability to identify the class of segmented objects and can only perform prompt-based mask extraction, making it difficult to achieve semantic segmentation in complex scenarios. Therefore, how to organically combine the advantages of the two and build an automated and low-cost plant phenotype analysis framework has become a research hotspot in this field. SUMMARY

[0006] The technical problem to be solved by the present application is to provide a method for estimating plant biomass based on automatic extraction and fusion of atlas multi-modal features to address the shortcomings of the prior art. The method for estimating plant biomass based on automatic extraction and fusion of atlas multi-modal features uses a dual-model collaborative segmentation framework based on zero-shot learning to achieve accurate segmentation of individual plants without the need for a large training set. At the same time, multi-modal information such as plant color, texture, binary, spectral reflectance, etc. is extracted from the segmented individual plant images, and an improved SHAP model is introduced to reduce the dimensionality of the feature space, thereby improving the inversion accuracy and computational efficiency while ensuring information effectiveness. By selecting appropriate variables based on variable contribution values, a high-precision biomass prediction model is established, achieving fast, non-destructive and accurate plant biomass phenotype acquisition.

[0007] To achieve the above technical purposes, the technical scheme adopted by the present application is:

[0008] The method for estimating plant biomass based on automatic extraction and fusion of atlas multi-modal features comprises the following steps:

[0009] (1) Collecting plant RGB images and multi-spectral images in the study area, and destructively sampling and measuring biomass data of plants in the study area;

[0010] (2) applying an improved ExG-P algorithm to the RGB image for pretreatment;

[0011] (3) inputting the pretreated RGB image into a double-model collaborative segmentation framework based on zero-shot learning to realize segmentation of single-plant canopy images; inputting the NIR band image in the multi-spectral image into the double-model collaborative segmentation framework based on zero-shot learning to realize segmentation of single-plant canopy images; wherein the double-model collaborative segmentation framework comprises a Grounding DINO model and a SAM model;

[0012] (4) using the segmented NIR band image to perform channel registration with other band images of the multi-spectral image to obtain segmented single-plant multi-spectral images;

[0013] (5) calculating binary image features, color features and texture features based on the segmented single-plant RGB images; calculating reflectances of multiple bands based on the segmented single-plant multi-spectral images, and calculating vegetation indices based on the reflectances of multiple bands;

[0014] (6) splicing the binary image features, color features, texture features, reflectances of multiple bands and vegetation indices corresponding to the single plant obtained in step (5) to construct high-dimensional features; matching the high-dimensional features of the single plant with the biomass information of the corresponding plant; inputting the high-dimensional features of the single plant into an improved SHAP model for feature dimension reduction to obtain effective features after dimension reduction;

[0015] (7) training a deep neural network using the effective features of the single plant and the biomass information of the corresponding plant to obtain a trained deep neural network model;

[0016] (8) collecting plant RGB images and multi-spectral images in the area to be estimated for biomass, and performing single-plant image segmentation on the RGB images and multi-spectral images using the method of steps (2)-(4) to calculate effective features corresponding to the single-plant images, and inputting the effective features into the trained deep neural network model to obtain the biomass information of the corresponding single plant.

[0017] As a further improved technical solution of the present application, the step (2) specifically comprises:

[0018] (2.1), pre-processing the RGB image by the improved ExG-P algorithm;

[0019] ExG-P = 3*G-0.4*R-B;

[0020] wherein R, G and B represent standard pixel values of the color channels of the image;

[0021] (2.2), generating a binary image by using the automatic Otsu threshold method on the RGB image processed by the improved ExG-P algorithm, and performing mask processing on the binary image and the RGB image processed by the improved ExG-P algorithm to obtain the final pre-processed RGB image.

[0022] As a further improved technical solution of the present application, the step (3) specifically comprises:

[0023] (3.a), segmentation of the RGB image:

[0024] (3.a.1), inputting the pre-processed RGB image into the Grounding DINO model, realizing semantic-level target detection by the Grounding DINO model through text prompts, calculating the intersection over union (IoU) between each detection box obtained by the Grounding DINO model and the corresponding labeled box of the actual plant canopy, and calculating the average value of all corresponding IoUs in the RGB image;

[0025] (3.a.2), if the average value of the IoU calculated in step (3.a.1) is greater than 90%, outputting the Grounding DINO model in step (3.a.1), the annotation image with the detection box and text information obtained by the Grounding DINO model, and the corresponding text file, and executing step (3.a.4);

[0026] (3.a.3), if the average value of the IoU calculated in step (3.a.1) is less than or equal to 90%, re-modifying the detection threshold of the Grounding DINO model, and returning to execute step (3.a.1);

[0027] (3.a.4), inputting the result output by the Grounding DINO model into the SAM model, calculating the intersection over union (IoU) between each segmentation region obtained by the SAM model and the corresponding artificially labeled actual segmentation region, and calculating the average value of all corresponding IoUs in the RGB image;

[0028] (3.a.5) If the average value of the intersection over union IoU calculated in step (3.a.4) is greater than 90%, output the SAM model in step (3.a.4) and the segmentation result of the single plant canopy image obtained by the SAM model;

[0029] (3.a.6) If the average value of the intersection over union IoU calculated in step (3.a.4) is less than or equal to 90%, re-modify the parameters in the SAM model, and return to execute step (3.a.4);

[0030] (3.b) Segmentation of the NIR band image in the multispectral image:

[0031] (3.b.1) Input the NIR band image in the multispectral image into the Grounding DINO model, and the Grounding DINO model realizes semantic-level target detection through text prompts, calculates the intersection over union IoU between each detection box obtained by the Grounding DINO model and the corresponding labeled box of the actual plant canopy, and calculates the average value of all corresponding intersection over union IoUs in the NIR band image;

[0032] (3.b.2) If the average value of the intersection over union IoU calculated in step (3.b.1) is greater than 90%, output the Grounding DINO model in step (3.b.1) and the annotated image with the detection box and text information and the corresponding text file obtained by the Grounding DINO model, and execute step (3.b.4);

[0033] (3.b.3) If the average value of the intersection over union IoU calculated in step (3.b.1) is less than or equal to 90%, re-modify the detection threshold of the Grounding DINO model, and return to execute step (3.b.1);

[0034] (3.b.4) Input the result output by the Grounding DINO model into the SAM model, calculate the intersection over union IoU between each segmentation region obtained by the SAM model and the corresponding manually labeled actual segmentation region, and calculate the average value of all corresponding intersection over union IoUs in the NIR band image;

[0035] (3.b.5) If the average value of the intersection over union IoU calculated in step (3.b.4) is greater than 90%, output the SAM model in step (3.b.4) and the segmentation result of the single plant canopy image obtained by the SAM model;

[0036] (3.b.6), if the average value of the intersection over union IoU calculated in step (3.b.4) is less than or equal to 90%, the parameters in the SAM model are re-modified, and step (3.b.4) is executed again.

[0037] As a further improved technical solution of the application, the step (6) of inputting the high-dimensional characteristics of the single plant into the improved SHAP model for dimension reduction is specifically:

[0038] The high-dimensional characteristics of the single plant are input into the SHAP model to obtain the SHAP contribution value corresponding to each characteristic. Then, the high-dimensional characteristics of the single plant are calculated by a plurality of rounds of Bootstrap sampling combined with a CatBoost regression model to obtain the feature importance score corresponding to each characteristic, and the feature importance scores corresponding to the same characteristics are averaged. Based on the average of the feature importance scores, the K-means clustering method is used to group similar characteristics, and the characteristics with a SHAP contribution value greater than a contribution threshold are selected from each group as effective characteristics after dimension reduction.

[0039] As a further improved technical solution of the application, the step (8) specifically includes:

[0040] (8.1), collecting the RGB image and the multispectral image of the plant to be estimated, and pre-processing the RGB image by the method of step (2);

[0041] (8.2), inputting the pre-processed RGB image into the Grounding DINO model output in step (3.a.2) for detection to obtain an annotated image with a detection frame and text information and a corresponding text file; inputting the result output by the Grounding DINO model into the SAM model output in step (3.a.5) for segmentation to obtain the segmentation result of the single plant canopy image;

[0042] (8.3), inputting the NIR band image in the multispectral image of the plant to be estimated into the Grounding DINO model output in step (3.b.2) for detection to obtain an annotated image with a detection frame and text information and a corresponding text file; inputting the result output by the Grounding DINO model into the SAM model output in step (3.b.5) for segmentation to obtain the segmentation result of the single plant canopy image;

[0043] (8.4), the multispectral image is channel registered with other band images after segmentation of the NIR band image in step (8.3) to obtain the segmented multispectral image of the single plant;

[0044] (8.5), based on the segmented single plant RGB image and the segmented single plant multispectral image, the effective features in step (6) are calculated, and the effective features are input into the trained deep neural network model to obtain the biomass information of the plant.

[0045] The present application overcomes the problems of high cost, long time, single data, and low accuracy of final phenotype evaluation caused by irrelevant imaging background in the existing image and spectrum-based plant phenotype evaluation technology. The present application develops an automatic and fast feature calculation and visualization program for RGB images and multispectral images, provides a multi-modal data fusion idea between different sensors, and establishes an accurate biomass prediction model. Specifically, the present application inputs the segmented single plant image into an automatic feature fast calculation system, extracts multi-modal information such as plant color, texture, binary, and spectral reflectance, selects appropriate variables through variable contribution value, establishes a high-precision biomass prediction model, and realizes fast, non-destructive, and accurate plant biomass phenotype acquisition.

[0046] Compared with the existing plant biomass estimation technology, the technical scheme provided by the present application has the following beneficial effects:

[0047] (1) The Grounding DINO and Segment Anything Model models are combined to complete single instance segmentation and autonomous saving and naming of visible light and multispectral plant two-dimensional image data, realize zero labeling and high efficiency based on double model cooperation and automatic process, significantly reduce artificial labeling cost and calculation time compared with mainstream deep learning algorithms, and the method has strong universality.

[0048] (2) Based on the accurate plant mask image data and using the image feature extraction system, the background interference is effectively excluded, the multi-modal features (binary, color, texture, and reflectivity) reflecting the plant growth state are quickly obtained, the accurate segmentation and phenotype feature automatic extraction are realized, and reliable data basis is provided for phenotype estimation and growth monitoring.

[0049] (3) The DNN biomass prediction model is constructed by fusing RGB and multispectral multi-modal image feature information, effectively overcoming the problem that the model accuracy cannot be effectively improved due to the data saturation problem of single image information, and through the improved SHAP variable screening method, although a small amount of variable screening time is increased, the variable dimension is reduced and the biomass estimation accuracy and speed are significantly improved.

[0050] (4) Compared with the traditional method, the artificial labeling cost is significantly reduced, and the processing efficiency is improved; the atlas information of the plant canopy is accurately obtained through automation, the feature representation capability is enhanced, and the limitation of single sensor data saturation on model accuracy is effectively overcome through fusion of multi-modal image information. The method is not affected by the variety limitation of plants, has strong universality, and can provide an efficient technical solution for high-throughput, non-destructive, accurate and rapid identification of plant phenotypes. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 It is a technical flowchart of the method of the application.

[0052] Figure 2 It is a double model collaboration segmentation framework diagram based on zero sample learning of the method of the application.

[0053] Figure 3 It is a plant image preprocessing effect comparison schematic diagram of the method of the application.

[0054] Figure 3 (a) in is a schematic diagram of the original image.

[0055] Figure 3 (b) in is a schematic diagram of the image after the original image is processed by the algorithm ExG.

[0056] Figure 3 (c) in is a schematic diagram of the image after the original image is processed by the improved super green algorithm ExG-P.

[0057] Figure 4 It is a schematic diagram of Grounding DINO completing target detection of different varieties of plants in the method of the application.

[0058] Figure 5 It is a schematic diagram of Segment Anything Model completing plant segmentation and saving in the method of the application.

[0059] Figure 5 (a) in is a schematic diagram of Segment Anything Model completing multispectral image segmentation and saving in the method of the application.

[0060] Figure 5 (b) in is a schematic diagram of Segment Anything Model completing RGB image segmentation and saving in the method of the application.

[0061] Figure 6 It is a schematic diagram of plant image part feature visualization in the method of the application.

[0062] Figure 6(a) in FIG. 1 is a schematic diagram of the visualization of the binary feature of the plant image part in the method of the present application.

[0063] Figure 6 (b) in FIG. 1 is a schematic diagram of the visualization of the color index feature of the plant image part in the method of the present application.

[0064] Figure 6 (c) in FIG. 1 is a schematic diagram of the visualization of the texture feature of the plant image part in the method of the present application.

[0065] Figure 6 (d) in FIG. 1 is a schematic diagram of the visualization of the reflectance feature of the plant image part in the method of the present application.

[0066] Figure 7 FIG. 2 is a schematic diagram of the accuracy comparison between the DNN biomass prediction model in the method of the present application and the conventional method.

[0067] Figure 7 (a) in FIG. 2 is a schematic diagram of the accuracy of the DNN biomass prediction model in the method of the present application (using the segmentation method of the present application and fusing multi-modal features).

[0068] Figure 7 (b) in FIG. 2 is a schematic diagram of the accuracy of the conventional method (without using the segmentation method of the present application, directly extracting features based on the original complete image, and fusing multi-modal features).

[0069] Figure 8 FIG. 3 is a schematic diagram of the comparison of the beneficial effects and contribution values of the improved SHAP feature screening and the original SHAP feature screening in the method of the present application.

[0070] Figure 8 (a) in FIG. 3 is a comparison diagram of the effects of the original SHAP feature screening and the improved SHAP feature screening in the method of the present application (from left to right, the number of variables selected by the model, the regression accuracy of the final model, and the variable screening time).

[0071] Figure 8 (b) in FIG. 3 is a comparison diagram of the types of the original SHAP feature screening and the improved SHAP feature screening in the method of the present application.

[0072] Figure 8 (c) in FIG. 3 is an original SHAP contribution analysis diagram.

[0073] Figure 8 (d) in FIG. 3 is an improved SHAP contribution analysis diagram in the method of the present application.

[0074] Figure 9 FIG. 4 is a schematic diagram of the accuracy evaluation of the biomass prediction model using only RGB or only multi-spectral image features.

[0075] Figure 9(a) in FIG. 1 is a model precision evaluation schematic diagram using only RGB image features.

[0076] Figure 9 (b) in FIG. 1 is a model precision evaluation schematic diagram using only multispectral image features. DETAILED DESCRIPTION

[0077] The specific embodiments of the present application are further described below with reference to the accompanying drawings:

[0078] A method for estimating plant biomass based on atlas multi-modal feature automatic extraction and fusion, comprising the following steps:

[0079] (1) Taking the plant in the sample plot using RGB and multispectral imaging sensors, and optionally using multiscale plant remote sensing image acquisition methods such as scale-fine ground self-propelled vehicles, medium-scale aerial unmanned aerial vehicles, etc. to collect plant RGB images and multispectral images in the study area, and destructively sampling the plant in the study area to determine biomass data.

[0080] (2) Preprocessing the RGB image using the improved ExG-P algorithm; that is, under the premise of not damaging the pixels of the plant itself, the improved ExG-P algorithm is used to remove part of the irrelevant background of the RGB image and reduce the storage cost, so as to improve the segmentation efficiency and feature calculation speed of the subsequent model.

[0081] (3) Inputting the preprocessed RGB image into the dual-model collaborative segmentation framework based on zero-shot learning, and performing single-plant canopy image segmentation through semantic-guided pixel segmentation; wherein the dual-model collaborative segmentation framework includes a Grounding DINO model and a SAM model.

[0082] Inputting the NIR band image in the multispectral image into the dual-model collaborative segmentation framework based on zero-shot learning to realize single-plant canopy image segmentation. Since plants have strong reflection characteristics in the NIR band, and non-vegetation backgrounds (such as soil and shadows) have weak reflection in this band, forming a significant contrast, segmentation based on the NIR band image is beneficial to improve the segmentation accuracy and robustness of the vegetation area.

[0083] (4) The RGB image in step (3) is directly segmented and saved, the multispectral image is channel-registered with the segmented NIR band image and other band images, then each sub-image is cropped and named according to the rule of from top to bottom and from left to right, during which the channel name needs to be defined for the multispectral image, and the image background is white, and finally the segmented single-plant multispectral image is obtained. The multispectral NIR band image is selected as the prior calculation for monitoring and segmentation tasks, and the result is used for matching subsequent other channels; all images are sequentially named and saved in the same format, and stored in JPG or PNG format.

[0084] (5) The plant image analysis system is used to automatically and batch calculate the binary image features, color features and texture features, etc. based on the segmented single-plant RGB image; the reflectance of multiple bands is automatically and batch calculated based on the segmented single-plant multispectral image, and the vegetation index is calculated based on the reflectance of multiple bands. The pixel value of 255 (white) is not involved in the calculation of all features, that is, the calculated feature information only includes the area where the plant itself is located, and is saved as a new EXCEL table.

[0085] (6) The binary image features, color features, texture features, reflectance of multiple bands and vegetation index, etc. corresponding to the single plant obtained in step (5) are spliced to construct a unified high-dimensional feature; the high-dimensional feature of the single plant is matched with the biomass information of the corresponding plant. In order to improve the calculation efficiency of subsequent modeling, SHAP (SHapley Additive exPlanations) value is introduced for feature contribution analysis, and redundant or weakly correlated features are removed. Specifically, the high-dimensional feature of the single plant is input into the improved SHAP model for feature dimension reduction, and the effective feature after dimension reduction is obtained.

[0086] (7) In the aspect of model construction, a deep neural network (DNN) is used as a basic regression model, and an automatic and manual parameter tuning strategy is combined to optimize the network structure and hyperparameter setting, so as to obtain the optimal model performance. Further, the network weight parameters obtained by model training are extracted and saved, providing a basis for subsequent rapid calling and deployment of the biomass prediction model. Specifically, the effective feature of the single plant and the biomass information of the corresponding plant are used to train the deep neural network, and a trained deep neural network model is obtained.

[0087] (8) Collecting the RGB image and the multispectral image of the plant in the region to be estimated, and using the method of steps (2)-(4) to segment the single plant image of the RGB image and the multispectral image, calculate the effective features corresponding to the single plant image, and input the effective features into the trained deep neural network model to obtain the biomass information of the corresponding single plant.

[0088] The step (2) comprises the following steps of:

[0089] (2.1) Preprocessing the RGB image by using the improved ExG-P algorithm:

[0090] ExG-P = 3*G-0.4*R-B;

[0091] Wherein, R, G, B represent the standard pixel value of the color channel of the image;

[0092] (2.2) Generating a binary image by using the automatic Otsu threshold method on the RGB image processed by the improved ExG-P algorithm, and performing mask processing on the binary image and the RGB image processed by the improved ExG-P algorithm to obtain the final preprocessed RGB image.

[0093] The step (3) comprises the following steps of:

[0094] (3.a) Segmentation of the RGB image:

[0095] (3.a.1) Inputting the preprocessed RGB image into the Grounding DINO model, and realizing semantic-level target detection by the text prompt of the Grounding DINO model, calculating the intersection over union (IoU) between each detection box obtained by the Grounding DINO model and the labeled box of the corresponding actual plant canopy, and calculating the average value of all the intersection over union (IoU) in the RGB image.

[0096] (3.a.2) If the average value of the intersection over union (IoU) calculated in step (3.a.1) is greater than 90%, outputting the Grounding DINO model in step (3.a.1), the annotation image with the detection box and the text information obtained by the Grounding DINO model, and the corresponding text file, and executing step (3.a.4).

[0097] (3.a.3) If the average value of the intersection over union (IoU) calculated in step (3.a.1) is less than or equal to 90%, re-modifying the detection threshold of the Grounding DINO model, and returning to execute step (3.a.1).

[0098] (3.a.4), inputting the result output by the Grounding DINO model into the SAM model, calculating the IoU of each segmentation region obtained by the SAM model and the actual segmentation region after artificial labeling, and calculating the average value of all corresponding IoUs in the RGB image.

[0099] (3.a.5), if the average value of the IoU calculated in step (3.a.4) is greater than 90%, outputting the SAM model in step (3.a.4) and the segmentation result of the single plant canopy image obtained by the SAM model;

[0100] (3.a.6), if the average value of the IoU calculated in step (3.a.4) is less than or equal to 90%, re-modifying the parameters in the SAM model, and returning to execute step (3.a.4);

[0101] (3.b), segmentation of the NIR band image in the multispectral image:

[0102] (3.b.1), inputting the NIR band image in the multispectral image into the Grounding DINO model, realizing semantic-level target detection by text prompts, calculating the IoU between each detection box obtained by the Grounding DINO model and the corresponding labeled box of the actual plant canopy, and calculating the average value of all corresponding IoUs in the NIR band image;

[0103] (3.b.2), if the average value of the IoU calculated in step (3.b.1) is greater than 90%, outputting the Grounding DINO model in step (3.b.1) and the annotated image with the detection box and text information and the corresponding text file obtained by the Grounding DINO model, and executing step (3.b.4);

[0104] (3.b.3), if the average value of the IoU calculated in step (3.b.1) is less than or equal to 90%, re-modifying the detection threshold of the Grounding DINO model, and returning to execute step (3.b.1);

[0105] (3.b.4), inputting the result output by the Grounding DINO model into the SAM model, calculating the IoU of each segmentation region obtained by the SAM model and the actual segmentation region after artificial labeling, and calculating the average value of all corresponding IoUs in the NIR band image;

[0106] (3.b.5) If the average value of the intersection over union IoU calculated in step (3.b.4) is greater than 90%, output the SAM model in step (3.b.4) and the segmentation result of the single plant canopy image obtained by the SAM model;

[0107] (3.b.6) If the average value of the intersection over union IoU calculated in step (3.b.4) is less than or equal to 90%, re-modify the parameters in the SAM model, and return to execute step (3.b.4).

[0108] In this embodiment, all pre-processed images are input into a double-model collaborative segmentation framework based on zero-shot learning. Through semantic guided pixel segmentation, the segmentation of plant canopy images is performed. When the model starts inputting the language prompt "plant", the Grounding DINO realizes semantic-level target detection through the text prompt ("plant"). By automatically checking the intersection over union (IoU) between the detection box and the actual plant canopy annotation box, when the precision (the average value of the intersection over union between all detection boxes and annotation boxes) is greater than 90%, the annotated image with the detection box and the text information and the corresponding text result file are output. If the detection box precision (the average value of the intersection over union between all detection boxes and annotation boxes) is less than or equal to 90%, the detection threshold of the Grounding DINO is re-modified for detection box detection, and after the precision is reached, the relevant files are output. The output result is input to the Segment Anything Model (SAM) for plant canopy pixel-level segmentation, thereby realizing double-model collaborative segmentation of plant zero-shot learning.

[0109] The detection information (including the original pre-processed image and the corresponding text file) output by the Grounding DINO is input to the Segment Anything Model (SAM), and the intersection over union IoU between the segmentation result and the manual annotation result is calculated by using a Python program. The IoU is compared by comparing the segmentation pixel area and the manual annotation pixel area. When the average value of all segmentation results corresponding to the IoU is greater than 0.9, the saving path of the final segmentation image, the mask image and the corresponding text information is set, a new folder is created, and each single plant sub-image obtained by segmenting the image is named as a new name of "original image name + sequence number" in the order of "from top to bottom, from left to right", and the background pixel value of the segmented image is set to white (RGB value is 255). When the average value of all segmentation results corresponding to the IoU is less than or equal to 0.9, the parameters or training model of the SAM are adjusted to improve the segmentation precision.

[0110] The Grounding DINO selected in this embodiment is used as a visual language model, which is pre-trained on large-scale image-text pairs to learn rich semantic information. In terms of effect, it can accurately locate the target objects in the image, even if the background is complex and there are many targets, it still has high accuracy, and it can also understand the semantic information of the text and find the matching objects or regions. In this embodiment, Grounding DINO is used as a target detection model to realize the recognition of segmented objects in the image and assign a bounding box and text information to the segmented objects.

[0111] The SAM selected in this embodiment is a general-purpose model with strong image segmentation capability, and its core goal is to achieve high-quality segmentation of any object or region. Through training on large-scale image datasets, SAM has mastered rich visual features and has good generality and migration ability. It supports multiple interactive prompting methods such as clicking, bounding box drawing or text prompting, and can flexibly adapt to segmentation tasks in different scenarios. In plant image processing, SAM can accurately generate a segmentation mask of the target region, even if it faces a complex structure or irregular plant canopy, it can still achieve a clear segmentation effect. However, SAM has a key limitation when performing image segmentation: it cannot identify the specific class information of the segmented objects, which poses a barrier to tasks that require semantic differentiation of different organs or targets in plant phenotype analysis. In addition, SAM may also perform meaningless segmentation on background elements in the image, affecting the accuracy of subsequent analysis.

[0112] To solve this problem and reduce the dependence of traditional segmentation models on manual annotation data, this example proposes a dual-model collaborative segmentation framework based on zero-shot learning. This method combines the text-guided target detection capability of Grounding DINO with the high-quality segmentation capability of SAM: first, Grounding DINO automatically detects and identifies the target plant region in the image according to the set text prompt, and clearly defines its semantic category and location; By setting the accuracy of the recognition box, the detection results are then passed to SAM as input prompts to further complete accurate image segmentation. And through the accuracy detection in stages to carry out precision inspection. This two-stage process not only effectively overcomes the problem that SAM cannot identify the category, but also realizes the automatic and semantic processing of plant images under the condition of "zero labeling". It greatly improves the calculation accuracy and automation of plant phenotype features (such as leaf area, canopy coverage, biomass estimation, etc.), and provides an efficient and reliable solution for high-throughput plant phenotype analysis.

[0113] The Grounding DINO model is used to realize zero-shot object detection of plant objects in an image, and the detection result containing a detection box and text information is output. Through the SAM model, according to the input detection box and text information, zero-shot instance segmentation of the plant object is realized; the Grounding DINO and the SAM cooperate to form a double-model cooperative segmentation framework based on zero-shot learning, which provides accurate plant mask images for subsequent feature extraction.

[0114] The IoU calculation formula is as follows: ;

[0115] Wherein:

[0116] A represents a predicted segmentation area, B represents a real labeled area, |A∩B| represents the intersection area of the predicted area and the real area, |A∪B| represents the union area of the predicted area and the real area, and the closer the IoU value is to 1, the more consistent the prediction result is with the real label.

[0117] When the average value of the Grounding DINO detection box IoU precision is less than or equal to 0.9, the detection threshold of the model is adjusted. When the average value of the SAM segmentation pixel area IoU precision is less than or equal to 0.9, the pre-training weight model of the SAM model is replaced. Among them, the official SAM has three pre-training model weights of ViT-B, ViT-L and ViT-H.

[0118] Step (4) specifically comprises: multi-spectral image registration is completed by calculating the phase difference between different channel images of the same plant, and specifically comprises four processes of Hessian matrix construction, scale space construction, feature point description, feature point matching and correction. After aligning in each channel, the mask region image segmented in the example channel is selected for all segmentation and saving processing of other channels.

[0119] Step (5) specifically comprises: inputting the single plant RGB image into the plant image analysis system (i.e. the automatic analysis system), and calculating the binary image features, texture features and color features respectively.

[0120] In order to effectively extract the plant area and eliminate the background interference, the vegetation coverage area is extracted by the image binary method, which provides basic information for estimating the canopy coverage and preliminary biomass. The pixel point number with a pixel value not equal to 255 is calculated from the single plant image part segmented by the combination of the Grounding DINO and the SAM model based on the RGB image binary image features, and the converted value after using the ground calibration object is used as the reference value of the plant canopy projection area Project Area.

[0121] Further, the RGB image texture features are calculated based on a Grey level co-occurrence matrix (GLCM). The texture features are used to describe the complexity of the canopy structure and the spatial heterogeneity of the leaf arrangement, and can indirectly reflect the vegetation density and organization degree, thereby providing a structural basis for biomass estimation. The GLCM is a matrix function related to the pixel distance and angle, and reflects the spatial variation of the image gray distribution by calculating the correlation between the gray levels of two points separated by a certain distance in a specific direction. In the examples of the method, nine common texture features are selected, and the variable calculation formula is shown in Table 1. In order to reduce the irrelevance and dimension of the data, the texture features in the GLCM are calculated in the 0° direction and with a 7*7 window size.

[0122] Table 1:

[0123]

[0124] wherein i is the gray value of any point, and j is the gray value deviating from the point, both of which are dimensionless units. x is the mean value of the texture parameter in the row direction, which is a dimensionless unit. x is the variance of the texture parameter in the row direction, which is a dimensionless unit. y is the mean value of the texture parameter in the column direction, which is a dimensionless unit. y is the variance of the texture parameter in the column direction, which is a dimensionless unit. P(i,j) is the frequency of the pixel point with a distance of d and a gray value of j from (x,y), which is a dimensionless unit. Mean is the mean value.

[0125] The color features reflect the color distribution and change of the leaves, and can be used to identify the nutritional status and health degree of the vegetation, which are closely related to the photosynthesis and biomass accumulation of the plants. The RGB color features are calculated by calculating the R, G and B single-channel average pixel values in the plant area, simultaneously extracting the color space indexes (H, S and V), and constructing color indexes reflecting different color information of the plants based on different function combinations, and the variable calculation formula selected is shown in Table 2.

[0126]

[0127] wherein R, G and B are the single-channel standard pixel values in the plant area, which are dimensionless units.

[0128] Vegetation index fuses multi-band reflectance information, which can sensitively capture plant carbon accumulation and photosynthetic activity, and is one of the core indicators for estimating vegetation biomass and productivity. In the automatic analysis system of single plant multispectral image input, the reflectance of R, G, B, NIR, RE five bands is calculated, and in order to distinguish the naming, the RGB band reflectance is defined as R_m, G_m, B_m here. In feature calculation, all sub-images are traversed through the naming of different bands, and the correction value required for reflectance calculation is obtained by processing the correction plate image before calculation. The reflectance of non-zero pixels is calculated by applying the band-specific correction value and the known scale factor, and then the vegetation index is calculated. The calculated and screened vegetation index in the multispectral image is specifically listed in Table 3:

[0129]

[0130] In the formula, i is the gray value of any point, which is a dimensionless unit.

[0131] On the basis of the above step (6), the calculated multi-modal variable is matched with the corresponding biomass information. The binary features, color features, texture features, reflectance features, color space features and vegetation index and other one-dimensional features extracted from the visible light image and the multispectral image are spliced to construct a unified high-dimensional feature set.

[0132] The high-dimensional features of single plants are input into the improved SHAP model for feature dimension reduction. Specifically, the high-dimensional features of single plants are input into the SHAP model to obtain the SHAP contribution value corresponding to each feature. Then, the high-dimensional features of single plants are calculated by multiple rounds of Bootstrap combined with CatBoost regression model to obtain the feature importance score corresponding to each feature, and the feature importance scores corresponding to the same feature are averaged. Based on the average of the feature importance scores and using K-means clustering, similar features are grouped, and representative features (i.e. features with SHAP contribution value greater than 0.5) are selected from each group as effective features after dimension reduction, thereby realizing automatic dimension reduction of high-dimensional features.

[0133] In step (7), when the DNN model is used to construct plant biomass regression analysis, in order to analyze the fitting effect, accuracy and reliability of the regression model, the coefficient of determination (Coefficient of determination, R 2 ), root mean squared error (Root mean squared error, RMSE), mean absolute error (Mean absolute error, MAE) are selected to evaluate the effect of each model.

[0134] The plant biomass prediction process based on a deep neural network (DNN) in step (7) specifically includes:

[0135] 1. A five-layer (input layer→512→256→128→32→1) fully connected network structure is constructed, with the number of layers decreasing from the input layer to the output layer, and effective feature data is mapped to biomass values through progressive feature compression; an improved convolution block attention module (CBAM) is integrated between the 256→128 dimensional transition layer of the network, the channel attention (Channel Attention) and the feature gating mechanism (Spatial Gate) are combined, and the average pooling and maximum pooling double paths and the 16:1 compression ratio are adopted, the attention weight is generated through the Sigmoid activation function, and the adaptive weighting of the key spectral band and the feature interaction mode is realized.

[0136] 2. Loss function and regularization: a customized Huber loss function (delta=1.0) is used to replace the mean square error, combined with L2 regularization, Dropout and batch normalization (BatchNorm1d) to prevent overfitting.

[0137] 3. Optimization and training stability: the AdamW optimizer (initial learning rate 0.002, weight decay 1×10 -5 ) is used in combination with the adaptive learning rate scheduler (Reduce Learning Rate On Plateau, ReduceLROnPlateau) (decay factor 0.3, patience value 20), and the gradient clipping (maximum norm 0.5) and early stopping strategy (patience value 50) are introduced to ensure stable convergence.

[0138] 4. Effective feature selection: based on the efficient fusion SHAP (SHapley Additive exPlanations) algorithm, the feature importance score is calculated through multiple rounds of Bootstrap combined with the CatBoost regressor, and the mean value is taken to reduce noise; K-means clustering is used to group similar importance features, and representative features are selected from each group to remove redundancy and maintain diversity, realizing automatic dimensionality reduction of variable features.

[0139] The step (8) specifically includes:

[0140] (8.1), collect the RGB image and multispectral image of the plant to be estimated, and pretreat the RGB image by the method of step (2);

[0141] (8.2), input the pre-processed RGB image into the Grounding DINO model output in step (3.a.2) for detection to obtain an annotated image with a detection frame and text information and a corresponding text file; input the result output by the Grounding DINO model into the SAM model output in step (3.a.5) for segmentation to obtain a segmentation result of the single-plant canopy image;

[0142] (8.3), input the NIR band image in the plant multispectral image to be estimated into the Grounding DINO model output in step (3.b.2) for detection to obtain an annotated image with a detection frame and text information and a corresponding text file; input the result output by the Grounding DINO model into the SAM model output in step (3.b.5) for segmentation to obtain a segmentation result of the single-plant canopy image;

[0143] (8.4), the multispectral image uses the NIR band image segmented in step (8.3) to perform channel registration with other band images to obtain a segmented single-plant multispectral image;

[0144] (8.5), based on the segmented single-plant RGB image and the segmented single-plant multispectral image, calculate the effective features in step (6), input the effective features into the trained deep neural network model to obtain the biomass information of the plant.

[0145] In summary, the method of the present application mainly includes multiple steps of image data preprocessing, plant target detection, multi-modal image segmentation, automatic feature calculation, and phenotype parameter evaluation model establishment. By inputting the preprocessed image into the zero-shot target detection model Grounding DINO, the plant object in the image is automatically recognized and the initial detection box and text information are generated; optionally, a small amount of detection error results are checked and corrected; then, the checked detection information is input into the zero-shot segmentation model Segment Anything Model (SAM), and the accurate instance segmentation of the single plant object is realized, and the whole process does not need manual annotation of training data. An automatic program is developed to batch calculate the spectral features (including binary features, color features, texture features and multispectral reflectance features) in the plant mask image region obtained by segmentation; the multi-modal feature information from the RGB and multispectral images is fused; based on the preferred feature variables, a plant biomass estimation model is constructed using a deep neural network (Deep Neural Network, DNN). Compared with the traditional method, the present application significantly reduces the manual annotation cost and improves the processing efficiency; by automatically and accurately obtaining the spectral information of the plant canopy, the feature representation ability is enhanced; by fusing the multi-modal image information, the limitation of the model accuracy caused by the saturation of single sensor data is effectively overcome. The method of the present application is not affected by the limitation of plant varieties, and has strong universality, and can provide an efficient technical solution for high-throughput, non-destructive, accurate and rapid identification of plant phenotypes.

[0146] The present application selects multiple varieties of poplar to carry out tests, sets different irrigation groups so that the plant growth reflects certain differences and enhances the data richness. The flowchart of the method of the present application is as follows Figure 1As shown, the RGB image input into the Grounding DINO model for target detection is preprocessed by ExG_P as described above, and the irrelevant imaging background of the RGB image is removed, thereby greatly reducing the storage size of the RGB image. The Grounding DINO model and the Segment Anything Model model can be directly configured and used on the official website. In order to realize batch processing of data, the file reading and saving in the model need to be changed accordingly. The TEST part specifically changes the text file and the original image saved after target detection of the Grounding DINO model into the labelimg program in python. Compared with traditional deep learning, the data change in this part is directly used as a test set, which can save a lot of image labeling time. The image features calculated from the mask region of the multispectral image are the reflectance R_m, G_m, B_m, NIR, RE of five bands, and the vegetation index constructed by a specific function combination; the features calculated from the mask region of the RGB image include binary features, texture features, single-band standard pixel values in the RGB color space, and color indices constructed by a specific function combination of them.

[0147] As shown in Figure 2 , the present study proposes a dual-model collaborative segmentation framework based on zero-shot learning. The framework shows the preprocessed RGB image and multispectral image inputting the Grounding DINO model from top to bottom to realize the target detection of the plant canopy, and then the IoU precision of the prediction box is evaluated by code, and the feedback mechanism is used to ensure that the detection result reaches the preset precision threshold. The detection box with high precision will be passed to the SAM model for further fine segmentation of the plant canopy. The segmentation result is also checked for accuracy by the feedback mechanism, and the final output is the canopy mask image that meets the requirements.

[0148] As shown in Figure 3 (a)-(c) in the present application, from left to right, the contrast of the original image, the ExG processed image by the traditional super green algorithm, and the ExG_P processed image by the improved super green algorithm is shown. As can be seen from the figure, the image processed by ExG eliminates most of the red bud pixels of the early stage of plant growth, which will reduce the comprehensiveness and accuracy of image feature calculation. The method of the present application increases the sensitivity to red channel values in ExG_P, and retains the red part of the plant.

[0149] As shown in Figure 4As shown, the target detection results of different plants by the Grounding DINO are visualized, the Grounding DINO model calculates the correlation degree of each position in the image feature sequence with other positions through the self-attention mechanism of the Transformer, helps the model pay attention to the relationship between plant leaves and branches, the relative position between different plants and other details, in this way, the model can adaptively capture the global and local features of the plant target, thereby completing the detection task in the content of the text prompt.

[0150] As shown in the present application Figure 5 As shown, Figure 5 (a) in the present application Figure 5 (b) in the present application is a schematic diagram of the RGB image segmentation and preservation completed by the Segment Anything Model in the method of the present application, that is, a schematic diagram of the plant segmentation and preservation completed by SAM, after inputting the target detection text data and the corresponding original image, the model starts to segment the content of the text prompt using the GPU model, and the mask image and single plant image after segmentation need to be saved and named;

[0151] After the above segmentation of the whole image, the pixel center point position of each object is found, and the image naming order is provided according to the naming rule from top to bottom and from left to right, for example, the plant at the left upper corner of an RGB image will be named as test serial number + culture group treatment name + variety name + plant image serial number, and the multispectral image will be named as test serial number + culture group treatment name + variety name + wave band name + plant image serial number for completion of saving, the hierarchical content of the saved folder is from one culture group treatment to one plant, and one plant contains two types of images, and this rule is mainly used for corresponding to the actual biomass data of the collected plants.

[0152] As shown in the present application Figure 6 As shown, Figure 6 (a) is the binary feature visualization of the RGB image mask region; Figure 6 (b) is the RGB image color index visualization (part), and the color scale used is Jet in the python language; Figure 6 (c) is the RGB texture feature visualization (part), and the color scale used is Gray in the python language; Figure 6(d) Vegetation index visualization (part) calculated for the masked region of the multispectral image, using the color scale Paired in Python. The calculated image is for a randomly selected treatment group.

[0153] As shown in the present application Figure 7 , the DNN algorithm is used to establish a biomass regression model. The model method is good at capturing local patterns and structural information in input data, and improves model efficiency and robustness through parameter sharing and translation invariance, which shows certain superiority in high-dimensional data processing. By comparing the feature extraction effect and model prediction accuracy under different segmentation strategies, the effectiveness of the double model cooperation mechanism in improving the performance of biomass estimation is further evaluated. Figure 7 (a) in and Figure 7 (b) in are the accuracy comparison of the two modeling methods, where the proportion of the divided data set is consistent, and the training set and test set are divided according to the proportion of 8:2, and the accuracy result shown is the test set result of the model. As Figure 7 shown, the R² of the model constructed by the method of the present application reaches 0.87, which is improved by 11 percentage points compared with the conventional method (R²=0.76); the RMSE and MAE are also significantly reduced.

[0154] Figure 8 (a)-(d) in compares the performance of improved SHAP (Efficient SHAP) and original SHAP in feature selection. Compared with the original method, the improved SHAP (Efficient SHAP) introduces a clustering analysis strategy in the feature selection process, making the variable selection more concise and targeted. Specifically, the improved SHAP (Efficient SHAP) only retains 8 key features (5 multispectral features and 3 RGB features), while the original SHAP selects 24 features (15 multispectral features and 9 RGB features). This significant dimension reduction effectively reduces redundant information, not only reducing the computational complexity of subsequent modeling, but also significantly shortening the model training time. In terms of prediction performance, the improved SHAP (Efficient SHAP) achieves a regression accuracy of 87%, which is better than the original SHAP of 85%, and also shows a clearer and more reasonable feature importance distribution in variable contribution analysis (c, 8d). In summary, the improved SHAP (Efficient SHAP) not only guarantees or improves the model accuracy, but also realizes more efficient feature selection and modeling process, highlighting its application advantages in large-scale, multi-source data modeling. Figure 8

[0155] As Figure 9 ​As shown in (a)-(b) in the table, the model R² using only RGB features is 0.56, and the model R² using only multispectral features is 0.62; compared with the two, the method (R²=0.87) of fusing multi-modal features of the application improves by 31 and 25 percentage points, respectively, and significantly overcomes the limitations of a single data source.

[0156] The method of the application realizes accurate segmentation of single plants under the premise of not requiring a large training set by using a double-model collaborative segmentation framework based on zero-shot learning; meanwhile, multi-modal information such as plant color, texture, binary, spectral reflectance and the like is extracted based on the segmented single plant images, an improved SHAP model is introduced to reduce the dimension of the feature space, so as to improve the inversion accuracy and operation efficiency while ensuring the effectiveness of the information; a high-precision biomass prediction model is established by selecting appropriate variables through variable contribution values, and fast, non-destructive and accurate plant biomass phenotype acquisition is realized.

[0157] To verify the effectiveness and applicability of the method of the application, the model constructed based on the multi-modal feature fusion and optimization strategy proposed in the application is compared with the model constructed based on the traditional original image features (without fusion and optimization processing). In addition, during the comparison experiment, the method of the application introduces multi-modal image information (such as RGB and multispectral data) in the feature fusion process, to a certain extent, it alleviates the precision bottleneck problem caused by the saturation effect of a single sensor data, significantly improves the recognition and prediction ability of different plant growth stage biomass, and further proves the wide applicability and promotion potential of the method in plant phenotype research.

[0158] The above is only a specific implementation case of the method of the application, but is not used to completely limit the implementation range of the method of the application. That is, any equivalent changes and modifications made in accordance with the scope of the claims of the method of the application, but as long as the essence does not deviate from the inventive points and effects of the method of the application, there can be many deformations, which are obvious to those skilled in the art. Therefore, such deformation examples are also entirely included in the protection scope of the method of the application.

[0159] The method of the application for estimating plant biomass by fusing multi-modal data based on automatic extraction of atlas multi-modal features is described in detail above, and the exemplary embodiments of the application are described above with reference to the drawings. Those skilled in the art should understand that the above embodiments are only examples for the purpose of illustrating the application, and are not used to limit, any modifications, equivalent replacements and the like made within the teaching and protection scope of the claims should be included in the protection scope of the application.

Claims

1. A method for estimating plant biomass based on atlas multi-modal feature extraction and fusion, characterized in that, It comprises the following steps: (1) Collecting the RGB image and the multi-spectral image of the plants in the research area, and destructively sampling the biomass data of the plants in the research area; (2) Applying the improved ExG-P algorithm to the RGB image for pretreatment; (3) Inputting the pretreated RGB image into the double-model collaborative segmentation framework based on zero-shot learning to realize the segmentation of the single-plant canopy image; inputting the NIR band image in the multi-spectral image into the double-model collaborative segmentation framework based on zero-shot learning to realize the segmentation of the single-plant canopy image; The double-model collaborative segmentation framework comprises a Grounding DINO model and a SAM model; (4) The multi-spectral image uses the segmented NIR band image to perform channel registration with other band images to obtain the segmented single-plant multi-spectral image; (5) Calculating the binary image features, color features and texture features based on the segmented single-plant RGB image; Calculating the reflectance of multiple bands based on the segmented single-plant multi-spectral image, and calculating the vegetation index based on the reflectance of multiple bands; (6) Splicing the binary image features, color features, texture features, reflectance of multiple bands and vegetation index corresponding to the single plant obtained in step (5) to construct high-dimensional features; matching the high-dimensional features of the single plant with the biomass information of the corresponding plant; inputting the high-dimensional features of the single plant into the improved SHAP model for feature dimension reduction to obtain the effective features after dimension reduction; (7) Training the deep neural network with the effective features of the single plant and the biomass information of the corresponding plant to obtain the trained deep neural network model; (8) Collecting the RGB image and the multi-spectral image of the plants in the area to be estimated, and using the method of steps (2)-(4) to segment the single-plant image of the RGB image and the multi-spectral image, calculate the effective features corresponding to the single-plant image, and input the effective features into the trained deep neural network model to obtain the biomass information of the corresponding single plant.

2. The method for estimating plant biomass based on atlas multi-modal feature extraction and fusion according to claim 1, characterized in that, The step (2) comprises the following steps: (2.1) Pretreating the RGB image by the improved ExG-P algorithm: ExG-P = 3*G-0.4*R-B; Wherein, R, G, B represent the standard pixel value of the image color channel; (2.2) Generating a binary image from the RGB image pretreated by the improved ExG-P algorithm using the automatic Otsu threshold method, and performing mask processing on the binary image and the RGB image pretreated by the improved ExG-P algorithm to obtain the final pretreated RGB image. 3.The method for estimating plant biomass based on atlas multi-modal feature extraction and fusion according to claim 1, characterized in that, The step (3) comprises the following steps: (3.a) Segmentation of the RGB image: (3.a.1), input the pre-processed RGB image into the Grounding DINO model, the Grounding DINO model realizes semantic-level target detection through a text prompt, and calculates the intersection over union (IoU) between each detection box obtained by the Grounding DINO model and the corresponding labeled box of the actual plant canopy; calculate the average value of all corresponding intersection over union (IoU) in the RGB image; (3.a.2), if the average value of the intersection over union (IoU) calculated in step (3.a.1) is greater than 90%, output the Grounding DINO model in step (3.a.1), the annotated image with detection boxes and text information obtained by the Grounding DINO model, and the corresponding text file, and execute step (3.a.4); (3.a.3), if the average value of the intersection over union (IoU) calculated in step (3.a.1) is less than or equal to 90%, the detection threshold of the Grounding DINO model is modified again, and step (3.a.1) is executed again; (3.a.4), input the result output by the Grounding DINO model into the SAM model, calculate the intersection over union (IoU) between each segmentation region obtained by the SAM model and the corresponding manually annotated actual segmentation region, and calculate the average value of all corresponding intersection over union (IoU) in the RGB image; (3.a.5), if the average value of the intersection over union (IoU) calculated in step (3.a.4) is greater than 90%, output the SAM model in step (3.a.4) and the segmentation result of the single plant canopy image obtained by the SAM model; (3.a.6), if the average value of the intersection over union (IoU) calculated in step (3.a.4) is less than or equal to 90%, the parameters in the SAM model are modified again, and step (3.a.4) is executed again; (3.b), segmentation of the NIR band image in the multispectral image: (3.b.1), input the NIR band image in the multispectral image into the Grounding DINO model, the Grounding DINO model realizes semantic-level target detection through a text prompt, and calculates the intersection over union (IoU) between each detection box obtained by the Grounding DINO model and the corresponding labeled box of the actual plant canopy; calculate the average value of all corresponding intersection over union (IoU) in the NIR band image; (3.b.2), if the average value of the intersection over union (IoU) calculated in step (3.b.1) is greater than 90%, output the Grounding DINO model in step (3.b.1), the annotated image with detection boxes and text information obtained by the Grounding DINO model, and the corresponding text file, and execute step (3.b.4); (3.b.3), if the average value of the intersection over union (IoU) calculated in step (3.b.1) is less than or equal to 90%, the detection threshold of the Grounding DINO model is modified again, and step (3.b.1) is executed again; (3.b.4), inputting the result output by the Grounding DINO model into the SAM model, and calculating the intersection over union IoU of each segmentation region obtained by the SAM model and the actual segmentation region after manual labeling corresponding to the actual segmentation region; (3.b.5), if the average value of all intersection over union IoUs calculated in step (3.b.4) is greater than 90%, outputting the SAM model in step (3.b.4) and the segmentation result of the single-plant canopy image obtained by the SAM model; (3.b.6), if the average value of all intersection over union IoUs calculated in step (3.b.4) is less than or equal to 90%, re-modifying the parameters in the SAM model, and returning to step (3.b.4) for execution.

4. The method for estimating plant biomass based on atlas multi-modal feature extraction and fusion according to claim 3, characterized in that, The step (6) of inputting the high-dimensional features of the single plant into the improved SHAP model for feature dimension reduction specifically comprises: inputting the high-dimensional features of the single plant into the SHAP model to obtain the SHAP contribution value corresponding to each feature, then the high-dimensional features of the single plant are calculated by the Bootstrap combined with the CatBoost regression model to obtain the feature importance score corresponding to each feature, and the feature importance scores corresponding to the same feature are averaged; based on the average value of the feature importance score and using the K-means clustering method, similar features are grouped, and features with a SHAP contribution value greater than a contribution threshold are selected from each group as effective features after dimension reduction.

5. The method for estimating plant biomass based on atlas multi-modal feature extraction and fusion according to claim 3, characterized in that, The step (8) specifically comprises: (8.1), collecting the RGB image and the multi-spectral image of the plant to be estimated, and pre-processing the RGB image by the method of step (2); (8.2), inputting the pre-processed RGB image into the Grounding DINO model output in step (3.a.2) for detection to obtain an annotated image with a detection frame and text information and a corresponding text file; inputting the result output by the Grounding DINO model into the SAM model output in step (3.a.5) for segmentation to obtain the segmentation result of the single-plant canopy image; (8.3), inputting the NIR band image in the multi-spectral image of the plant to be estimated into the Grounding DINO model output in step (3.b.2) for detection to obtain an annotated image with a detection frame and text information and a corresponding text file; inputting the result output by the Grounding DINO model into the SAM model output in step (3.b.5) for segmentation to obtain the segmentation result of the single-plant canopy image; (8.4), performing channel registration on the NIR band image segmented in step (8.3) and other band images of the multi-spectral image to obtain the segmented single-plant multi-spectral image; (8.5), based on the segmented single-plant RGB image and the segmented single-plant multi-spectral image, calculating the effective features in step (6), and inputting the effective features into the trained deep neural network model to obtain the biomass information of the plant.

Citation Information

Cited By

  • Construction method and system of cucumber leaf health state detection model

    CN121305122A

  • A method and system for constructing a cucumber leaf health state detection model

    CN121305122B

  • Panda staple food bamboo variety drawing method based on unmanned aerial vehicle image and deep learning

    CN122157071A