Intelligent identification method for crown layer growth of large-leaf hydrangea based on multi-modal computer vision

Abstract

The application discloses a large-leaf hydrangea crown layer growth condition intelligent identification method based on multi-modal computer vision, collects a digital image of a large-leaf hydrangea plant, and cuts the image into image blocks after pretreatment; a deep neural network model pre-trained unsupervised is used to extract local morphological features of the image blocks; a text-image alignment deep neural network based on cross-modal contrast learning is used for weakly supervised training by using labeled text information to realize associated learning of the image and the text information; multi-modal feature fusion is performed on the text features and the image features; based on the fusion representation features, output branch models are designed for different tasks to predict or identify the large-leaf hydrangea crown layer growth condition. The application effectively fuses multi-modal information of two modes of images and texts, extracts different levels of visual representations by using two complementary pre-training methods, realizes multi-task processing, and achieves extremely high precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a method for identifying and predicting the crown layer growth of Hydrangea macrophylla. BACKGROUND

[0002] Hydrangea macrophylla is a popular flowering shrub with a wide variety of varieties, and it is necessary to select varieties with high garden application value. Spring is the golden period of Hydrangea macrophylla growth, and is also the key stage for determining the annual flowering amount. Warm and humid environment in spring is easy to breed pests such as aphids and red spiders, so prevention should be done in advance. Hydrangea macrophylla is a typical "foodie" plant, and after the new buds germinate in spring, the demand for nutrients increases sharply. If there is a lack of nutrients at this time, the new buds will become weak, and the flowering amount will also be greatly reduced. Therefore, the identification and prediction of the growth state of Hydrangea macrophylla can be targeted for maintenance.

[0003] In the prior art, the monitoring of the growth state of plants is mostly based on computer vision method to identify the growth state of plants, so as to judge the growth state of plants, but has the following disadvantages: 1. Simple image processing has low recognition accuracy; 2. A large amount of plant growth parameters and various marker data are needed to train the model in the identification process, which is time-consuming and difficult to guarantee the effect of the model; 3. Based on the preset neural network model, the text information recorded by plant maintenance experts cannot be effectively combined for training, and the recognition / prediction accuracy is still difficult to guarantee. SUMMARY

[0004] The purpose of the present application is to solve the defects in the prior art, and to provide an intelligent identification method for the crown layer growth of Hydrangea macrophylla, which effectively associates the visual features of plants with text information and improves the prediction accuracy.

[0005] In order to achieve the above purpose, the present application provides an intelligent identification method for the crown layer growth of Hydrangea macrophylla based on multi-modal computer vision, which comprises the following methods:

[0006] (1) Data acquisition: acquiring digital images of Hydrangea macrophylla plants;

[0007] (2) Image preprocessing: removing the non-target background area of the collected image, and cutting to generate a plurality of small size image blocks which do not overlap each other;

[0008] (3) Multi-modal neural network construction:

[0009] Image encoding branch: a deep neural network model pre-trained unsupervised is used to extract the local morphological features of the image block, and the morphological features of Hydrangea macrophylla are obtained;

[0010] Text encoding branch: text information is labeled on the collected digital image, a text-image alignment deep neural network based on cross-modal contrast learning is used for weakly supervised training, and the association learning of image and text information is realized;

[0011] (4) Multi-modal feature fusion:

[0012] Image feature aggregation: a deep attention aggregation module is used to calculate the attention score of each image block, and the unified image feature is integrated;

[0013] Text feature aggregation: a deep attention aggregation module is used to calculate the attention score of the text keyword, and a unified text feature is formed;

[0014] Cross-modal fusion: adaptively aggregate text features and image features to obtain fusion representation features;

[0015] (5) Recognition of the crown layer growth of Hydrangea macrophylla:

[0016] Based on the fusion representation features obtained in step (4), the to-be-predicted or to-be-identified information is deleted from the text information during prediction or identification, and an output branch model is designed for different tasks, and the crown layer growth of Hydrangea macrophylla is predicted or identified.

[0017] The present application effectively fuses multi-modal information of two modes of image and text, and uses two complementary pre-training methods to extract different levels of visual representation: unsupervised pre-training for image block level feature recognition and weakly supervised pre-training for full picture mode recognition. The model can realize multi-task processing in one model, such as disease and pest identification, water condition identification and nutrient index prediction, etc., and achieve very high precision, serving plant maintenance work.

[0018] Among them, the morphological features of Hydrangea macrophylla obtained in step (3) include leaf texture, flower morphology and branch structure; the text information uses the labeled text provided by the fabric maintenance expert, including growth stage, environmental parameter, variety characteristic, maintenance record and pest description.

[0019] Further, step (5) uses a multi-task prediction network P-net calculation model to recognize the crown layer growth of Hydrangea macrophylla, and performs multi-task prediction through a multi-layer adaptive regression network (fully connected regression layer, variational autoencoder or Transformer decoder); wherein different tasks include classification tasks, identification tasks and regression tasks.

[0020] Among them, the crown layer growth of Hydrangea macrophylla includes the crown layer nutrient index of Hydrangea macrophylla, the growth trend score, the water state and the pest risk level.

[0021] Further, in step (1), a high-resolution camera (such as EOS90D) is used to take a digital image (such as a RAW format image, 6960x4640 pixels) of the largeleaf hydrangea plant.

[0022] In step (2), the specific method for pre-processing the image is as follows: the image data is divided into 256x256 pixel image blocks, and then the U-net is used to segment the image and remove the invalid background.

[0023] Further, in step (3), the specific implementation method of the image encoding branch is as follows: a CNN+Transformer image encoding network is used, and then pre-training is performed based on contrastive learning.

[0024] Among them, the contrastive learning adopts the contrastive learning paradigm of SimCLR or Moco v3, that is, different views are generated through data augmentation, the encoder is trained using the InfoNCE loss function, and data augmentation methods such as color perturbation, affine transformation, and local occlusion are used to improve the feature extraction capability of the image encoder in the present application.

[0025] Further, the specific implementation method of the text encoding branch in step (4) is as follows: the learning paradigm of the CLIP model is adopted, the similarity between the positive samples (correct text-image pairs) and the negative samples (unmatched text-image pairs) is calculated through cross-modal contrastive learning, so that the matching feature vectors are closer and the unmatched ones are farther apart, thereby enhancing the image and text matching capability.

[0026] Further, the specific implementation method of the above cross-modal fusion is as follows: the Transformer architecture is used for feature aggregation, and an initialized trainable parameter [FUS]Token is added to absorb global information, and the final [FUS]vector is used as the aggregated feature representation of the image or text; the unified text features and image features are spliced and transformed by a fully connected layer MLP to obtain the data fusion representation features.

[0027] More specifically, the above regression task includes quantitative prediction of the canopy nutrient index, when the quantitative prediction of the canopy nutrient index is performed, the nutrient information [FERTILIZER] in the text information data is deleted from the text information, and the nutrient information [FERTILIZER] is saved in the quantitative value The feature obtained by fusing the text features and image features The canopy nutrient index quantitative prediction branch model The regression loss is defined as follows:

[0028] Or

[0029] ;

[0030] Where, Represents the parameters of the quantitative prediction branch model of nutrient index; by optimizing the model parameters to minimize the above regression loss in all samples, the current nutrient index value is predicted by training model.

[0031] The classification task includes qualitative prediction of canopy nutrient index. When performing qualitative prediction of canopy nutrient index, the nutrient information [FERTILIZER] in the text information data is deleted from the text information, and the nutrient information [FERTILIZER] is qualitatively converted into [1, 2, 3, 4, 5] categories, which represent extreme lack, lack, moderate, abundant, and extremely abundant respectively. Then the qualitative code is converted into one-hot coding and saved in Middle; Features obtained by fusing text features and image features , qualitative prediction of branching model using canopy nutrient index The KL loss for regression is defined as follows:

[0032] ;

[0033] Among them, the canopy nutrient index qualitative prediction branch model The last layer is the Softmax classification layer. Represents the model parameter set of the qualitative prediction branch of the canopy nutrient index; by optimizing the model parameters to minimize the above KL loss on all samples, the current nutrient index is predicted by training Model of qualitative values.

[0034] The recognition task includes the recognition of various pests and diseases. Branch models are established separately and trained using KL divergence as the loss function. The last layer of the branch model uses the Sigmoid activation function.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] This invention intelligently identifies and predicts canopy growth in Hydrangea macrophylla, based on multimodal visual and textual information. It effectively integrates multimodal information from both image and textual modalities and utilizes two complementary pre-training methods to extract visual representations at different levels: unsupervised pre-training for block-level feature recognition and weakly supervised pre-training for full-image pattern recognition. This model can handle multiple tasks, such as pest and disease identification, water status identification, and nutrient index prediction, in a single model, achieving extremely high accuracy, significantly improving plant maintenance efforts.

[0037] The present invention uses a contrastive learning pre-training method for the visual branch of the multimodal coding network to improve the characterization capabilities of detecting leaf texture anomalies and identifying the developmental status of floral organs. The pre-trained network model using text-image alignment is fine-tuned through contrastive learning, making it more targeted at plant maintenance issues and contributing to better recognition and prediction results. Furthermore, based on the current large text-image model, the present invention, compared to the prior art that directly uses various parameter-annotated data such as plant growth conditions for model training, can achieve multimodal end-to-end text-image training and effectively learn on datasets with some missing information, effectively ensuring the model's prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of the intelligent identification of canopy growth of Hydrangea macrophylla based on visual and textual multimodal information in the present invention. DETAILED DESCRIPTION

[0039] The present invention is described in detail below with reference to specific embodiments.

[0040] Example 1

[0041] like Figure 1 As shown, the intelligent identification method of the canopy growth of Hydrangea macrophylla based on multimodal computer vision of the present invention includes the following steps:

[0042] Step 1: Data Collection

[0043] Under controlled lighting conditions, digital images of Hydrangea macrophylla plants (RAW format images, 6960 × 4640 pixels) were acquired using a high-resolution image acquisition device (e.g., EOS90D, Canon);

[0044] Text information recording the growing environment and growth conditions, including growth stages, environmental parameters, variety characteristics, maintenance records, and descriptions of pests and diseases.

[0045] Step 2: Image preprocessing

[0046] An image segmentation algorithm (U-net) is used to remove non-target background areas to remove invalid background information;

[0047] Generate multiple non-overlapping small-size image blocks through sliding window cropping (e.g. 256×256 pixels) for analysis by deep neural network models.

[0048] Step 3: Multimodal neural network construction

[0049] Image encoding branch (I-net): a deep neural network model I-net with unsupervised pre-training is adopted to extract the local morphological features of the image block, and the morphological features of the large-leaf hydrangea include leaf texture, flower morphology and branch structure features (i.e. visual features);

[0050] The image encoding branch I-net adopts a cnn+transformer image encoding network, and is then pre-trained based on contrast learning (such as SimCLR or Moco v3), uses an InfoNCE loss to train the encoder, and uses data enhancement methods such as color perturbation, affine transformation, and local occlusion to achieve the effect.

[0051] SimCLR training process: input image, generate two views through two different random data augmentations. Use a shared image encoder to extract image features. Map to a contrast learning space through a projection head (MLP). Calculate the contrast loss so that the features of different augmented versions of the same image are close, and the features of different images are far apart.

[0052] Text encoding branch (T-net): a text-image alignment deep neural network based on cross-modal contrast learning, which processes the text information (i.e. text features) annotated by plant maintenance experts, including growth stage, environmental parameters, variety characteristics, maintenance records, and pest and disease description. Through weakly supervised fine-tuning, the correlation learning between image and text information is realized;

[0053] The text encoding branch T-net adopts the CLIP model to realize efficient text-image alignment through cross-modal contrast learning, which belongs to the weakly supervised / self-supervised category. By calculating the similarity between positive samples (correct text-image pairs) and negative samples (incorrect text-image pairs), the matching feature vectors are closer, and the mismatching ones are farther apart. Specific method: use the labels (such as pest and disease, excessive nutrients, insufficient nutrients, etc.) of the images provided by the botanical garden maintenance experts through a large number of experiments and manual annotation for weakly supervised training, for example: "This is [CLS] the growth cycle [TIME] of the large-leaf hydrangea image, the water amount [WATER], the pest and disease [BUG], the growth condition [QUANLITY], and the fertilizer content [FERTILIZER]." The idea is to encode the text description (such as pest and disease, nutrient content, and water sufficiency rate) into a text embedding feature that matches the image. To ensure the effectiveness of the text feature representation, we use the text encoder of the CLIP model to extract the text embedding. The text encoder is a Transformer-based model and is pre-trained on a dataset of 400 million image-text pairs through multi-modal contrast learning. Align the image features with the text embeddings provided by experts' experience to enhance the model's supervision of the relevant patterns on the large-leaf hydrangea image.

[0054] Step four: multi-modal feature fusion Fusion-net

[0055] Image feature aggregation: the deep attention aggregation module is used to calculate the attention score of each image block, and the unified image feature is integrated;

[0056] Text feature aggregation: the deep attention aggregation module is used to calculate the attention score of the text keyword, and the unified text feature is formed;

[0057] Cross-modal fusion: adaptively aggregate text features and image features.

[0058] Among them, the feature fusion adopts a variable number of feature aggregation method based on the Transformer architecture, and uses [FUS]Token to absorb global information. The specific method is as follows: the feature obtained by each image block through I-net or the text feature obtained by each text information through T-net is regarded as 1 Token. When facing variable number of input features, a special Token ([FUS]Token) is introduced, which acts as the center of convergence of global features. First, add a [FUS]Token (initialized as a trainable parameter) in the input sequence. After passing through multiple Transformer layers, the [FUS]Token will absorb the information of the entire sequence through attention mechanism. The final [FUS]vector is used as the aggregated feature representation of the image or text. The text feature and the image feature after shape size adjustment are spliced and fused through a fully connected layer MLP transformation mode to obtain a data fusion representation feature. Mathematically, let T and F represent the text embedding feature (CLIP embedding) and the image embedding feature respectively. The text embedding is further passed through a multilayer perceptron (MLP) with two fully connected layers, and then merged and connected with the visual feature on the image branch; that is, fusion feature=MLP(T)+F. Through the pre-training process, the application learns to associate visual features with corresponding text descriptions, so as to improve the recognition effect by utilizing their cross-modal semantic correlation.

[0059] Step five: multi-task prediction (P-net)

[0060] The multi-task prediction P-net shares one backbone network, i.e., the fusion representation features obtained in step four. After the last feature layer, different output branches (Heads) are designed for different tasks. Each task branch may be: Classification Head: fully connected layer + Softmax; Recognition Head: CNN + LSTM or Transformer; Regression Head: fully connected layer + ReLU + Mean Squared Error (MSE), according to the different tasks.

[0061] Specific prediction or recognition tasks include:

[0062] Canopy nutrient index prediction (qualitative or quantitative value), pest type and severity identification, water status assessment (lack, suitable, excess), growth trend prediction (excellent, good, medium, poor).

[0063] Embodiment 2

[0064] This embodiment uses the method of embodiment 1 to quantitatively predict the canopy nutrient index.

[0065] The nutrient information [FERTILIZER] in the text data is deleted from the text information, and the nutrient information [FERTILIZER] is saved in the quantitative value The features obtained after fusing the pruned text features and image features , assuming that the quantitative prediction branch is , here represents the parameter of the nutrient index quantitative prediction branch model (regression task, Regression Head), then the regression loss can be defined as:

[0066] Or

[0067] ;

[0068] By optimizing the model parameters to minimize the above regression loss on all samples, a model method that can well predict the current nutrient index value based on other text information and image information can be trained.

[0069] Embodiment 3

[0070] This embodiment uses the method of embodiment 1 to quantitatively predict the canopy nutrient index.

[0071] When the canopy nutrient index prediction is qualitatively predicted (for example: extremely deficient, deficient, adequate, excessive, extremely excessive), it becomes a classification task (Classification Head) at this time. When the canopy nutrient index prediction is qualitatively classified, the text data about the nutrient information [FERTILIZER] is deleted from the text information, and the nutrient information [FERTILIZER] is qualitatively converted into a numerical value [1, 2, 3, 4, 5] category, and then the qualitative coding is converted into one-hot coding encoding value The reduced text features and image features are fused to obtain the features , assuming that the quantitative prediction branch is and the last layer is a Softmax classification layer, here represents the parameter set of the nutrient index qualitative prediction branch model, then the KL loss of regression can be defined as:

[0072] ;

[0073] By optimizing the model parameters to minimize the above KL loss on all samples, a model method that can well identify the current nutrient index value qualitative value based on other text information and image information can be trained.

[0074] Example 4

[0075] This embodiment adopts the method of Example 1 to identify plant diseases and insect pests.

[0076] When identifying plant diseases and insect pests, the qualitative prediction of the canopy nutrient index prediction described above is actually a classification task (Classification Head), and the categories are (none, A plant disease and insect pest, B plant disease and insect pest, C plant disease and insect pest, etc.). The last layer is a Sigmoid activation function, and the KL divergence is also used as the loss function (the loss function definition of the classification task is similar to the above canopy nutrient index prediction).

[0077] Therefore, in addition to the first category (no plant diseases and insect pests) which cannot appear at the same time as other categories, the above categories may have multiple categories corresponding to larger probability activation values, at which time multiple plant diseases and insect pests can be identified at the same time.

[0078] In addition, the type and severity of plant diseases and insect pests, water status assessment (deficient, suitable, excessive), and growth trend prediction (excellent, good, medium, poor) can all be considered for continuous quantitative prediction or discrete qualitative prediction and identification. When predicting, the predicted information is deleted from the text information as target data .

Claims

1. An intelligent identification method for canopy growth of Hydrangea macrophylla based on multimodal computer vision, characterized in that: This includes the following methods: (1) Data collection: Collect digital images of Hydrangea macrophylla plants; (2) Image preprocessing: remove the non-target background area from the acquired image and crop it to generate multiple non-overlapping small-sized image blocks; (3) Multimodal neural network construction: Image coding branch: An unsupervised pre-trained deep neural network model is used to extract local morphological features of image blocks and obtain the morphological features of Hydrangea macrophylla. Text encoding branch: This branch annotates the collected digital images with text information, uses a text-image alignment deep neural network based on cross-modal contrastive learning, and uses text information for weakly supervised training to achieve associative learning between image and text information. (4) Multimodal feature fusion: Image feature aggregation: The deep attention aggregation module is used to calculate the attention score of each image block and integrate it into a unified image feature; Text feature aggregation: Use the deep attention aggregation module to calculate the attention scores of text keywords to form a unified text feature; Cross-modal fusion: Adaptively aggregate text features and image features to obtain fused representation features; (5) Identification of canopy growth of bigleaf hydrangea: Based on the fusion representation features obtained in step (4), for different tasks, the information to be predicted or identified is deleted from the text information during prediction or identification, and used as the target data to design an output branch model to predict or identify the canopy growth of Hydrangea macrophylla; In the step (5), the canopy growth of the bigleaf hydrangea is predicted or identified, and a computational model of a multi-task prediction network P-net is used to perform multi-task prediction through a multi-layer adaptive regression network; wherein the different tasks targeted include classification tasks, recognition tasks, and regression tasks.

2. The intelligent identification method for canopy growth of Hydrangea macrophylla according to claim 1, characterized in that: The morphological characteristics of the large-leaf hydrangea obtained in step (3) include leaf texture, flower morphology and branch structure; the text information in step (3) includes growth stage, environmental parameters, variety characteristics, maintenance records and pest and disease descriptions.

3. The intelligent identification method for canopy growth of Hydrangea macrophylla according to claim 1 or 2, characterized in that: The canopy growth conditions of the bigleaf hydrangea include the canopy nutrient index, growth trend score, moisture status and pest and disease risk level of the bigleaf hydrangea.

4. The intelligent identification method for canopy growth of Hydrangea macrophylla according to claim 1 or 2, characterized in that: In step (1), the data is collected by taking a digital image of the Hydrangea macrophylla plant with a high-resolution camera. In step (2), the specific method for image preprocessing is as follows: the image data is divided into image blocks of 256×256 pixels, and then the image is segmented using U-net to remove invalid background.

5. The intelligent identification method for canopy growth of Hydrangea macrophylla according to claim 1 or 2, characterized in that: The specific implementation method of the image coding branch in step (3) is as follows: using a CNN+Transformer image coding network, and then performing pre-training based on contrastive learning.

6. The intelligent identification method for canopy growth of Hydrangea macrophylla according to claim 5, characterized in that: The contrastive learning adopts the contrastive learning paradigm of SimCLR or Moco v3, generates different views through data augmentation, and trains the image encoder using the InfoNCE loss function.

7. The intelligent identification method for canopy growth of Hydrangea macrophylla according to claim 1 or 2, characterized in that: The specific implementation method of the text encoding branch in step (4) is as follows: adopting the learning paradigm of the CLIP model, through cross-modal contrast learning, by calculating the similarity between positive samples and negative samples, the image and text matching ability is enhanced.

8. The intelligent identification method for canopy growth of Hydrangea macrophylla according to claim 1 or 2, characterized in that: The specific implementation method of the cross-modal fusion is as follows: using the Transformer architecture for feature aggregation, and adding an initialization trainable parameter [FUS] to absorb global information. The final [FUS] vector is used as the aggregated feature representation of the image or text; the unified text features and image features are spliced ​​and fused using the fully connected layer MLP transformation to obtain the data fusion representation feature.

9. The intelligent identification method for canopy growth of Hydrangea macrophylla according to claim 1 or 2, characterized in that: The regression task includes quantitative prediction of canopy nutrient index. When performing quantitative prediction of canopy nutrient index, the nutrient information [FERTILIZER] in the text information data is deleted from the text information, and the nutrient information [FERTILIZER] is saved in the quantitative value In the above example, the features obtained by fusion of text features and image features are , quantitative prediction of branching model based on canopy nutrient index The regression loss is defined as follows: or ； Where, Represents the parameters of the quantitative prediction branch model of nutrient index; by optimizing the model parameters to minimize the above regression loss in all samples, the current nutrient index value is predicted by training Model; The classification task includes qualitative prediction of canopy nutrient index. When performing qualitative prediction of canopy nutrient index, the nutrient information [FERTILIZER] in the text information data is deleted from the text information, and the nutrient information [FERTILIZER] is qualitatively converted into [1, 2, 3, 4, 5] categories, which respectively represent extreme lack, lack, moderate, abundant, and extremely abundant. Then the qualitative code is converted into one-hot coding and saved in Middle; Features obtained by fusing text features and image features , qualitative prediction of branching model using canopy nutrient index The KL loss for regression is defined as follows: ； Among them, the canopy nutrient index qualitative prediction branch model The last layer is the Softmax classification layer. Represents the model parameter set of the qualitative prediction branch of the canopy nutrient index; by optimizing the model parameters to minimize the above KL loss on all samples, the current nutrient index is predicted by training Models of qualitative values; The recognition task includes the recognition of multiple pests and diseases, which is trained by establishing branch models respectively and using KL divergence as the loss function; the last layer of the branch model uses the Sigmoid activation function.

Images