Hydrangea macrophylla canopy growth condition intelligent identification method based on multi-modal computer vision

Abstract

The invention discloses a multimodal computer vision-based hydrangea macrophylla canopy growth condition intelligent identification method, which comprises the following steps of: acquiring a digital image of a hydrangea macrophylla plant, preprocessing the digital image, and cutting the preprocessed digital image into image blocks; extracting local morphological features of the image blocks by adopting an unsupervised pre-trained deep neural network model; on the basis of a text-image alignment deep neural network of cross-modal comparative learning, weak supervision training is carried out by using the labeled text information, and associated learning of the image and the text information is realized; performing multi-modal feature fusion on the text features and the image features; based on the fusion representation features, aiming at different tasks, designing an output branch model, and predicting or identifying the growth condition of the hydrangea macrophylla canopies. According to the method, multi-modal information of two modes of images and texts is effectively fused, visual representations of different levels are extracted by utilizing two complementary pre-training methods, multi-task processing is realized, and extremely high precision is achieved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method for identifying and predicting the canopy growth of large-leaf hydrangeas. Background Art

[0002] Bigleaf hydrangea (Hydrangea macrophylla) is a popular flowering shrub with numerous varieties, and we selected varieties with high value for garden applications. Spring is the golden growth period for hydrangeas and a critical stage in determining the annual flowering output. The warm and humid spring environment is conducive to the breeding of pests such as aphids and spider mites, so preventative measures are necessary. Hydrangeas are typical "foodies," and their demand for nutrients increases dramatically after new buds sprout in spring. If nutrients are insufficient at this time, the new buds will become thin and weak, and the flowering output will be greatly reduced. Therefore, by identifying and predicting the growth status of bigleaf hydrangeas, targeted maintenance can be implemented.

[0003] In the existing technology, plant growth status monitoring is mostly based on computer vision methods to identify plant growth conditions and thus judge plant growth status, but it has the following disadvantages: 1. Simple image processing results in low recognition accuracy; 2. The recognition process requires a large amount of various labeled data such as plant growth parameters to train the model, which is a large workload and difficult to guarantee the model effect; 3. Based on the preset neural network model, it cannot be effectively combined with the text information recorded by plant maintenance experts for training, and the recognition / prediction accuracy is still difficult to guarantee. Summary of the Invention

[0004] The purpose of the present invention is to solve the defects in the prior art and provide an intelligent identification method for the canopy growth of bigleaf hydrangea that effectively associates the visual features of the plant with text information to improve the prediction accuracy.

[0005] To achieve the above objectives, the present invention provides an intelligent identification method for the canopy growth of Hydrangea macrophylla based on multimodal computer vision, including the following methods: (1) Data collection: Collect digital images of Hydrangea macrophylla plants; (2) Image preprocessing: remove the non-target background area from the acquired image and crop it to generate multiple non-overlapping small-sized image blocks; (3) Multimodal neural network construction: Image coding branch: An unsupervised pre-trained deep neural network model is used to extract local morphological features of image blocks and obtain the morphological features of Hydrangea macrophylla. Text encoding branch: This branch annotates the collected digital images with text information, uses a text-image alignment deep neural network based on cross-modal contrastive learning, and uses text information for weakly supervised training to achieve associative learning between image and text information. (4) Multimodal feature fusion: Image feature aggregation: The deep attention aggregation module is used to calculate the attention score of each image block and integrate it into a unified image feature; Text feature aggregation: Use the deep attention aggregation module to calculate the attention scores of text keywords to form a unified text feature; Cross-modal fusion: Adaptively aggregate text features and image features to obtain fused representation features; (5) Identification of canopy growth of bigleaf hydrangea: Based on the fusion representation features obtained in step (4), for different tasks, the information to be predicted or identified is deleted from the text information during prediction or identification, and used as the target data to design an output branch model to predict or identify the canopy growth of Hydrangea macrophylla.

[0006] This invention effectively integrates multimodal information from both image and text formats and utilizes two complementary pre-training methods to extract visual representations at different levels: unsupervised pre-training for block-level feature recognition and weakly supervised pre-training for full-image pattern recognition. This model can handle multiple tasks in a single model, such as pest and disease identification, water status recognition, and nutrient index prediction, achieving extremely high accuracy and benefiting plant maintenance.

[0007] The morphological characteristics of the large-leaf hydrangea obtained in step (3) include leaf texture, flower morphology, and branch structure; the text information uses the annotated text provided by a textile maintenance expert, including growth stages, environmental parameters, variety characteristics, maintenance records, and descriptions of pests and diseases.

[0008] Furthermore, step (5) uses a computational model of a multi-task prediction network P-net to identify the canopy growth of Hydrangea macrophylla, and performs multi-task prediction through a multi-layer adaptive regression network (fully connected regression layer, variational autoencoder or Transformer decoder); wherein different tasks include classification tasks, recognition tasks and regression tasks.

[0009] Among them, the canopy growth conditions of bigleaf hydrangea include the canopy nutrient index, growth trend score, water status and pest and disease risk level of bigleaf hydrangea.

[0010] Furthermore, in step (1), a high-resolution camera (such as EOS90D) is used to capture a digital image of the Hydrangea macrophylla plant (such as a RAW format image, 6960×4640 pixels); The specific method of image preprocessing in step (2) is as follows: the image data is divided into image blocks of 256×256 pixels, and then the U-net is used to segment the image and remove the invalid background.

[0011] Furthermore, the specific implementation method of the image coding branch in step (3) is as follows: adopt the CNN+Transformer image coding network, and then perform pre-training based on contrastive learning.

[0012] Among them, contrastive learning adopts the contrastive learning paradigm of SimCLR or Moco v3, that is, generating different views through data enhancement, using the InfoNCE loss function to train the encoder, and using data enhancement methods such as color perturbation, affine transformation, and local occlusion to improve the feature extraction capability of the image encoder in the present invention.

[0013] Furthermore, the specific implementation method of the text encoding branch in step (4) is as follows: adopting the learning paradigm of the CLIP model, through cross-modal contrastive learning, by calculating the similarity between positive samples (correct text-image pairs) and negative samples (mismatched text-image pairs), the matching feature vectors are made closer and the mismatched ones are made farther, thereby enhancing the image and text matching ability.

[0014] Furthermore, the specific implementation method of the above cross-modal fusion is as follows: using the Transformer architecture for feature aggregation, and adding an initialized trainable parameter [FUS]Token to absorb global information, the final [FUS] vector is used as the aggregated feature representation of the image or text; the unified text features and image features are spliced ​​and fused using the fully connected layer MLP transformation to obtain the data fusion representation feature.

[0015] More specifically, the above regression task includes quantitative prediction of canopy nutrient index. When performing quantitative prediction of canopy nutrient index, the nutrient information [FERTILIZER] in the text information data is deleted from the text information, and the nutrient information [FERTILIZER] is saved in the quantitative value In the above example, the features obtained by fusion of text features and image features are , quantitative prediction of branching model based on canopy nutrient index The regression loss is defined as follows: or ; Where, Represents the parameters of the quantitative prediction branch model of the nutrient index; by optimizing the model parameters to minimize the above regression loss on all samples, the current nutrient index value is predicted by training model.

[0016] The classification task includes qualitative prediction of canopy nutrient index. When performing qualitative prediction of canopy nutrient index, the nutrient information [FERTILIZER] in the text information data is deleted from the text information, and the nutrient information [FERTILIZER] is qualitatively converted into [1, 2, 3, 4, 5] categories, which represent extreme lack, lack, moderate, abundant, and extremely abundant respectively. Then the qualitative code is converted into one-hot coding and saved in Middle; Features obtained by fusing text features and image features , qualitative prediction of branching model using canopy nutrient index The KL loss for regression is defined as follows: ; Among them, the canopy nutrient index qualitative prediction branch model The last layer is the Softmax classification layer. Represents the model parameter set of the qualitative prediction branch of the canopy nutrient index; by optimizing the model parameters to minimize the above KL loss on all samples, the current nutrient index is predicted by training Model of qualitative values.

[0017] The recognition task includes the recognition of various pests and diseases. Branch models are established separately and trained using KL divergence as the loss function. The last layer of the branch model uses the Sigmoid activation function.

[0018] Compared with the prior art, the present invention has the following advantages: This invention intelligently identifies and predicts canopy growth in Hydrangea macrophylla, based on multimodal visual and textual information. It effectively integrates multimodal information from both image and textual modalities and utilizes two complementary pre-training methods to extract visual representations at different levels: unsupervised pre-training for block-level feature recognition and weakly supervised pre-training for full-image pattern recognition. This model can handle multiple tasks, such as pest and disease identification, water status identification, and nutrient index prediction, in a single model, achieving extremely high accuracy, significantly improving plant maintenance efforts.

[0019] The present invention uses a contrastive learning pre-training method for the visual branch of the multimodal coding network to improve the characterization capabilities of detecting leaf texture anomalies and identifying the developmental status of floral organs. The pre-trained network model using text-image alignment is fine-tuned through contrastive learning, making it more targeted at plant maintenance issues and contributing to better recognition and prediction results. Furthermore, based on the current large text-image model, the present invention, compared to the prior art that directly uses various parameter-annotated data such as plant growth conditions for model training, can achieve multimodal end-to-end text-image training and effectively learn on datasets with some missing information, effectively ensuring the model's prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of the intelligent identification of canopy growth of Hydrangea macrophylla based on visual and textual multimodal information in the present invention. DETAILED DESCRIPTION

[0021] The present invention is described in detail below with reference to specific embodiments.

[0022] Example 1

[0023] like Figure 1 As shown, the intelligent identification method of the canopy growth of Hydrangea macrophylla based on multimodal computer vision of the present invention includes the following steps: Step 1: Data Collection Under controlled lighting conditions, digital images of Hydrangea macrophylla plants (RAW format images, 6960 × 4640 pixels) were acquired using a high-resolution image acquisition device (e.g., EOS90D, Canon); Text information recording the growing environment and growth conditions, including growth stages, environmental parameters, variety characteristics, maintenance records, and descriptions of pests and diseases.

[0024] Step 2: Image preprocessing An image segmentation algorithm (U-net) is used to remove non-target background areas to remove invalid background information; Generate multiple non-overlapping small-size image blocks through sliding window cropping (e.g. 256×256 pixels) for analysis by deep neural network models.

[0025] Step 3: Multimodal neural network construction Image encoding branch (I-net): using an unsupervised pre-trained deep neural network model I-net to extract local morphological features of image blocks, obtaining morphological features of the large-leaf hydrangea, including leaf texture, flower morphology, and branch structure features (i.e., visual features); The image encoding branch I-net uses a CNN+Transformer image encoding network, and then pre-trains based on contrastive learning (such as SimCLR or Moco v3). The encoder is trained using InfoNCE loss and data augmentation methods such as color perturbation, affine transformation, and local occlusion are used. SimCLR training process: An image is input and two views are generated using two different random data augmentations. A shared image encoder is used to extract image features. A projection head (MLP) is used to map the image to a contrastive learning space. A contrastive loss is calculated to ensure that features of different augmented versions of the same image are close and features of different images are far apart.

[0026] Text encoding branch (T-net): A text-image alignment deep neural network based on cross-modal contrastive learning processes textual information (i.e., text features) annotated by plant care experts, including growth stages, environmental parameters, variety characteristics, care records, and pest and disease descriptions. Through weakly supervised fine-tuning, it achieves the association learning between image and text information. This text encoding branch, T-net, employs the CLIP model to achieve efficient text-image alignment through cross-modal contrastive learning. This approach falls under the weakly supervised / self-supervised category. By calculating the similarity between positive examples (correct text-image pairs) and negative examples (incorrect text-image pairs), it brings matching feature vectors closer together and mismatches further apart. The specific method utilizes weakly supervised training using image labels (such as pest and disease, nutrient overload, and nutrient underload) provided by botanical garden conservation experts through extensive experiments and manual analysis. For example, "This is an image of a [CLS] Hydrangea macrophylla during its growth cycle [TIME], with [WATER] water level, [BUG] pests and diseases, growth status [QUANLITY], and fertilizer content [FERTILIZER]." The idea is to encode text descriptions (such as pest and disease, nutrient content, and water adequacy) into text embedding features that match the image. To ensure the effectiveness of the text feature representation, we utilize the CLIP model's text encoder to extract text embeddings. The text encoder is a Transformer-based model pre-trained on a dataset of 400 million image-text pairs. Through multimodal contrastive learning, it aligns image features with text embeddings given by expert experience, enhancing the model's supervision of relevant patterns of various parts on the Hydrangea macrophylla image.

[0027] Step 4: Multimodal feature fusion Fusion-net Image feature aggregation: The deep attention aggregation module is used to calculate the attention score of each image block and integrate it into a unified image feature; Text feature aggregation: Use the deep attention aggregation module to calculate the attention scores of text keywords to form a unified text feature; Cross-modal fusion: adaptively aggregating text features and image features.

[0028] Feature fusion employs a variable-number feature aggregation method based on the Transformer architecture and uses the [FUS] token to absorb global information. The specific method is as follows: the features obtained from each image block through the I-net or the text features obtained from each text element through the T-net are treated as a token. When faced with a variable number of input features, a special token (the [FUS] token) is introduced to serve as a global feature aggregation center. First, a [FUS] token (initialized as a trainable parameter) is added to the input sequence. After passing through multiple Transformer layers, this [FUS] token absorbs information from the entire sequence through an attention mechanism. The final [FUS] vector serves as the aggregated feature representation of the image or text. The resized text features and image features are concatenated and fused using a fully connected layer (MLP) transformation to obtain a fused data representation. Mathematically, let T and F denote the text embedding feature (CLIP embedding) and image embedding feature, respectively. The text embedding is further passed through a multi-layer perceptron (MLP) with two fully connected layers and then merged with the visual features from the image branch; that is, the fusion feature = MLP(T) + F. Through the pre-training process, the present invention learns to associate visual features with corresponding text descriptions, thereby utilizing their cross-modal semantic relevance to improve recognition effects.

[0029] Step 5: Multi-task prediction (P-net) The multi-task prediction P-net shares a common backbone network, based on the fused representation features obtained in step 4. After the final feature layer, different output branches (heads) are designed for different tasks. Each task branch, depending on its task, may be: Classification Head: Fully Connected Layer + Softmax; Recognition Head: CNN + LSTM or Transformer; Regression Head: Fully Connected Layer + ReLU + Mean Squared Error (MSE).

[0030] Specific prediction or recognition tasks include: Canopy nutrient index prediction (qualitative or quantitative value), identification of pest and disease types and severity, water status assessment (deficient, suitable, excessive), growth trend prediction (excellent, good, fair, poor).

[0031] Example 2

[0032] This example uses the method of Example 1 to quantitatively predict the canopy nutrient index.

[0033] Delete the nutrient information [FERTILIZER] from the text data and save the nutrient information [FERTILIZER] in the quantified value In the above example, the features are obtained by fusing the deleted text features and image features. , assuming that the quantitative prediction branch is ,here represents the parameters of the nutrient index quantitative prediction branch model (regression task, Regression Head), then the regression loss can be defined as: or ; By optimizing the model parameters to minimize the above regression loss on all samples, we can train a model that can predict the current nutrient index value based on other text information and image information. model method.

[0034] Example 3

[0035] This example uses the method of Example 1 to perform qualitative prediction of the canopy nutrient index.

[0036] When the canopy nutrient index is predicted qualitatively (e.g., extremely lacking, lacking, moderate, abundant, extremely abundant), it becomes a classification task (Classification Head). First, when the canopy nutrient index is predicted qualitatively, the nutrient information [FERTILIZER] in the text data is deleted from the text information, and the nutrient information [FERTILIZER] is converted to a qualitative value [1, 2, 3, 4, 5] class, and then the qualitative code is converted to a one-hot coding code to save the value. The features obtained by fusion of deleted text features and image features , assuming that the quantitative prediction branch is And the last layer is the Softmax classification layer, here represents the parameter set of the qualitative prediction branch model of the nutrient index, then the KL loss of regression can be defined as: ;

[0037] By optimizing the model parameters to minimize the above KL loss on all samples, we can train a model that can well identify the current nutrient index value based on other text information and image information. Model methods for qualitative values.

[0038] Example 4

[0039] This embodiment adopts the method of embodiment 1 to identify pests and diseases.

[0040] When performing pest and disease identification, similar to the canopy nutrient index prediction described above, qualitative prediction is actually a classification task (Classification Head), with categories (absent, present pest A, present pest B, present pest C, etc.). The final layer uses a Sigmoid activation function, and the KL divergence is also used as the loss function (the loss function definition for the classification task is similar to that for the canopy nutrient index prediction described above).

[0041] Therefore, except for the first category (no pests and diseases), which cannot appear at the same time with other categories, there may be multiple categories in the above categories with corresponding large probability activation values. At this time, multiple pests and diseases can be identified simultaneously.

[0042] In addition, continuous quantitative prediction or discrete qualitative prediction and identification can be considered for the identification of pest and disease types and severity; water status assessment (lack, suitable, excess); and growth trend prediction (excellent, good, medium, poor). During the prediction, the predicted information is deleted from the text information and used as the target data. That's it.

Claims

1. An intelligent identification method for canopy growth of Hydrangea macrophylla based on multimodal computer vision, characterized in that: This includes the following methods: (1) Data collection: Collect digital images of Hydrangea macrophylla plants; (2) Image preprocessing: remove the non-target background area from the acquired image and crop it to generate multiple non-overlapping small-sized image blocks; (3) Multimodal neural network construction: Image coding branch: An unsupervised pre-trained deep neural network model is used to extract local morphological features of image blocks and obtain the morphological features of Hydrangea macrophylla. Text encoding branch: This branch annotates the collected digital images with text information, uses a text-image alignment deep neural network based on cross-modal contrastive learning, and uses text information for weakly supervised training to achieve associative learning between image and text information. (4) Multimodal feature fusion: Image feature aggregation: The deep attention aggregation module is used to calculate the attention score of each image block and integrate it into a unified image feature; Text feature aggregation: Use the deep attention aggregation module to calculate the attention scores of text keywords to form a unified text feature; Cross-modal fusion: Adaptively aggregate text features and image features to obtain fused representation features; (5) Identification of canopy growth of bigleaf hydrangea: Based on the fusion representation features obtained in step (4), for different tasks, the information to be predicted or identified is deleted from the text information during prediction or identification, and used as the target data to design an output branch model to predict or identify the canopy growth of Hydrangea macrophylla; In the step (5), the canopy growth of the bigleaf hydrangea is predicted or identified, and a computational model of a multi-task prediction network P-net is used to perform multi-task prediction through a multi-layer adaptive regression network; wherein the different tasks targeted include classification tasks, recognition tasks, and regression tasks.

2. The intelligent identification method for canopy growth of Hydrangea macrophylla according to claim 1, characterized in that: The morphological characteristics of the large-leaf hydrangea obtained in step (3) include leaf texture, flower morphology and branch structure; the text information in step (3) includes growth stage, environmental parameters, variety characteristics, maintenance records and pest and disease descriptions.

3. The intelligent identification method for canopy growth of Hydrangea macrophylla according to claim 1 or 2, characterized in that: The canopy growth conditions of the bigleaf hydrangea include the canopy nutrient index, growth trend score, moisture status and pest and disease risk level of the bigleaf hydrangea.

4. The intelligent identification method for canopy growth of Hydrangea macrophylla according to claim 1 or 2, characterized in that: In step (1), the data is collected by taking a digital image of the Hydrangea macrophylla plant with a high-resolution camera. In step (2), the specific method for image preprocessing is as follows: the image data is divided into image blocks of 256×256 pixels, and then the image is segmented using U-net to remove invalid background.

5. The intelligent identification method for canopy growth of Hydrangea macrophylla according to claim 1 or 2, characterized in that: The specific implementation method of the image coding branch in step (3) is as follows: using a CNN+Transformer image coding network, and then performing pre-training based on contrastive learning.

6. The intelligent identification method for canopy growth of Hydrangea macrophylla according to claim 5, characterized in that: The contrastive learning adopts the contrastive learning paradigm of SimCLR or Moco v3, generates different views through data augmentation, and trains the image encoder using the InfoNCE loss function.

7. The intelligent identification method for canopy growth of Hydrangea macrophylla according to claim 1 or 2, characterized in that: The specific implementation method of the text encoding branch in step (4) is as follows: adopting the learning paradigm of the CLIP model, through cross-modal contrast learning, by calculating the similarity between positive samples and negative samples, the image and text matching ability is enhanced.

8. The intelligent identification method for canopy growth of Hydrangea macrophylla according to claim 1 or 2, characterized in that: The specific implementation method of the cross-modal fusion is as follows: using the Transformer architecture for feature aggregation, and adding an initialization trainable parameter [FUS] to absorb global information. The final [FUS] vector is used as the aggregated feature representation of the image or text; the unified text features and image features are spliced ​​and fused using the fully connected layer MLP transformation to obtain the data fusion representation feature.

9. The intelligent identification method for canopy growth of Hydrangea macrophylla according to claim 1 or 2, characterized in that: The regression task includes quantitative prediction of canopy nutrient index. When performing quantitative prediction of canopy nutrient index, the nutrient information [FERTILIZER] in the text information data is deleted from the text information, and the nutrient information [FERTILIZER] is saved in the quantitative value In the above example, the features obtained by fusion of text features and image features are , quantitative prediction of branching model based on canopy nutrient index The regression loss is defined as follows: or ； Where, Represents the parameters of the quantitative prediction branch model of the nutrient index; by optimizing the model parameters to minimize the above regression loss on all samples, the current nutrient index value is predicted by training Model; The classification task includes qualitative prediction of canopy nutrient index. When performing qualitative prediction of canopy nutrient index, the nutrient information [FERTILIZER] in the text information data is deleted from the text information, and the nutrient information [FERTILIZER] is qualitatively converted into [1, 2, 3, 4, 5] categories, which respectively represent extreme lack, lack, moderate, abundant, and extremely abundant. Then the qualitative code is converted into one-hot coding and saved in Middle; Features obtained by fusing text features and image features , qualitative prediction of branching model using canopy nutrient index The KL loss for regression is defined as follows: ； Among them, the canopy nutrient index qualitative prediction branch model The last layer is the Softmax classification layer. Represents the model parameter set of the qualitative prediction branch of the canopy nutrient index; by optimizing the model parameters to minimize the above KL loss on all samples, the current nutrient index is predicted by training Models of qualitative values; The recognition task includes the recognition of multiple pests and diseases, which is trained by establishing branch models respectively and using KL divergence as the loss function; the last layer of the branch model uses the Sigmoid activation function.

Images