Plant time-series image contrastive learning method embedded with prior distance

Abstract

The application discloses a plant time-series image contrast learning method embedded with prior distance, comprising the following steps: reading plant time-series images, obtaining phenological information, and generating four kinds of image pairs; recording the corresponding prior distance of different kinds of image pairs; inputting the image pairs x and y into a contrast model, performing data enhancement to obtain images v1 and v2; extracting feature vectors h1 and h2 of the images v1 and v2 in an encoder; after extracting the feature vectors h1 and h2, a small neural network projection head is used to map the representation to a space where the contrast loss is applied; through two ways of hierarchical distance and classification distance, the prior distance of different kinds of image pairs and the actual distance of the corresponding vectors z1 and z2 are fused to obtain the contrast loss, and then training is performed. The application uses contrast learning to obtain pre-training weights specially used for crops, so that the self-supervised contrast learning method can be effectively applied to the pre-training of plant time-series images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of plant time series image processing of deep learning, and particularly relates to a plant time series image contrast learning method embedded with prior distance. BACKGROUND

[0002] Due to genetic differences and environmental influences, plants usually exhibit different characteristics and traits during growth and development. The shape, structure, size, color, and all other measurable external manifestations and internal physiological and biochemical characteristics of an organism determined by genotype and environment are referred to as plant phenotypes. Understanding the phenotypic characteristics and traits of plants is an important proposition in biological research. Without detailed phenotypic data, the complex effects of the genome and environmental factors on plant phenotypes cannot be fully understood. Traditional plant phenotype analysis mainly relies on manual measurement of various parameters, which has small analysis scale, low efficiency, and large errors.

[0003] The general research steps of plant time series images mainly include two stages of information extraction and time series modeling. In the information extraction stage, digital image processing methods, especially various deep learning methods such as image classification, object detection, and semantic segmentation, are used to extract phenotypic data from single image data. In the time series modeling stage, information is accumulated from the time dimension, and data from different growth stages are fused to establish a specific model, which can be jointly analyzed with other external factors.

[0004] In the process of collecting plant time series images, imaging equipment usually takes pictures of plants in a specific area at regular intervals, which can conveniently obtain plant time series images. However, in the process of training deep learning models for image information extraction, a large number of labeled images are usually required. Since plant images have many details such as flower and leaf edges, manually labeling the data set requires high cost, which requires the model to use fewer manually labeled data to obtain better training results.

[0005] To address the above situation, contrast learning using unlabeled data for pre-training is a way to achieve efficient training with labels, such as SimCLR, MoCo, SimSia, etc. The model learns representations by maximizing the consistency between different augmented views of the same data examples, and achieves good results when transferred to downstream tasks. Exploratory application research on plants and crops has also begun, such as plant phenotype segmentation, plant remote sensing, disease and pest monitoring, and seed classification.

[0006] However, the number of comparative learning researches in the field of plants is far less than that in other fields, and there is no research on self-supervised contrastive learning of plant time-series images. Plants grow slowly, and the images in the image sequence change little over time and are relatively similar. The semantic information of organs such as flowers, leaves and trunks in the image is simple. In the past contrastive learning model, although the image is easy to obtain, the labeling cost is high, and due to the semantic similarity of different images, the model will have difficulty in determining whether it is a positive sample of the same image after different data augmentation or a negative sample from different but similar images during contrastive training. SUMMARY

[0007] The purpose of the present application is to provide a plant time-series image contrastive learning method embedded with prior distance, which can be effectively applied to the pre-training of plant time-series images and has a wide application prospect in various computer vision plant phenotype researches.

[0008] To achieve the above-mentioned purpose, the present application provides a plant time-series image contrastive learning method embedded with prior distance, comprising:

[0009] Read the plant time-series image, obtain the phenological information, and generate four types of image pairs, i.e. image pairs of the same sequence and the same period, image pairs of the same sequence and different periods, image pairs of different sequences and the same period, and image pairs of different sequences and different periods;

[0010] Record the corresponding prior distance of different types of image pairs;

[0011] After inputting an image pair x and y into the contrastive model, data augmentation is performed to weaken the influence of color on model training, especially the green color of large-area leaves, so that the model can focus on higher-level semantic information other than color. Different views of the same image can also be generated in different epoch training, so that the model can better utilize similar images. The data-augmented images are v1 and v2.

[0012] The image v1 and v2 extract feature vectors h1 and h2 in the encoder; the encoder is arbitrarily determined according to the downstream task.

[0013] After extracting the feature vectors h1 and h2, a small neural network projection head is used to map the representation to the space where the contrastive loss is applied; the present application uses a 2-layer MLP with ReLU and BN layers to project the feature vectors hi and hj to 256-dimensional vectors z1 and z2; the projection head does not participate in the training of the downstream task.

[0014] In the calculation of contrastive loss, the present application proposes two ways of hierarchical distance and classification distance, which fuse the prior distance of different types of image pairs and the actual distance of corresponding vectors z1 and z2 to obtain the contrastive loss.

[0015] Compared with existing technologies, the technical solution adopted in this invention has the following advantages: This method transforms important domain knowledge into prior distances between image pairs and performs contrastive learning pre-training. This self-supervised contrastive learning method can be effectively applied to the pre-training of plant time-series images and has broad application prospects in various computer vision plant phenotyping studies. Attached Figure Description

[0016] Figure 1 Flowchart of a plant temporal image contrast learning method for embedding prior distance;

[0017] Figure 2 This is a schematic diagram illustrating the division of phenological periods and image extraction.

[0018] Figure 3 A diagram illustrating the process of obtaining the distance metric and the contrastive loss function;

[0019] Figure 4 This is a schematic diagram of the pre-training and transfer process. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit the application; that is, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0021] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0022] Example 1

[0023] like Figure 1 As shown, this application provides a plant temporal image contrast learning method embedding prior distance, specifically including:

[0024] Phenological information is obtained from plant images. For n image sequences of a given plant, 3–5 example sequences are randomly selected. For each example sequence, budding is set as the baseline time day0. By manually interpreting the image sequences, the start time and duration of different phenological stages can be obtained. The average value of each time point in the example sequences is calculated. Since a specific plant has a relatively fixed annual growth cycle, this average value can approximately represent the phenological stages of all n image sequences.

[0025] From the image of the good phenological phase division, the required image is extracted. Phenological change is a process, at the end of a phenological phase and at the beginning of the next phenological phase, the corresponding image may contain similar semantic information. In order to realize the automatic and accurate extraction of different phenological period images, and maximize the difference of image semantics during different phenological periods, the image at the junction of adjacent phenological periods is discarded, and only the image far from the time critical point is selected and recorded.

[0026] The images are paired two by two, that is, the Cartesian product of the image set itself. The obtained image pair can be recorded as same sequence same period, same sequence different period, different sequence same period, and different sequence different period according to the sequence and phenological phase of the two images, and saved as label by One-hot encoding.

[0027] According to the prior distance of different kinds of image pairs, the contrast loss is calculated, and the hierarchical distance and the classification distance are proposed. The hierarchical distance is as shown in Figure 3 (a), and specifically:

[0028] For l kind of image pairs, first define a distance coefficient k l to represent the relative distance between the image pairs of this kind. For any image pair x and y, the distance p xy can be obtained:

[0029]

[0030] a1, a2, a3, a4 are the original labels of the image pair. After this processing, the prior distance p xy of any image pair x and y is marked as a number between 0 and 1, which can be regarded as the probability between the same meaning image pairs. xy

[0031] For the vectors z1 and z2 of the image pair x and y, the loss value loss is calculated as:

[0032]

[0033] In the formula, Simz1,z2 is the similarity of z1 and z2. The function can be regarded as calculating the similarity of z1 and z2, and after Sigmoid, the cross entropy with the probability represented by p xy is calculated. τ is the temperature coefficient, which can be used to adjust the distribution of similarity. The similarity can be measured by negative cosine similarity or Euclidean distance. When negative cosine similarity is used:

[0034]

[0035] The Euclidean distance is the l2 norm: ​

[0036] Sim(z1,z2) = Euc(z1,z2) = ||z1-z2||2

[0037] For each mini-batch, if the batch size is n, the contrastive loss loss of the batch is:

[0038]

[0039] The classification distance is as shown in (b), specifically: the distance information of the contrastive model is implicitly mapped to a full connection layer, and the loss is calculated directly with the classification of different image pairs. Figure 3

[0040] When calculating the Euclidean distance of z1 and z2, it can be further rewritten as:

[0041]

[0042] After the Euclidean distance is calculated, a number is obtained, which is not conducive to connection to the full connection layer for classification. When calculating the contrastive loss of the classification distance, z1 and z2 are directly subtracted by bit, that is:

[0043] e = z1-z2

[0044] e is a vector with the same dimension as z1 and z2. It can be found that the process of calculating e is the core step of calculating the Euclidean distance of z1 and z2, which makes e contain the prior distance information of z1 and z2. Linear projection of e to a full connection layer t with 4 nodes, after softmax processing, the output o can be obtained. That is:

[0045] t = eW

[0046] o = Softmax(t)

[0047] Cross-entropy is used to calculate the error between the category information in o and the prior distance information in the label category of the image pair, that is:

[0048]

[0049] After the pre-training of the contrastive learning is completed, the obtained weight can be migrated to various supervised tasks downstream. Based on the pre-trained weight, a small amount of labeled plant image is supervised trained. And the trained semantic segmentation network is used as an information extraction model, the semantic segmentation result is recorded in the time dimension, and a plant growth model is established.

[0050] As shown in Figure 4 ​As shown, the U-Net semantic segmentation network is used as the information extraction model to perform pixel-level segmentation on the branches, flowers, leaves, fruits and background regions in the plant image. The U-Net network structure is a classical Encoder-Decoder structure, and various main stems can be used as the Encoder. The Encoder network can obtain five preliminary effective feature layers. The Decoder network uses deconvolution to up-sample the five feature layers and perform feature fusion, and can obtain an effective feature layer that fuses all the features. Each feature point of the finally obtained feature layer is classified, and the semantic segmentation result can be obtained.

[0051] The Kalman filter is used to reduce the influence of environmental light, climate, production activities and the like, remove the noise and interference in the growth model system, and obtain more stable observation data.

[0052] The foregoing description of specific exemplary embodiments of the application is intended to be illustrative only and is not intended to limit the application to the precise forms described. Many modifications and variations are possible in light of the above teachings without departing from the spirit or essential characteristics of the present application. The exemplary embodiments are chosen and described in order to explain the principles of the application and its practical application and to allow others skilled in the art to understand the application for various exemplary embodiments with various modifications as are suited to the particular use contemplated. It is intended that the scope of the application be defined by the claims and their equivalents.

Claims

1. A plant timing image contrast learning method embedded with prior distance, characterized in that, The method comprises the following steps: reading plant timing images to obtain phenological information and generate four types of image pairs, i.e., same-sequence same-period, same-sequence different-period, different-sequence same-period and different-sequence different-period image pairs; recording the corresponding prior distances of different types of image pairs; after inputting an image pair x and y into a contrast model, data augmentation is performed to obtain images v1 and v2; different views are generated for the same image in different epoch training; the images v1 and v2 are extracted into feature vectors h1 and h2 in an encoder; after extracting the feature vectors h1 and h2, a neural network projection head is used to map the representation to a space where the contrast loss is applied, to obtain vectors z1 and z2; by means of hierarchical distance and classification distance, the prior distances of different types of image pairs and the actual distances of corresponding vectors z1 and z2 are fused to obtain a contrast loss; The hierarchical distance is specifically: first, a distance coefficient is defined to represent the relative distance between the image pairs; for image pairs x and y, the distance is obtained as follows: :​ For image pairs original label, after this processing, the prior distance of any image pair x and y Marked as a number between [0, 1], will As the probability between the same meaning between samples for the vectors z1 and z2 of the image pair x and y, the loss value loss is calculated as: where is the similarity of z1 and z2; the above formula is considered to be the similarity of z1 and z2, and after the Sigmoid, the cross-entropy with the probability represented by is calculated; is the temperature coefficient, used to adjust the distribution of the similarity; the similarity is measured by negative cosine distance or Euclidean distance; when the negative cosine distance is used: The Euclidean distance is the l2 norm: for each mini-batch, if the batch size is n, the contrast loss loss of the mini-batch is: 。 2. The plant timing image contrast learning method with embedded prior distance according to claim 1, characterized in that, the classification distance is specifically that the distance information of the contrast model is implicitly mapped to a fully connected layer, and the loss is directly calculated with the classification of different image pairs; when calculating the Euclidean distance of z1 and z2, the following formula is rewritten as: When calculating the contrastive loss of the classification distance, z1 and z2 are directly subtracted bit by bit, that is: The parameter e is a vector with the same dimension as z1 and z2. The process of calculating e is the core step of calculating the Euclidean distance of z1 and z2, which makes e contain the prior distance information of z1 and z2.

3. The plant timing image contrast learning method with embedded prior distance according to claim 2, characterized in that, e is linearly projected to a 4-node fully connected layer t, and the output o is obtained after softmax processing; that is: cross entropy is used to calculate the error between the category information in o and the prior distance information in the label category of the image pair, that is: 。 4. The plant time-series image contrastive learning method with embedded prior distance according to claim 1, characterized in that, a U-Net semantic segmentation network is used as an information extraction model to perform pixel-level segmentation on the branch, flower, leaf, fruit and background regions in the plant image; the U-Net semantic segmentation network has an Encoder-Decoder structure, and the Encoder network obtains five feature layers; the Decoder network uses deconvolution to upsample the five feature layers and perform feature fusion to obtain an effective feature layer that integrates all features; by classifying each feature point of the effective feature layer, the semantic segmentation result can be obtained.

Images