Plant leaf hyperspectral trait estimation method, device, equipment and storage medium

Abstract

A plant leaf hyperspectral trait estimation method, device, equipment and storage medium are disclosed, and belong to the technical field of plant trait estimation. The method comprises the following steps: obtaining a plant leaf hyperspectral dataset; based on the plant leaf hyperspectral dataset, a trait estimation model is pre-trained in a self-supervised learning manner; in the process of pre-training the trait estimation model in the self-supervised learning manner, after data in the plant leaf hyperspectral dataset is embedded into a high-dimensional feature sequence, self-supervised learning is performed based on the high-dimensional feature sequence; the high-dimensional feature sequence comprises a plurality of high-dimensional features, and the high-dimensional feature is a feature in a hidden space; for a target trait estimation task, the pre-trained trait estimation model is fine-tuned; and the fine-tuned trait estimation model is used for estimating the target trait based on the hyperspectral data of the plant leaf. The method can accurately perform plant leaf hyperspectral trait estimation and can be widely applied to various trait estimation.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of plant trait estimation, and in particular to a plant leaf hyperspectral trait estimation method, device, equipment and storage medium. BACKGROUND

[0002] With the increasing global climate change and environmental protection demand, remote sensing monitoring of plant leaf traits (such as chlorophyll content, nitrogen content, water content, etc.) is of great significance in ecosystem research and agroforestry management. Hyperspectral remote sensing technology has the potential to monitor multiple traits simultaneously due to its high spectral resolution, which can extract fine features of plant leaves.

[0003] The plant leaf hyperspectral trait estimation method in the related art usually relies on a supervised learning mechanism to extract features and make predictions by modeling the relationship between spectrum and trait. In order to solve the problem that data-driven models can only be applied to single trait problems under limited conditions and to avoid overfitting, some researchers try to use transfer learning strategies in the hope of applying data-driven models to the prediction of multiple traits.

[0004] However, plant leaf hyperspectral trait estimation based on supervised learning requires a large amount of labeled data, and the cost of obtaining labeled data is high. The effectiveness of the transfer learning method is highly dependent on the correlation between traits. When the correlation between traits is weak, transfer learning may not effectively improve the prediction performance, and may even lead to performance degradation. That is, the plant leaf hyperspectral trait estimation method in the related art cannot have both accuracy and versatility. SUMMARY

[0005] The present disclosure provides a plant leaf hyperspectral trait estimation method, device, equipment and storage medium, which can accurately estimate plant leaf hyperspectral traits and be widely applied to multiple trait estimation. The technical solution at least includes the following schemes:

[0006] In a first aspect, a plant leaf hyperspectral trait estimation method is provided, comprising: obtaining a plant leaf hyperspectral dataset; pre-training a trait estimation model in a self-supervised learning manner based on the plant leaf hyperspectral dataset, wherein in the process of pre-training the trait estimation model in the self-supervised learning manner, after embedding data in the plant leaf hyperspectral dataset into high-dimensional feature sequences, self-supervised learning is performed based on the high-dimensional feature sequences, the high-dimensional feature sequences comprising a plurality of high-dimensional features, and the high-dimensional features being features in a hidden space; fine-tuning the pre-trained trait estimation model for a target trait estimation task, and the fine-tuned trait estimation model being used to estimate a target trait based on plant leaf hyperspectral data.

[0007] Optionally, the embedding data in the plant leaf hyperspectral dataset into high-dimensional features comprises: calculating first derivatives and second derivatives of reflectivity of first spectral data at each wavelength band, the first spectral data being data in the plant leaf hyperspectral dataset; dividing the first spectral data into a plurality of wavelength intervals according to a set wavelength interval; performing linear projection on the reflectivity, the first derivatives and the second derivatives of the first spectral data in each wavelength interval respectively to obtain three linear projections corresponding to each wavelength interval; and splicing the three linear projections corresponding to each wavelength interval in a hidden space to obtain a feature vector corresponding to each wavelength interval.

[0008] Optionally, the self-supervised learning based on the high-dimensional features comprises: obtaining a first high-dimensional feature sequence corresponding to the first spectral data, the first high-dimensional feature sequence comprising a feature vector corresponding to each wavelength interval of the first spectral data; setting a mask in the first high-dimensional feature with the feature vector as the minimum unit; and performing self-supervised learning training on the trait estimation model according to the first high-dimensional feature with the mask set.

[0009] Optionally, the trait prediction model comprises an encoder, a cross-attention module and a multi-layer perceptron connected in sequence, the encoder being configured to process the high-dimensional feature sequence to obtain a general spectral feature, the general spectral feature being applicable to a plurality of trait prediction tasks; the cross-attention module being configured to perform cross-attention processing on the general spectral feature based on a query vector to obtain an intermediate feature; and the multi-layer perceptron being configured to predict a plant leaf hyperspectral trait corresponding to the high-dimensional feature based on the intermediate feature.

[0010] Optionally, the fine-tuning the pre-trained trait estimation model for a target trait estimation task comprises: fine-tuning parameters of the query vector and the multi-layer perceptron using mean square error for the target trait estimation task.

[0011] The second aspect also provides a plant leaf hyperspectral trait estimation device, comprising: an acquisition module configured to acquire a plant leaf hyperspectral dataset; a pre-training module configured to pre-train a trait estimation model in a self-supervised learning manner based on the plant leaf hyperspectral dataset, wherein, in the process of pre-training the trait estimation model in the self-supervised learning manner, data in the plant leaf hyperspectral dataset is embedded into a high-dimensional feature sequence after the embedding, the high-dimensional feature sequence comprising a plurality of high-dimensional features, the high-dimensional features being features in a hidden space, and self-supervised learning is performed based on the high-dimensional feature sequence; and a fine-tuning module configured to fine-tune the pre-trained trait estimation model for a target trait estimation task, the fine-tuned trait estimation model being configured to estimate a target trait based on hyperspectral data of a plant leaf.

[0012] Optionally, the pre-training module is further configured to calculate a first derivative and a second derivative of reflectivity of the first spectral data in each wavelength band, the first spectral data being data in the plant leaf hyperspectral data set; divide the first spectral data into a plurality of wavelength intervals according to a set wavelength interval; perform linear projection on the reflectivity, the first derivative and the second derivative of the first spectral data in each wavelength interval respectively to obtain three linear projections corresponding to each wavelength interval; and splice the three linear projections corresponding to each wavelength interval in the latent space to obtain a feature vector corresponding to each wavelength interval.

[0013] Optionally, the pre-training module is further configured to obtain a first high-dimensional feature sequence corresponding to the first spectral data, the first high-dimensional feature sequence including the feature vector corresponding to each wavelength interval of the first spectral data; set a mask in the first high-dimensional feature with the feature vector as a minimum unit; and perform self-supervised learning training on the trait estimation model according to the first high-dimensional feature with the mask set.

[0014] Optionally, the trait prediction model includes an encoder, a cross-attention module and a multi-layer perception machine connected in sequence, the encoder is configured to process the high-dimensional feature sequence to obtain a general spectral feature, the general spectral feature is applicable to a plurality of trait prediction tasks; the cross-attention module is configured to perform cross-attention processing on the general spectral feature based on a query vector to obtain an intermediate feature; and the multi-layer perception machine is configured to predict a plant leaf hyperspectral trait corresponding to the high-dimensional feature based on the intermediate feature.

[0015] Optionally, the fine-tuning module is further configured to fine-tune the parameters of the query vector and the multi-layer perception machine by using mean square error for the target trait estimation task.

[0016] In a third aspect, a computer device is also provided, including a memory and a processor, at least one computer program is stored in the memory, the at least one computer program is loaded and executed by the processor, so as to execute the plant leaf hyperspectral trait method described in the above embodiments.

[0017] In a fourth aspect, a computer readable storage medium is also provided, at least one computer program is stored in the computer readable storage medium, the at least one computer program is loaded and executed by a processor, so as to execute the plant leaf hyperspectral trait method described in the above embodiments.

[0018] In a fifth aspect, a computer program product is provided, including computer programs / instructions, when the computer programs / instructions are executed by a processor, the method of the first aspect is implemented.

[0019] The technical solutions provided by the embodiments of the present disclosure have at least the following beneficial effects:

[0020] Traditional self-supervised learning usually only learns low-dimensional features, and the low-dimensional features are usually for a specific task. If only low-dimensional features are learned, overfitting and poor generality are prone to occur. In the embodiments of the present disclosure, a trait estimation model is pre-trained in a self-supervised learning manner. In the process of pre-training the trait estimation model in the self-supervised learning manner, the data in the plant leaf hyperspectral dataset is embedded into high-dimensional features, and then self-supervised learning is performed based on the high-dimensional features. Compared with learning low-dimensional features, high-dimensional features can express implicit features and have good generality. Therefore, the trait estimation model pre-trained based on high-dimensional features can be applied to a variety of different trait estimation tasks. When a specific trait estimation task is performed, the pre-trained trait estimation model only needs to be fine-tuned, and the fine-tuned trait estimation model can accurately perform trait estimation for the trait estimation task. Therefore, the plant leaf hyperspectral trait estimation method in the embodiments of the present disclosure has both accuracy and generality. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor.

[0022] Figure 1 FIG. 1 shows a flowchart of a plant leaf hyperspectral trait estimation method provided by an example embodiment of the present disclosure;

[0023] Figure 2 FIG. 2 shows a flowchart of a plant leaf hyperspectral trait estimation method provided by another example embodiment of the present disclosure;

[0024] Figure 3 FIG. 3 is a structural schematic diagram of a trait estimation model;

[0025] Figure 4 FIG. 4 is a self-supervised learning flowchart;

[0026] Figure 5 FIG. 5 is a comparison of the performance of different models in the related art and the trait estimation model in the embodiments of the present disclosure in a trait estimation task;

[0027] Figure 6 FIG. 6 is a precision diagram of the trait estimation model in the embodiments of the present disclosure and different models in the related art in estimating a plurality of traits under different amounts of labeled data;

[0028] Figure 7The cross-attention weight diagram of each general spectral feature for different trait estimation tasks is shown.

[0029] Figure 8 A structural diagram of a plant leaf hyperspectral trait estimation device provided by an example embodiment of the present disclosure is shown.

[0030] Figure 9 A structural diagram of a computer device provided by an example embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0031] Unless otherwise defined, technical terms or scientific terms used herein should be interpreted as is normally understood by one of ordinary skill in the art to which the present disclosure pertains. The use of "first", "second", "third", and like terms in the present patent application specification and claims does not represent any order, quantity, or importance, but is only used to distinguish different components. Similarly, "one" or "a" and like terms do not represent a quantity limitation, but represent the existence of at least one. "Including" or "containing" and like terms mean that the elements or objects appearing before "including" or "containing" encompass the elements or objects listed after "including" or "containing" and equivalents thereof, and do not exclude other elements or objects. "Connected" or "connected" and like terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0032] To make the purposes, technical solutions, and advantages of the present disclosure clearer, the present embodiments will be described in further detail below with reference to the accompanying drawings.

[0033] Figure 1 A flowchart of a plant leaf hyperspectral trait estimation method provided by an example embodiment of the present disclosure is shown, which can be executed by a computer device. Referring to Figure 1 , the method comprises:

[0034] In step 101, a plant leaf hyperspectral dataset is obtained.

[0035] Here, the hyperspectral dataset includes hyperspectral remote sensing data of multiple plant leaves. The hyperspectral dataset can be obtained by field collection using a hyperspectral remote sensing device, or can be obtained using an open-source hyperspectral dataset.

[0036] After obtaining the plant leaf hyperspectral dataset, the hyperspectral data in the plant leaf hyperspectral dataset needs to be preprocessed. The preprocessing includes filtering and resampling.

[0037] The filtering can realize smoothing processing on the original hyperspectral data and reduce noise. Exemplarily, a Savizky-Golay filter is used to realize the filtering. The implementation of the Savizky-Golay filter is relatively common in the related art, and thus is not described in detail herein.

[0038] The resampling is to resample the filtered hyperspectral data into a unified format according to a set format. Exemplarily, the filtered hyperspectral data is resampled into unified format hyperspectral data with a resolution of 1 nm and in a range of 350-2500 nm by using a linear interpolation method.

[0039] In step 102, a trait estimation model is pre-trained in a self-supervised learning manner based on the plant leaf hyperspectral data set.

[0040] In the process of pre-training the trait estimation model in the self-supervised learning manner, after the data in the plant leaf hyperspectral data set is embedded into high-dimensional features, self-supervised learning is performed based on the high-dimensional features. The high-dimensional features are features in a hidden space.

[0041] In step 103, the pre-trained trait estimation model is fine-tuned for a target trait estimation task.

[0042] The fine-tuned trait estimation model is used to estimate a target trait based on plant leaf hyperspectral data.

[0043] Traditional self-supervised learning usually only learns low-dimensional features. The low-dimensional features are usually for a specific task. If only low-dimensional features are learned, overfitting and poor generality problems are likely to occur. In the embodiments of the present disclosure, a trait estimation model is pre-trained in a self-supervised learning manner. In the process of pre-training the trait estimation model in the self-supervised learning manner, after the data in the plant leaf hyperspectral data set is embedded into high-dimensional features, self-supervised learning is performed based on the high-dimensional features. Compared with learning low-dimensional features, the high-dimensional features can express implicit features and have good generality. Therefore, the trait estimation model pre-trained based on the high-dimensional features can be applicable to multiple different trait estimation tasks. When a specific trait estimation task is performed, the pre-trained trait estimation model only needs to be fine-tuned. The fine-tuned trait estimation model can accurately perform trait estimation for the trait estimation task. Therefore, the plant leaf hyperspectral trait estimation method in the embodiments of the present disclosure has both accuracy and generality.

[0044] Figure 2 A flowchart of a plant leaf hyperspectral trait estimation method provided by another exemplary embodiment of the present disclosure is shown. The method can be executed by a computer device. Referring to FIG. 10, the method includes the following steps. Figure 2 The method includes the following steps.

[0045] In step 201, a plant leaf hyperspectral dataset is acquired.

[0046] In step 202, a trait estimation model is pre-trained in a self-supervised learning manner based on the plant leaf hyperspectral dataset.

[0047] In the process of pre-training the trait estimation model in the self-supervised learning manner, after data in the plant leaf hyperspectral dataset is embedded into a high-dimensional feature sequence, the self-supervised learning is performed based on the high-dimensional feature sequence, the high-dimensional feature sequence including multiple high-dimensional features, and the high-dimensional feature being a feature in a latent space.

[0048] Figure 3 is a structural schematic diagram of the trait estimation model. In order to facilitate understanding of the embodiments of the present disclosure, the structure of the trait estimation model will be described first in combination with Figure 3 the trait estimation model.

[0049] As shown in Figure 3 , the trait estimation model includes an embedder 301, an encoder 302, a cross-attention module 303 and a multi-layer perception 304 connected in sequence.

[0050] The embedder 301 is configured to embed data in the plant leaf hyperspectral dataset input to the trait estimation model into a high-dimensional feature. In the process of pre-training the trait estimation model in the self-supervised learning manner, the embedder 301 is configured to embed data in the plant leaf hyperspectral dataset into a high-dimensional feature.

[0051] Optionally, the embedder 301 is configured to implement embedding of data in the plant leaf hyperspectral dataset into a high-dimensional feature by the following steps a-d.

[0052] Step a, calculating the first derivative and the second derivative of the reflectivity of the first spectral data at each wavelength band.

[0053] The first spectral data is data in the plant leaf hyperspectral dataset.

[0054] The wavelength band in step a refers to the wavelength band of light, and the wavelength band includes the visible light wavelength band (the wavelength range is 400 nm to 700 nm, the visible light is from violet light to red light), the near-infrared light wavelength band (the wavelength range is 700 nm to 1100 mm), and the short-wave infrared wavelength band (the wavelength range is 1100 nm to 2500 mm). It can be further subdivided, such as dividing the visible light wavelength band into violet light wavelength band, blue light wavelength band, green light wavelength band, yellow light wavelength band, orange light wavelength band and red light wavelength band.

[0055] When step a is performed, the first derivative and the second derivative need to be calculated for the reflectivity of each wavelength band.

[0056] Exemplarily, the first derivative and the second derivative are calculated by using the difference method, and the front end and the rear end are zero-padded to align the sequence length. In this case, the first derivative of the reflectivity of the nth waveband is calculated by using formula (1), and the second derivative of the reflectivity of the nth waveband is calculated by using formula (2).

[0057] (1)

[0058] (2)

[0059] In formula (1) and formula (2), is the first derivative of the reflectivity of the nth waveband, is the second derivative of the reflectivity of the nth waveband, n is a positive integer, and the value range of n is 1 to s, and s is the total number of wavebands. represents the reflectivity of the nth waveband, represents the reflectivity of the nth waveband, represents the reflectivity of the nth waveband, represents the first derivative of the reflectivity of the nth waveband, represents the first derivative of the reflectivity of the nth waveband. By calculating the first derivative and the second derivative of the reflectivity of each waveband respectively, the representation ability of the spectral information can be enhanced. Step b, according to the set wavelength interval, the first spectral data is divided into a plurality of wavelength intervals.

[0060] For example, the wavelength range of the first spectral data is A-B, and the set wavelength interval is C, then the first spectral data can be divided into (B-A) / C wavelength intervals in total, the wavelength in each wavelength interval is continuous, and the difference between the terminal wavelength and the starting wavelength of each wavelength interval is less than or equal to the wavelength interval C. For example, when (B-A) / C is not an integer, the difference between the terminal wavelength and the starting wavelength of the last wavelength interval is less than C, and the difference between the terminal wavelength and the starting wavelength of the other wavelength intervals except the last wavelength interval is equal to the wavelength interval C.

[0061] Exemplarily, the wavelength interval is 30 nm, and the wavelength range of the first spectral data is 350-2500 nm, then the first spectral data can be divided into 72 wavelength intervals in total.

[0062] Step c, the reflectivity, the first derivative and the second derivative of the first spectral data in each wavelength interval are linearly projected respectively, and three linear projections corresponding to each wavelength interval are obtained.

[0063]

[0064] Step c, the reflectivity, the first derivative and the second derivative of the first spectral data in each wavelength interval are linearly projected respectively, and three linear projections corresponding to each wavelength interval are obtained.

[0065] ​​Exemplarily, the linear projection of the reflectance of the first spectral data in the i-th wavelength interval is represented by formula (3), the linear projection of the first derivative of the reflectance of the first spectral data in the i-th wavelength interval is represented by formula (4), and the linear projection of the second derivative of the reflectance of the first spectral data in the i-th wavelength interval is represented by formula (5).

[0066] (3)

[0067] (4)

[0068] (5)

[0069] In formula (3) to formula (5), is the reflectance of the first spectral data in the i-th wavelength interval, is the linear projection of the reflectance of the first spectral data in the i-th wavelength interval; is the first derivative of the reflectance of the first spectral data in the i-th wavelength interval, is the linear projection of the first derivative of the reflectance of the first spectral data in the i-th wavelength interval; is the second derivative of the reflectance of the first spectral data in the i-th wavelength interval, is the linear projection of the second derivative of the reflectance of the first spectral data in the i-th wavelength interval. Wherein, i is a positive integer, the value range of i is 1 to k, and k is the number of wavelength intervals of the first spectral data.

[0070] The linear projection can be implemented by using a fully connected layer, and different fully connected layers can be used when linearly projecting the reflectance, the first derivative and the second derivative respectively. In this case, represents the weight (network parameter) of the fully connected layer for linearly projecting the reflectance, represents the bias of the fully connected layer for linearly projecting the reflectance. represents the weight (network parameter) of the fully connected layer for linearly projecting the first derivative, represents the bias of the fully connected layer for linearly projecting the first derivative. represents the weight (network parameter) of the fully connected layer for linearly projecting the second derivative, represents the bias of the fully connected layer for linearly projecting the second derivative.

[0071] By linearly projecting the reflectivity, the first derivative and the second derivative of each wavelength interval respectively, the reflectivity, the first derivative and the second derivative of each wavelength interval can be converted into the latent space, and the conventional features (reflectivity, first derivative and second derivative) can be converted into high-dimensional features. That is, each feature vector is a high-dimensional feature.

[0072] Step d, splicing the three linear projections corresponding to each wavelength interval in the latent space to obtain the feature vector corresponding to each wavelength interval.

[0073] Exemplarily, the feature vector corresponding to the i-th wavelength interval obtained after splicing the three linear projections corresponding to the i-th wavelength interval in the latent space can be represented by formula (6).

[0074] (6)

[0075] In formula (6), is the feature vector corresponding to the i-th wavelength interval. The meanings of other parameters in formula (6) are the same as those of the parameters in formula (3) to formula (5), and details are omitted here. Exemplarily, when the projection dimension of the full connection layer is 256, the dimension of the feature vector obtained by splicing the three linear projections is 768.

[0076] In this case, the feature vectors corresponding to all wavelength intervals of the first spectral data can be represented as .

[0077] When implementing the embedder, position encoding also needs to be added in . First, a position encoding vector with the same dimension as is generated, and then the position encoding vector is fused with . This process can be represented by formula (7). Wherein, the position encoding vector is a learnable parameter in the pre-training process.

[0078] (7)

[0079] In formula (7), is the output of the embedder, that is, the first high-dimensional feature sequence, which includes the feature vector corresponding to each wavelength interval of the first spectral data, and also includes the position encoding vector .

[0080] For other hyperspectral data except the first spectral data, the embedder 301 can also be processed in the same way as steps a-d above, so as to obtain the high-dimensional feature sequence corresponding to the spectral data.

[0081] ​That is, after the hyperspectral data is input into the embedder 301, the embedder 301 processes the hyperspectral data in the manner of steps a-d above, and outputs the high-dimensional feature sequence corresponding to the hyperspectral data. The high-dimensional feature sequence is input into the encoder 302 for processing.

[0082] The encoder 302 is used to process the high-dimensional feature sequence to obtain a general spectral feature. The general spectral feature is suitable for multiple trait prediction tasks.

[0083] Optionally, the encoder 302 includes but is not limited to a Transformer and its derivative versions. Illustratively, the feature dimension in the encoder 302 is 768, the network depth is 8, the number of attention heads is 12, the number of multilayer perception neurons is 3072, the activation function used in the network is GELU (Gaussian Error Linear Unit), and the normalization strategy is pre-normalization.

[0084] The self-supervised learning based on the high-dimensional feature sequence in step 202 is actually a training of the parameters of the encoder 302 and the embedder 301 (including the position encoding vector and the parameters of the three fully connected layers), so that the encoder 302 has the ability to extract general spectral features.

[0085] Before the self-supervised learning based on the high-dimensional feature sequence, the parameters of the initial encoder 302 need to be copied first, and at this time there are two encoders with the same parameters. One of the encoders is set as a context encoder, and the other encoder is set as a target encoder. During the self-supervised learning, the parameters between the context encoder and the target encoder are updated asynchronously; and after the self-supervised learning is completed, the parameters between the context encoder and the target encoder tend to be consistent, and at this time the parameters of any one of the encoders can be selected as the parameters of the encoder 302.

[0086] In addition, an additional predictor is also needed to be constructed for self-supervised learning, and the predictor is only used in the pre-training process (the parameters of the predictor are also updated in the pre-training process), and after the pre-training is completed, the predictor is no longer used.

[0087] In this case, the self-supervised learning based on the high-dimensional feature sequence includes steps e-g as follows. Figure 4 is a schematic diagram of the self-supervised learning process, and steps e-g will be described below in combination with Figure 4 .

[0088] Step e, obtaining a first high-dimensional feature sequence corresponding to the first spectral data.

[0089] The first high-dimensional feature sequence includes a feature vector corresponding to each wavelength interval of the first spectral data.

[0090] As shown in the following, Figure 4 After the first spectral data is input into the embedder 301, the embedder 301 can output the first high-dimensional feature sequence 401, which includes a plurality of feature vectors.

[0091] Step f, setting a mask in the first high-dimensional feature sequence with the feature vector as the minimum unit.

[0092] Step g, performing self-supervised learning training on the trait estimation model according to the first high-dimensional feature sequence with the mask set.

[0093] Optionally, step g includes the following three steps.

[0094] First step, input the first high-dimensional feature sequence into the target encoder, and input the first high-dimensional feature sequence with the mask set into the context encoder.

[0095] As shown in the following, Figure 4 The first high-dimensional feature sequence 401 is input into the target encoder, and the first high-dimensional feature sequence with the mask 402 set is input into the context encoder. Then the target encoder processes the first high-dimensional feature sequence 401 to obtain the general spectral feature 405 corresponding to the first high-dimensional feature sequence 401; similarly, then the context encoder processes the first high-dimensional feature sequence 401 with the mask 402 set to obtain the general spectral feature 403 with the mask 402 set.

[0096] Here, the essence of the encoder extracting the general spectral feature is to extract certain features from each feature vector of the high-dimensional feature sequence. For the first high-dimensional feature sequence 401 with the mask 402 set, the context encoder cannot extract features from the mask part when processing, so the general spectral feature 403 output by the context encoder also contains the mask.

[0097] Second step, input the output of the context encoder and the position data of each mask in the output of the context encoder into the predictor to obtain the mask feature predicted by the predictor.

[0098] After the output of the context encoder and the position data of each mask in the output of the context encoder are input into the predictor, the predictor can predict the features of the mask part in the general spectral feature 403 output by the context encoder. These mask part features are the mask features 404.

[0099] Third step, updating the parameters of the context encoder and the target encoder based on the mask features predicted by the predictor and the output of the target encoder.

[0100] The parameters of the predictor and the parameters in the embedder 301 are also updated while updating the parameters of the context encoder and the target encoder.

[0101] The general spectral features 405 output by the target encoder correspond to the true values, and any one of the mask features 403 output by the context encoder can find a corresponding true value in the general spectral features 405 output by the target encoder. Therefore, each mask in the mask features 404 can find a true value in 405, and at this time, the loss can be calculated based on the difference between the mask features 404 and the corresponding mask true value in the general spectral features 405 output by the target encoder, and the parameters of the model are updated based on the loss, thereby realizing the pre-training of the trait estimation model.

[0102] Exemplarily, the loss is calculated by taking the SmoothL1 function as the loss function, the parameters of the embedder 301, the context encoder and the predictor are updated by using the back propagation algorithm, and then the parameters of the target encoder are updated by using the EMA (Exponential Moving Average) algorithm, thereby realizing the asynchronous updating of the parameters between the context encoder and the target encoder. Through the asynchrony of the parameter updating, the representation collapse phenomenon in the training process can be effectively prevented.

[0103] In the formula (8), the SmoothL1 function is represented by formula (8).

[0104] (8)

[0105] In formula (8), is the SmoothL1 function, and x is the residual, that is, the difference between the predicted value (the mask feature output by the predictor) and the true value (the mask true value corresponding to the mask in the general spectral features 405 output by the target encoder).

[0106] In the process of pre-training the trait estimation model, the AdamW optimizer can be used, and the learning rate and its decay are dynamically adjusted by combining the warm-up strategy and the cosine annealing strategy. For example, the total training rounds can be set to 3000, the warm-up rounds can be set to 10, and the batch size can be 800. The initial learning rate is set to , the maximum learning rate lrp is set to , and the final learning rate lrf is set to .

[0107] In addition, to prevent the training process from relying on the secondary processing of the predictor, the parameter scale of the predictor needs to be controlled to be much smaller than that of the context encoder. Exemplarily, the predictor has a feature dimension of 96, a network depth of 2, 12 attention heads, and 384 multilayer perceptron neurons.

[0108] After the self-supervised learning is completed, the parameters between the context encoder and the target encoder tend to be consistent, and at this time, the parameters of one of the encoders can be optionally taken as the parameters of the encoder 302. In this case, the encoder 302 has the ability to accurately extract general spectral features in the hyperspectral data.

[0109] The conventional method of learning based on labeled data has the following disadvantages: 1. Lack of labeled data makes training difficult. Model training needs a loss function for back propagation, which is obtained by the difference between the model output and the corresponding label. However, the cost of obtaining labeled data is high, resulting in insufficient labeled data for model training. Under this condition, a model with a large number of parameters is difficult to be effectively trained; 2. The model trained according to specific labels can learn features limited to the training task. For a simple example: after training the model in an end-to-end manner to estimate the chlorophyll content using hyperspectral data, only a small part (near 500 nm and 700 nm) of the features in the entire spectral range (350-2500 nm) contributes to this task. Then, after training, the parameters of the model tend to capture only the features in this region, ignoring all other features. Moreover, it also has a high risk of overfitting, such as accidentally discovering that the feature at 2000 nm is effective for this task (may be due to the bias or noise of the data set used, which is actually incorrect). The errors introduced by overfitting will lead to the inability to apply in new scenarios, which include other traits (such as water content) and other data sets (such as another plant's chlorophyll).

[0110] The method of self-supervised learning based on hyperspectral data to pretrain the trait estimation model in the embodiments of the present disclosure has the following advantages: 1. Hyperspectral data has high dimensionality and rich information, and contains all the content that may be involved by traits; 2. Hyperspectral data is large in quantity and uniform in format, and is easier to learn. Through such a pre-training task, the model can predict one part of the spectral features from another part of the spectral features, indicating that the model has already understood and inferred the complex band interaction relationship.

[0111] In addition, the shape estimation model in the embodiment of the present disclosure is trained based on the features in the high-dimensional hidden space (i.e., high-dimensional features), which are not limited to the spectral reflectance details in the recovery band, but are some general representations with larger information amount. In this way, the pre-trained shape estimation model has the ability to extract general spectral features from the original spectrum (whether partial or complete), which are not limited to one or more traits (because the training does not depend on the annotations) and are not limited to a certain data set (because all samples can be used, whether or not there are annotations), and have good generality.

[0112] In addition, by introducing the predictor in the self-supervised learning process, a condensed hidden space is built between the context and target hidden spaces, preventing the two hidden spaces (the output of the context encoder and the output of the target encoder) from being too easily aligned. If the output of the context encoder and the output of the target encoder are directly used to predict the mask, the model may choose a simple solution (such as directly copying the existing information by local smoothing or interpolation), rather than learning deeper semantic structures. The introduction of the predictor can break this direct mapping, so that the encoder has to learn more global and abstract representations so that the encoder can correctly complete the task and learn the ability to extract more general spectral features.

[0113] Compared with the traditional method of directly learning in an end-to-end mode, the method in the embodiment of the present disclosure can accurately extract general spectral features in a high-dimensional hidden space, and is suitable for estimation tasks of multiple traits, and has a significant advantage in precision. In the scenario where the labeled data is scarce, its robust estimation ability is more prominent, effectively solving the problem of serious shortage of trait labeled data in the field.

[0114] In addition, since the self-supervised learning method is adopted, the method in the embodiment of the present disclosure does not need to rely on large-scale trait labeled data, and only uses hyperspectral data to pre-train the model, thereby significantly reducing the data acquisition and labeling cost.

[0115] In step 203, the pre-trained shape estimation model is fine-tuned for the target trait estimation task.

[0116] The fine-tuned shape estimation model is used to estimate the target trait based on the hyperspectral data of the plant leaf.

[0117] In the shape estimation model, the cross-attention module 303 is used to perform cross-attention processing on the general spectral features based on the query vector to obtain intermediate features.

[0118] The dimension of the query vector is the same as the dimension of the feature vector corresponding to each wavelength interval in the high-dimensional feature sequence. For example, if the dimension of the feature vector corresponding to each wavelength interval in the high-dimensional feature sequence is 768, the query vector can be a vector with a length of 1 and a dimension of 768.

[0119] The cross-attention module 303 is configured to extract, from the general features, features most relevant to the trait estimation as intermediate features, and then input the intermediate features into the multi-layer perceptron.

[0120] The multi-layer perceptron 304 is configured to predict the hyperspectral trait of the plant leaf corresponding to the high-dimensional features based on the intermediate features.

[0121] Optionally, the step 203 comprises: fine-tuning the parameters of the query vector and the multi-layer perceptron based on the mean square error for the target trait estimation task. That is, after the pre-training of the trait estimation model is completed, for different trait estimation tasks, only the query vector and the multi-layer perceptron need to be fine-tuned, thereby reducing the training cost and improving the adaptability of the trait estimation model to different application scenarios. The calculation method of the mean square error is well known in the related art, and is not described in detail here.

[0122] In implementation, a small amount of labeled data can be collected for the target trait estimation task, the labeled data including hyperspectral data and trait true values corresponding to each hyperspectral data, and then the hyperspectral data is input into the trait estimation model to calculate the mean square error between the trait estimation result output by the trait estimation model and the corresponding trait true value, so as to minimize the mean square error and fine-tune the parameters of the query vector and the multi-layer perceptron.

[0123] Figure 5 is a comparison between different models in the related art and the trait estimation model in the embodiments of the present disclosure in terms of performance in the trait estimation task. In Figure 5 , the coefficient of determination R 2and the Normalized Root Mean Square Error (nRMSE) as reference indicators. The involved traits include: Leaf Mass Area (LMA), Water, Chlorophyll (CHL), Carotenoid (CAR), Nitrogen, and Carbon. The related art models include: Partial Least Squares Regression (PLSR) model 501, end-to-end model 502, trait estimation model 503 in the embodiments of the present disclosure, and transfer models 504-509 for different traits. Among them, the transfer model 504 is for the LMA trait, the transfer model 505 is for the Water trait, the transfer model 506 is for the CHL trait, the transfer model 507 is for the CAR trait, the transfer model 508 is for the Nitrogen trait, and the transfer model 509 is for the Carbon trait. The least squares regression model 501, the end-to-end model 502, and the transfer models 504-509 are identical in network structure to the trait estimation model in the embodiments of the present disclosure except for the training strategy.

[0124] By Figure 5 It can be seen that, compared with the PLSR model 501, the end-to-end model 502, and the transfer learning model 504-509, the trait estimation model 503 in the embodiments of the present disclosure shows higher precision levels (R 2 in multiple traits).

[0125] Figure 6 is a precision diagram of estimating multiple traits by different models in the related art and the trait estimation model in the embodiments of the present disclosure under different amounts of labeled data. As Figure 6 shown, the left scatter plot shows the R 2 aggregated by five-fold cross-validation, where each model draws a logarithmic curve to fit the relationship between the proportion of the simulated training set and the R 2 . The right radar chart shows the standard deviation of the five-fold cross-validation. Each row corresponds to a specific trait, and different colors represent model types (same as in Figure 5 ). It can be seen from Figure 6 that the trait estimation model 503 in the embodiments of the present disclosure shows strong robustness in the scenario of sparse labeled data, and maintains R 2 >0.5 for all 6 traits in the scenario of more than 25% training set.

[0126] Figure 7For different trait estimation tasks, the cross-attention weight of each general spectral feature is shown in the figure. Here, the cross-attention module 303 can give different cross-attention weights to different features in the general spectral features output by the encoder 302, and the weight can reflect the importance of a certain feature for the corresponding trait estimation task. Figure 7 The color in the figure represents the importance of the general spectral feature of the wave band for estimating a certain trait, and the redder the color, the more important it is, and the bluer the color, the less important it is. That is, the redder the color of a certain general spectral feature, the more important the general spectral feature is in the corresponding trait estimation task.

[0127] For several traits used for verification, the trait-spectral response rule obtained in the embodiment of the present disclosure is basically consistent with the existing research conclusion. For example, in the moisture content estimation task, the general spectral feature in the wavelength region around 1200 nm is more important, which corresponds to the first overtone response of the O-H chemical bond and is closely related to the water content; in the chlorophyll content estimation, the general spectral feature in the red edge and green light wave band is more important, which is the characteristic interval of plant photosynthesis; in the nitrogen content estimation, in addition to the key red edge wave band, attention is also distributed in several near-infrared wave bands, which reflects the role of nitrogen elements in participating in photosynthesis (forming chlorophyll) and cell structure (protein).

[0128] In addition to the above-mentioned 6 traits, the above-mentioned plant trait estimation method is also applicable to the estimation of lignin, cellulose, sucrose, glucose, potassium element, sulfur element content and the like, and also supports the prediction of other various plant traits. The pre-training process in the embodiment of the present disclosure only needs to be performed once, and subsequent only needs to be fine-tuned for different trait estimation tasks, which significantly reduces the model training and inference cost and improves the computing efficiency.

[0129] The following is a device embodiment of the present application. For details not described in detail in the device embodiment, reference can be made to the above-mentioned method embodiment.

[0130] Figure 8 The structure of the plant leaf hyperspectral trait estimation device provided by an example embodiment of the present disclosure is shown. Referring to Figure 8 The plant leaf hyperspectral trait estimation device 800 includes an acquisition module 801, a pre-training module 802, and a fine-tuning module 803.

[0131] The acquisition module 801 is configured to acquire a plant leaf hyperspectral dataset.

[0132] The pre-training module 802 is configured to pre-train the trait estimation model based on the plant leaf hyperspectral dataset in a self-supervised learning manner, and in the process of pre-training the trait estimation model in the self-supervised learning manner, the data in the plant leaf hyperspectral dataset is embedded into a high-dimensional feature sequence, and then self-supervised learning is performed based on the high-dimensional feature sequence. The high-dimensional feature sequence includes a plurality of high-dimensional features, and the high-dimensional features are features in a hidden space.

[0133] The fine-tuning module 803 is configured to fine-tune the pre-trained trait estimation model for a target trait estimation task, and the fine-tuned trait estimation model is configured to estimate the target trait based on the hyperspectral data of the plant leaf.

[0134] Optionally, the pre-training module 802 is further configured to calculate first-order derivatives and second-order derivatives of reflectivity of the first spectral data at each wavelength band, the first spectral data being data in the plant leaf hyperspectral dataset; divide the first spectral data into a plurality of wavelength intervals according to a set wavelength interval; perform linear projection on the reflectivity, the first-order derivative and the second-order derivative of the first spectral data in each wavelength interval, respectively, to obtain three linear projections corresponding to each wavelength interval; and concatenate the three linear projections corresponding to each wavelength interval in the hidden space to obtain a feature vector corresponding to each wavelength interval.

[0135] Optionally, the pre-training module 802 is further configured to obtain a first high-dimensional feature sequence corresponding to the first spectral data, the first high-dimensional feature sequence including the feature vector corresponding to each wavelength interval of the first spectral data; set a mask in the first high-dimensional feature with the feature vector as the smallest unit; and perform self-supervised learning training on the trait estimation model according to the first high-dimensional feature with the mask set.

[0136] Optionally, the trait prediction model includes an encoder, a cross-attention module and a multi-layer perception machine connected in sequence, the encoder is configured to process the high-dimensional feature sequence to obtain a general spectral feature, the general spectral feature is applicable to a plurality of trait prediction tasks; the cross-attention module is configured to perform cross-attention processing on the general spectral feature based on a query vector to obtain an intermediate feature; and the multi-layer perception machine is configured to predict a plant leaf hyperspectral trait corresponding to the high-dimensional feature based on the intermediate feature.

[0137] Optionally, the fine-tuning module 803 is further configured to fine-tune parameters of the query vector and the multi-layer perception machine for the target trait estimation task in a mean square error manner.

[0138] It should be noted that the plant leaf hyperspectral trait estimation device provided in the above embodiment is used to estimate the plant leaf hyperspectral trait, and only the division of the above functional modules is used as an example for illustration. In actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the plant leaf hyperspectral trait estimation device and the plant leaf hyperspectral trait estimation method provided in the above embodiment belong to the same concept, and the specific implementation process is described in the method embodiment, which will not be repeated here.

[0139] The division of the modules in the embodiments of the present disclosure is illustrative, and is only a logical function division. In actual implementation, there can be another division manner. In addition, each functional module in each embodiment of the present disclosure can be integrated in one processor, or can be a separate physical existence, or two or more modules can be integrated into one module. The integrated module can be realized in the form of hardware or in the form of a software functional module.

[0140] When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present disclosure essentially or the part that contributes to the prior art or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing an end device (which can be a personal computer, a mobile phone, or a communication device, etc.) or a processor to execute all or part of the steps of the method of each embodiment of the present disclosure. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0141] Figure 9 is a structural schematic diagram of the computer device provided in the embodiments of the present disclosure. As shown in Figure 9 The computer device 900 includes a processor 901 and a memory 902.

[0142] The processor 901 can include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 901 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 901 can also include a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also known as a CPU (Central Processing Unit), and the coprocessor being a low-power processor for processing data in a standby state. In some embodiments, the processor 901 can be integrated with a GPU (Graphics Processing Unit) that is responsible for rendering and drawing content required to be displayed by the display screen. In some embodiments, the processor 901 can further include an AI (Artificial Intelligence) processor for processing machine learning related computing operations.

[0143] The memory 902 can include one or more computer-readable storage media that can be non-transitory. The memory 902 can also include a high-speed random access memory, and a nonvolatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one instruction for being executed by the processor 901 to implement the plant leaf hyperspectral trait estimation method provided in the embodiments of the present disclosure.

[0144] Those skilled in the art can understand that the structure shown in the figure does not constitute a limitation on the computer device 900, and can include more or fewer components than those shown, or combine certain components, or use different component arrangements. Figure 9

[0145] The embodiments of the present disclosure further provide a non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a computer device, the computer device is enabled to perform the plant leaf hyperspectral trait estimation method provided in the embodiments of the present disclosure.

[0146] The embodiments of the present disclosure further provide a computer program product, including computer programs / instructions, when the computer programs / instructions are executed by a processor, the plant leaf hyperspectral trait estimation method provided in the embodiments of the present disclosure is implemented. ​

[0147] The above merely provides the optional embodiments of the present disclosure, but does not intend to limit the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for estimating hyperspectral traits of plant leaves, characterized in that, The method comprises: acquiring a plant leaf hyperspectral dataset; pre-training a trait estimation model in a self-supervised learning manner based on the plant leaf hyperspectral dataset, wherein in the process of pre-training the trait estimation model in the self-supervised learning manner, after embedding data in the plant leaf hyperspectral dataset into a high-dimensional feature sequence, the high-dimensional feature sequence is used for self-supervised learning, the high-dimensional feature sequence comprises a plurality of high-dimensional features, and the high-dimensional features are features in a hidden space; fine-tuning the pre-trained trait estimation model for a target trait estimation task, and the fine-tuned trait estimation model is used for estimating a target trait based on hyperspectral data of a plant leaf; the embedding of data in the plant leaf hyperspectral dataset into a high-dimensional feature comprises: calculating first-order and second-order derivatives of reflectivity of first spectral data at each wavelength band, wherein the first spectral data is data in the plant leaf hyperspectral dataset; dividing the first spectral data into a plurality of wavelength intervals according to a set wavelength interval; performing linear projection on the reflectivity, the first-order derivative and the second-order derivative of the first spectral data in each wavelength interval respectively to obtain three linear projections corresponding to each wavelength interval; splicing the three linear projections corresponding to each wavelength interval in the hidden space to obtain a feature vector corresponding to each wavelength interval; the trait prediction model comprises an encoder, a cross-attention module and a multilayer perceptron connected in sequence, the encoder is used for processing the high-dimensional feature sequence to obtain a general spectral feature, and the general spectral feature is applicable to a plurality of trait prediction tasks; the cross-attention module is used for cross-attention processing on the general spectral feature based on a query vector to obtain an intermediate feature; the multilayer perceptron is used for predicting a plant leaf hyperspectral trait corresponding to the high-dimensional feature based on the intermediate feature.

2. The method of claim 1, wherein, the self-supervised learning based on the high-dimensional feature comprises: acquiring a first high-dimensional feature sequence corresponding to the first spectral data, wherein the first high-dimensional feature sequence comprises a feature vector corresponding to each wavelength interval of the first spectral data; setting a mask in the first high-dimensional feature with the feature vector as the minimum unit; performing self-supervised learning training on the trait estimation model according to the first high-dimensional feature with the mask.

3. The method of claim 1, wherein, the fine-tuning of the pre-trained trait estimation model for the target trait estimation task comprises: fine-tuning parameters of the query vector and the multilayer perceptron in the mean square error for the target trait estimation task.

4. A plant leaf hyperspectral trait estimation device, comprising: The device comprises: an acquisition module configured to acquire a plant leaf hyperspectral dataset; a pre-training module configured to pre-train a trait estimation model in a self-supervised learning manner based on the plant leaf hyperspectral dataset, wherein in the process of pre-training the trait estimation model in the self-supervised learning manner, after embedding data in the plant leaf hyperspectral dataset into a high-dimensional feature sequence, the high-dimensional feature sequence is used for self-supervised learning, the high-dimensional feature sequence comprises a plurality of high-dimensional features, and the high-dimensional features are features in a hidden space; The fine-tuning module is configured to fine-tune the pre-trained trait estimation model for a target trait estimation task, and the fine-tuned trait estimation model is configured to estimate the target trait based on the hyperspectral data of the plant leaf. The pre-training module is further configured to calculate first-order derivatives and second-order derivatives of reflectivity of the first spectral data at respective wavelength bands, the first spectral data being data in the plant leaf hyperspectral data set; The first spectral data is divided into a plurality of wavelength intervals according to a set wavelength interval; The reflectivity, the first-order derivative and the second-order derivative of the first spectral data in each wavelength interval are respectively linearly projected to obtain three linear projections corresponding to each wavelength interval; The three linear projections corresponding to each wavelength interval are spliced in a hidden space to obtain a feature vector corresponding to each wavelength interval; The trait prediction model comprises an encoder, a cross-attention module and a multilayer perceptron connected in sequence, the encoder is configured to process the high-dimensional feature sequence to obtain a general spectral feature, and the general spectral feature is applicable to multiple trait prediction tasks; The cross-attention module is configured to perform cross-attention processing on the general spectral feature based on a query vector to obtain an intermediate feature; The multilayer perceptron is configured to predict a plant leaf hyperspectral trait corresponding to the high-dimensional feature based on the intermediate feature.

5. A computer device, comprising: The computer device comprises a memory and a processor, at least one computer program is stored in the memory, the at least one computer program is loaded and executed by the processor to implement the method of any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one computer program, the at least one computer program is loaded and executed by the processor to implement the method of any one of claims 1 to 3.

7. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to implement the method of any one of claims 1 to 3.

Images