A multi-task learning plant phenotype processing method and device based on VGG16

Abstract

The present invention discloses a multi-task learning plant phenotype processing method and device based on VGG16, the method comprising: (1) constructing a plant phenotype processing and analysis model: using the VGG16 network as a hard parameter sharing network for multi-task learning, removing the last maximum pooling layer and all subsequent fully connected layers of the original VGG16, adding task-related fully connected layers, and applying them to three specific task layers respectively; (2) training the model: using task-dependent uncertainty to weigh the loss function of each specific task layer to balance the loss of regression and classification tasks, using flexible factors to connect different tasks, automatically optimizing the weights of different tasks, and allowing each task to be optimally trained; (3) processing and analyzing plant phenotypes based on the trained model. The present invention can simultaneously complete three tasks related to Arabidopsis thaliana phenotype processing and analysis, greatly improving the overall classification and prediction effect of multi-tasks.

Description

Technical Field

[0001] The present invention relates to a multi-task learning plant phenotype processing method and device based on VGG16, belonging to the technical field of artificial intelligence and botany. Background Art

[0002] At present, the processing of plant phenotypes mostly focuses on a specific phenotypic trait and feature of the plant, and can only achieve a single function. Although the existing multi-task learning methods can process and analyze multiple phenotypic characteristics of plants, the loss connection between multiple tasks is not flexible, and its classification and regression performance needs to be improved.

[0003] Moreover, for the same plant, existing products need to construct multiple different models and use different data sets to achieve multiple tasks such as plant classification, leaf number prediction, leaf area prediction, etc. The multi-task learning method that uses one model and data set to complete multiple tasks still has much room for improvement in plant phenotypic processing and reasoning. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-task learning plant phenotyping method and device based on VGG16, which can simultaneously identify the plant phenotypic image categories of different genotypes and calculate their leaf number and leaf area information according to data sets such as plant leaf segmentation, counting and classification.

[0005] The technical solution adopted by the present invention is as follows:

[0006] In a first aspect, the present invention provides a multi-task learning plant phenotyping method based on VGG16, which comprises the following steps:

[0007] (1) Constructing a plant phenotyping and analysis model: Using the VGG16 network as a hard parameter sharing network for multi-task learning, modifying the traditional VGG16 model structure, removing the last maximum pooling layer and all subsequent fully connected layers of the original VGG16, adding task-related fully connected layers, and applying them to three specific task layers. Each specific task layer is composed of fully connected layers to complete its own specific feature extraction tasks.

[0008] (2) Training model: Use task-dependent uncertainty to weigh the loss function of each specific task layer to balance the loss of regression and classification tasks. Use flexible factors to connect different tasks and automatically optimize the weights of different tasks so that each task can be optimally trained.

[0009] (3) Process and analyze plant phenotypes based on the trained model.

[0010] Furthermore, the three specific task layers are: genotype classification task, leaf number regression task and leaf area regression task. Among them, the specific task layer of the genotype classification task consists of a 1×1×512 fully connected layer, a 1×1×256 fully connected layer and a softmax layer, and finally outputs the classification result through the softmax layer; the specific task layer of the leaf number regression task passes through a 1×1×512 layer and a 1×1×1 fully connected layer, and outputs the regression result; the specific task layer of the leaf area regression task is constructed the same as the specific task layer of the leaf number regression task.

[0011] Furthermore, after the plant phenotype image is input into the model, the corresponding feature vector is extracted through the basic shared network VGG16 and the specific task fully connected layer of each task. The genotype classification task performs multi-classification of the feature vector through the softmax layer to obtain the probability of each category. The leaf number regression task and the leaf area regression task apply the Relu activation function in the specific task fully connected layer to obtain the final prediction result.

[0012] Furthermore, the softmax layer applies the Softmax activation function to map the inputs of multiple neurons between [0,1], and the sum of the mapped elements in the vector is 1. The formula for calculating its value is:

[0013]

[0014] In the above formula, i and j are the class labels in the multi-classification problem, and x i Represents the i-th category value in the input x value and maps the value to the exponential function, Represents the sum of all category value indices.

[0015] Furthermore, the formula of the Relu activation function is:

[0016] Relu(x)=max(0,x)

[0017] In the above formula, x is the input value, and the Relu function takes the maximum value between 0 and x. Therefore, when the input is negative, the function is not activated and is not applicable to the case where all inputs are negative.

[0018] Furthermore, the regression and classification task losses are derived by maximizing the maximum Gaussian likelihood of the task-dependent uncertainty:

[0019] (2.1) Likelihood function for regression task

[0020] The input is x, the true label is y, the parameter matrix is ​​W, and the output is f W (x), for the regression task, its probability is defined as the Gaussian likelihood with the output as the mean, that is:

[0021] p(y|f W (x)) = N(f W (x),σ 2 ) (2.1)

[0022] In the Gaussian likelihood shown in formula (2.1), σ is the observation noise parameter of the model; based on this Gaussian likelihood, the logarithmic likelihood of the regression task in multi-task joint learning can be obtained:

[0023]

[0024] (2.2) Likelihood function for classification tasks

[0025] In the classification task, a standard softmax layer is used, and the Gaussian likelihood of its probability is:

[0026] p(y|f W (x)) = Softmax(f W (x)) (2.3)

[0027] In formula (2.3), the noise parameter σ is introduced as a scaling factor. This parameter is related to the uncertainty of the distribution and determines the flatness of the discrete distribution. The logarithmic likelihood of the classification task is finally obtained as shown in the following formula (2.4), where c represents the number of classifications;

[0028]

[0029] (2.3) Multi-task joint loss

[0030] According to the above formula, adaptive weights based on loss are constructed for different tasks, and different tasks are connected through flexible factors to balance the influence of each task. Under the condition of maximizing the log likelihood, that is, minimizing the negative log likelihood, the joint loss of multiple tasks is constructed:

[0031]

[0032] In formula (2.5), y1, y2, and y3 are the true labels of the leaf area regression task, leaf number regression task, and genotype classification task respectively; correspondingly, σ1, σ2, and σ3 are the noise parameters of the leaf area regression task, leaf number regression task, and genotype classification task respectively; L1(W) = || y1-f W (x)|| 2 , represents the Euclidean distance between the leaf area regression subtask output and the true label y1; L2(W) represents the Euclidean distance between the leaf number regression subtask output and the true label y2; L3(W) = -log(Softmax(f W (x))) is the cross entropy loss for the classification subtask;

[0033] At the same time, the flexible factors of the leaf area regression task, leaf number regression task, and genotype classification task are α, β, and γ, respectively. The values ​​of the three flexible factors fluctuate in the range of (0,1) and satisfy α+β+γ=1. This allows the model to continuously learn and adjust the loss of each subtask to minimize the multi-task loss during the training process. At the same time, it can also control the impact of each task on the model to achieve a balanced state.

[0034] Furthermore, when learning the parameters of the weight relationship factors of the model, the weight-related parameters are combined together according to formula (2.5) to form a joint loss as a prerequisite, and then the weight-related parameters are gradually updated in the iterative rounds, and finally a combination of a weight parameter matrix and multiple task weight-related flexible factors is learned, so that the loss function of multiple tasks reaches the minimum value, and the influence of each task on the model training is in a balanced state.

[0035] Furthermore, during the model training process, the model optimizer will automatically update the network parameters, including the flexible factor values ​​of each task, according to the gradient information of the network back propagation, to reduce the calculated value of the loss function; if after the previous round of iterative training, the joint loss value of the model is calculated, the leaf area loss accounts for a large proportion of the total loss, that is, in formula (2.5) If the value is large, it means that the prediction effect of the leaf area task is not good. Then the optimizer will choose to train the model in a direction that is beneficial to the leaf area regression task in the next iteration, and reduce the flexibility factor of the leaf area regression task, that is, α in formula (2.5), reduce the proportion of the loss of this task, and appropriately adjust β and γ to meet α+β+γ=1, so that the proportion of each task loss to the total joint loss is similar; after the next round of model training is completed, the leaf area loss value decreases, the weight decreases, and the proportion of the total loss decreases. Other tasks with a higher proportion of the total loss value can be selected for training; if the loss of the genotype classification task accounts for a large proportion of the total loss, the genotype classification task will dominate the training, and the flexibility factor of the genotype classification task will be reduced, and the flexibility factor values ​​of other tasks will be adjusted; for the leaf number regression task, the same is true. After each model training, the weight parameters and flexibility factors of each task are adjusted so that in the process of continuous model training, the influence of each task on the model training reaches a balanced state.

[0036] In a second aspect, the present invention provides a multi-task learning plant phenotype processing device based on VGG16, comprising:

[0037] Model building module, used to build a plant phenotyping and analysis model. The constructed model uses the VGG16 network as a hard parameter sharing network for multi-task learning. The last maximum pooling layer and all subsequent fully connected layers of the original VGG16 are removed, and task-related fully connected layers are added. They are applied to three specific task layers respectively. Each specific task layer is composed of fully connected layers to complete their own specific feature extraction tasks.

[0038] The model training module is used to automatically learn and train the constructed plant phenotyping and analysis model. It uses task-dependent uncertainty to weigh the loss function of each specific task layer to balance the loss of regression and classification tasks, uses flexible factors to connect different tasks, and automatically optimizes the weights of different tasks, so that each task can be optimally trained.

[0039] The image recognition module is used to receive the plant image to be recognized and use the trained model to recognize and analyze the plant image to be recognized.

[0040] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the plant phenotype processing method described in the first aspect when executing the computer program.

[0041] In summary, the present invention has the following characteristics:

[0042] (1) Using a multi-task learning method based on VGG16 hard parameter sharing, we can simultaneously complete three tasks related to Arabidopsis phenotypic processing and analysis, namely, classifying the relevant genotypes of Arabidopsis, predicting the number of leaves in Arabidopsis images, and predicting the leaf area.

[0043] (2) Task-dependent uncertainty is used to weigh the joint loss of multiple tasks to prevent a single task from dominating model training. While minimizing the joint loss of multiple tasks, the impact of each task on model training is balanced, thereby improving the overall classification and prediction effects of multiple tasks.

[0044] (3) Use flexible factors to connect multiple tasks, and automatically optimize the weights of each task in the multi-task according to the flexible factors, so that the impact of each task on model training is balanced and the inference model is optimized without manually setting the weight parameters of each task.

[0045] Compared with the prior art, the present invention has the following advantages:

[0046] (1) Using the hard parameter sharing framework of multi-task learning, VGG16 is used as the basic network for hard parameter sharing. The basic network is shared by tasks such as plant phenotype classification, leaf number prediction, and leaf area prediction. A specific task layer is established for each task to extract different features to obtain their respective classification and regression results.

[0047] (2) When performing multi-task joint training, the task-dependent uncertainty is used to weigh the loss function of each specific task layer to balance the loss of regression and classification tasks. Flexible factors are used to connect different tasks so that each task can be optimally trained to obtain better genotype classification and predictive reasoning performance and results.

[0048] (3) Multi-task learning shares the shallow representation of related tasks, and can extract useful information from related tasks as a supplement to the current task for learning, which improves the generalization ability and accuracy of the model. Even when the amount of data is small and the annotations are scarce, a model that effectively solves the problem can still be obtained. This method can be extended to different plants and different phenotypes, and can provide a research framework and reference for other plant phenotypes.

[0049] (4) Through automatic optimization of parameters of automated learning, the optimizer will automatically adjust the weight-related parameters according to the proportion of each task loss to the total loss after each round of iteration, and automatically optimize the weights of different tasks according to the flexible factor. Without the need to manually set the weight parameters related to each task, the impact of each task on model training can be balanced, preventing a single task from dominating the training and obtaining the overall optimal solution.

[0050] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a schematic diagram of the model structure constructed by the present invention;

[0052] Figure 2 It is a schematic diagram of model training of the present invention. DETAILED DESCRIPTION

[0053] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] The present invention provides a multi-task learning plant phenotype processing method based on VGG16. The VGG16 network is selected as the basic shared network in hard parameter sharing multi-task learning, the multi-task dependency uncertainty is used to balance the multi-task loss, and the weights of different tasks are automatically optimized according to the flexible factor, so that each task in the multi-task learning is trained in an optimized manner, thereby obtaining better overall prediction and genotype classification effects.

[0055] As a result, a plant phenotyping and analysis model based on multi-task learning hard parameter sharing and task-dependent uncertainty of improved VGG16 was obtained. Its overall network structure is as follows: Figure 1 shown.

[0056] 1. VGG16 hard parameter sharing network

[0057] The present invention uses the VGG16 network as a hard parameter sharing network for multi-task learning, modifies the traditional VGG16 model structure, removes the last maximum pooling layer of the original VGG16 and all subsequent fully connected layers, adds task-related fully connected layers, and applies them to three specific task layers respectively. Each specific task layer is composed of fully connected layers to complete their own specific feature extraction tasks. The three specific task layers are: genotype classification task, leaf number calculation (regression) task and leaf area calculation (regression) task.

[0058] (1) The specific task layer of the genotype classification task consists of a 1×1×512 fully connected layer, a 1×1×256 fully connected layer, and a softmax layer. Finally, the classification result is output through the softmax layer.

[0059] (2) The leaf number calculation task passes through a 1×1×512 layer and a 1×1×1 fully connected layer to output the regression result;

[0060] (3) The specific task layer for leaf area calculation constructs the leaf number calculation task.

[0061] The comparison of the network parameters of each task with the original VGG16 network is shown in Table 1.

[0062] Table 1 Network parameter comparison

[0063]

[0064] After the plant phenotype image is input into the network model, the corresponding feature vector is extracted through the basic shared network VGG16 and the specific task fully connected layer of each task. The genotype classification task performs multi-classification of the feature vector through the softmax layer to obtain the probability of each category. The leaf number regression task and the leaf area regression task apply the Relu activation function in the specific task fully connected layer to obtain the final prediction result.

[0065] 1.1 Softmax function

[0066] The softmax layer applies the Softmax activation function, which is suitable for multi-classification problems. It maps the inputs of multiple neurons between [0,1], and the sum of the mapped elements in the vector is 1. The formula for calculating its value is:

[0067]

[0068] In formula (1.1), i and j are the class labels in the multi-classification problem, and x i Represents the i-th category value in the input x value and maps the value to the exponential function, Represents the sum of all category value indices.

[0069] 1.2 Relu activation function

[0070] The formula of the Relu activation function is:

[0071] Relu(x)=max(0,x) (1.2)

[0072] In formula (1.2), x is the input value, and the Relu function takes the maximum value between 0 and x. Therefore, when the input is negative, the function is not activated, and is not applicable to the case where all inputs are negative. The Relu function overcomes the problem of gradient disappearance, and because of its linear and non-saturated form, it has a fast training speed.

[0073] 2. Using Task Dependency Uncertainty to Balance the Joint Losses at the Specific Task Layer

[0074] In the present invention, the tasks of plant phenotype include: leaf area and leaf number calculation based on regression, and genotype classification. The training adopts a hard parameter sharing model. In hard parameter sharing, no matter how many tasks there are, each task uses the same basic model and shares most of the underlying parameters. The specific task layer assigned to each task has parameters unique to each task. Since hard parameters share most parameters, when the tasks are weakly related, simply adding the loss functions defined in the task-specific layer for joint learning will result in a task dominating the entire loss due to the different loss scales of each task, leaving the remaining tasks with no chance to affect the learning process of the shared layer, so that the reasoning effect of one task is good, while the reasoning effects of the remaining tasks are not ideal, resulting in a negative transfer effect.

[0075] The present invention uses task-dependent uncertainty to weigh the loss functions of different specific task layers, uses the joint loss to guide the training of the next model, balances the impact of each task on model training, and can reduce the impact of negative transfer on weakly related tasks. Task-dependent uncertainty, also known as homoscedastic uncertainty, is a numerical value that does not depend on input data, remains constant for all input data, and varies between different tasks.

[0076] Task uncertainty is used to weigh different losses, and the final multi-task joint loss is derived by maximizing the maximum Gaussian likelihood that maximizes task-dependent uncertainty.

[0077] 2.1 Likelihood Function for Regression Tasks

[0078] The input is x, the true label is y, the parameter matrix is ​​W, and the output is f W (x), for the regression task, its probability is defined as the Gaussian likelihood with the output as the mean, that is:

[0079] p(y|f W (x)) = N(f W (x),σ 2 ) (2.1)

[0080] In the Gaussian likelihood shown in formula (2.1), σ is the observation noise parameter of the model. Based on this Gaussian likelihood, the logarithmic likelihood of the regression task in multi-task joint learning can be obtained:

[0081]

[0082] 2.2 Likelihood Function for Classification Tasks

[0083] In the classification task, a standard softmax layer is used, and the Gaussian likelihood of its probability is:

[0084] p(y|f W (x)) = Softmax(f W (x)) (2.3)

[0085] In formula (2.3), the noise parameter σ is introduced as a scaling factor. This parameter is related to the uncertainty of the distribution and determines the flatness of the discrete distribution. The final log-likelihood of the classification task is shown in formula (2.4), where c represents the number of classifications.

[0086]

[0087] 2.3 Multi-task Joint Loss

[0088] The multi-task model in the present invention consists of a classification task and two regression tasks. According to the above formula, adaptive weights based on losses are constructed for different tasks, and different tasks are connected by flexible factors to balance the influence of each task. Under the condition of maximizing the log-likelihood, that is, minimizing the negative log-likelihood, the joint loss of multiple tasks is constructed:

[0089]

[0090] In formula (2.5), y1, y2, and y3 are the true labels of the leaf area regression task, leaf number regression task, and genotype classification task respectively; correspondingly, σ1, σ2, and σ3 are the noise parameters of the leaf area regression task, leaf number regression task, and genotype classification task respectively; L1(W) = || y1-f W (x)||2 , represents the Euclidean distance between the leaf area regression subtask output and the true label y1; L2(W) represents the Euclidean distance between the leaf number regression subtask output and the true label y2; L3(W) = -log(Softmax(f W (x))) is the cross entropy loss for the classification subtask.

[0091] At the same time, the flexible factors of the leaf area regression task, leaf number regression task, and genotype classification task are α, β, and γ, respectively. The values ​​of the three flexible factors fluctuate in the range of (0,1) and satisfy α+β+γ=1. This allows the model to continuously learn and adjust the loss of each subtask to minimize the multi-task loss during the training process. At the same time, it can also control the impact of each task on the model to achieve a balanced state.

[0092] In the joint loss constructed using task-dependent uncertainty, the larger the task loss value, the higher the proportion of the loss in the total loss, the greater the impact on the network structure update, and the model will be guided to optimize in a direction that is beneficial to the task, so that the loss value corresponding to the task will be reduced after the next round of training. If the loss value of other tasks increases, it can also be adjusted continuously through iterative training to optimize the overall loss value. The model weights and the flexible factors of each task are continuously updated through back propagation during the training process, and eventually the overall optimal solution can be found, and the overall best genotype classification, leaf number regression, and leaf area regression results are obtained.

[0093] 2.4 Automatic learning parameters

[0094] When learning the parameters of the weight relationship factors of the model, the weight-related parameters are combined together according to formula (2.5) to form a joint loss as a prerequisite, and then the weight-related parameters are gradually updated in the iterative rounds. Finally, a combination of a weight parameter matrix and multiple task weight-related flexible factors can be learned, so that the loss function of multiple tasks reaches the minimum value, and the influence of each task on the model training is in a balanced state.

[0095] During the model training process, the model optimizer will automatically update the network parameters, including the flexibility factor value of each task, according to the gradient information of the network back propagation to reduce the calculated value of the loss function. The update process of the weight-related parameters is as follows: Figure 2 If the joint loss value of the model is calculated after the previous round of iterative training, the leaf area loss accounts for a large proportion of the total loss, that is, If the value is large, it means that the prediction effect of the leaf area task is not satisfactory. Then the optimizer will choose to train the model in a direction that is beneficial to the leaf area regression task in the next iteration, and reduce the flexibility factor of the leaf area regression task, that is, α in formula (2.5), reduce the proportion of the loss of this task, and appropriately adjust β and γ to meet α+β+γ=1, so that the proportion of each task loss to the total joint loss is similar. After the next round of model training is completed, the leaf area loss value decreases, the weight decreases, and the proportion of the total loss decreases. Other tasks with a higher proportion of the total loss value can be selected for training. If the loss of the genotype classification task accounts for a large proportion of the total loss, the genotype classification task will dominate the training, and the flexibility factor of the genotype classification task will be reduced, and the flexibility factor values ​​of other tasks will be adjusted. The same is true for the leaf number regression task. After each model training, the weight parameters and flexibility factors of each task are adjusted so that the influence of each task on the model training reaches a balanced state during the continuous training of the model.

[0096] The optimizer automatically adjusts the weight parameters and flexible factors according to the proportion of each task loss to the total loss after each round of iteration, so that the proportion of each task in the total loss is closer than that of the previous round of iteration, and the total loss gradually decreases. There is no need to manually set the weight-related parameters of each task. The impact of each task on the model training can reach a balanced state after continuous training. At the same time, it can also prevent a single task from dominating the training, and finally obtain the overall optimal solution.

[0097] The present invention also provides a multi-task learning plant phenotype processing device based on VGG16, which includes:

[0098] Model building module, used to build a plant phenotyping and analysis model. The constructed model uses the VGG16 network as a hard parameter sharing network for multi-task learning. The last maximum pooling layer and all subsequent fully connected layers of the original VGG16 are removed, and task-related fully connected layers are added. They are applied to three specific task layers respectively. Each specific task layer is composed of fully connected layers to complete their own specific feature extraction tasks.

[0099] The model training module is used to automatically learn and train the constructed plant phenotyping and analysis model. It uses task-dependent uncertainty to weigh the loss function of each specific task layer to balance the loss of regression and classification tasks, uses flexible factors to connect different tasks, and automatically optimizes the weights of different tasks, so that each task can be optimally trained.

[0100] The image recognition module is used to receive the plant image to be recognized and use the trained model to recognize and analyze the plant image to be recognized.

[0101] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned plant phenotype processing method when executing the computer program.

[0102] The memory includes at least one type of computer-readable storage medium, and the processor in some embodiments may be a central processing unit, a controller, a microcontroller, a microprocessor, or other data processing chips.

[0103] In combination with the above scheme, the present invention compares the evaluation indicators of the single-task learning model, the multi-task learning model using soft parameter sharing, and the VGG16 hard parameter sharing multi-task learning model that does not use task-dependent uncertainty to balance the loss function on the Arabidopsis genotype classification, leaf number regression, and leaf area regression tasks. In this data set, the results are when α, β, and γ are all 0.3333.

[0104] In the genotype classification task, the results of accuracy, precision, recall and F1 value are shown in Table 2.

[0105] Table 2 Classification results

[0106]

[0107] As can be seen from Table 2, the hard parameter sharing multi-task learning model based on the VGG16 network proposed in the present invention that uses task-dependent uncertainty to balance the loss function has the best classification effect, reaching 96.88% accuracy, 97.50% precision, 96.88% recall, and 96.74% F1 value, and is about 4.00% to 6.00% higher than other methods in all indicators.

[0108] For the leaf number regression task, the results of the coefficient of determination, mean square error, and mean absolute error are shown in Table 3.

[0109] Table 3. Leaf number regression results

[0110]

[0111] It can be seen from Table 3 that the leaf number regression effect of the model proposed in the present invention is the best, in which the regression determination coefficient reaches 0.7944, which is about 9.00% higher than other methods on average, and various error evaluation indicators are lower than other methods, MSE is 0.5892, which is about 25.00% lower than other methods, and MAE is 0.5846, which is about 14.00% lower than other methods.

[0112] For the leaf area regression task, the results of the coefficient of determination, mean square error, and mean absolute error are shown in Table 4.

[0113] Table 4 Leaf area regression results

[0114]

[0115] As can be seen from Table 4, the leaf area regression effect of the model proposed in the present invention is also the best, wherein the regression determination coefficient reaches 0.9787, which is about 1.75% higher than other methods on average, and various error evaluation indicators are lower than other methods, MSE is 0.3029, which is about 25.00% lower than other methods, and MAE is 0.4551, which is about 13.00% lower than other methods.

[0116] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the protection scope of the present invention in any form, and all technical solutions obtained by equivalent replacement and the like fall within the protection scope of the present invention.

[0117] The parts not involved in the present invention are the same as the prior art or can be implemented by using the prior art.

Claims

1. A multi-task learning plant phenotyping method based on VGG16, characterized in that The steps include: (1) Constructing a plant phenotypic processing and analysis model: Using the VGG16 network as a hard parameter sharing network for multi-task learning, modifying the traditional VGG16 model structure, removing the last maximum pooling layer and all subsequent fully connected layers of the original VGG16, adding task-related fully connected layers, and applying them to three specific task layers. Each specific task layer is composed of fully connected layers, and each completes its own specific feature extraction task; the three specific task layers are: genotype classification task, leaf number regression task, and leaf area regression task; after the plant phenotype image is input into the model, the corresponding feature vector is extracted through the basic shared network VGG16 and the specific task fully connected layer of each task. For the genotype classification task, the softmax layer is used to perform multi-classification of the feature vector to obtain the probability of each category. For the leaf number regression task and the leaf area regression task, the Relu activation function is applied to the specific task fully connected layer to obtain the final prediction result; (2) Training model: Use task-dependent uncertainty to weigh the loss function of each specific task layer to balance the loss of regression and classification tasks. Use flexible factors to connect different tasks and automatically optimize the weights of different tasks so that each task can be optimally trained. The regression and classification task losses are derived by maximizing the maximum Gaussian likelihood of the task-dependent uncertainty: (2.1) Likelihood function for regression task The input is x, the true label is y, the parameter matrix is ​​W, and the output is f W (x), for the regression task, its probability is defined as the Gaussian likelihood with the output as the mean, that is: p(y|f W (x))=N(f W (x),σ 2 )(2.1) In the Gaussian likelihood shown in formula (2.1), σ is the observation noise parameter of the model; based on this Gaussian likelihood, the logarithmic likelihood of the regression task in multi-task joint learning can be obtained: (2.2) Likelihood function for classification tasks In the classification task, a standard softmax layer is used, and the Gaussian likelihood of its probability is: p(y|f W (x))=Softmax(f W (x))(2.3) In formula (2.3), the noise parameter σ is introduced as a scaling factor. This parameter is related to the uncertainty of the distribution and determines the flatness of the discrete distribution. The logarithmic likelihood of the classification task is finally obtained as shown in the following formula (2.4), where c represents the number of classifications; (2.3) Multi-task joint loss According to the above formula, adaptive weights based on loss are constructed for different tasks, and different tasks are connected through flexible factors to balance the influence of each task. Under the condition of maximizing the log likelihood, that is, minimizing the negative log likelihood, the joint loss of multiple tasks is constructed: In formula (2.5), y1, y2, and y3 are the true labels of the leaf area regression task, leaf number regression task, and genotype classification task respectively; correspondingly, σ1, σ2, and σ3 are the noise parameters of the leaf area regression task, leaf number regression task, and genotype classification task respectively; L1(W) = || y1-f W (x)|| 2 , represents the Euclidean distance between the leaf area regression subtask output and the true label y1; L2(W) represents the Euclidean distance between the leaf number regression subtask output and the true label y2; L3(W) = -log(Softmax(f W (x))) is the cross entropy loss for the classification subtask; At the same time, the flexible factors of the leaf area regression task, leaf number regression task, and genotype classification task are α, β, and γ, respectively. The values ​​of the three flexible factors fluctuate within the range of (0, 1) and satisfy α+β+γ=1. This allows the model to continuously learn and adjust the loss of each subtask to minimize the multi-task loss during the training process. At the same time, it can also control the impact of each task on the model to achieve a balanced state. During the model training process, the model optimizer will automatically update the network parameters, including the flexible factor values ​​of each task, according to the gradient information of the network back propagation, in order to reduce the calculated value of the loss function; if after the previous round of iterative training, the joint loss value of the model is calculated, the leaf area loss accounts for a large proportion of the total loss, that is, in formula (2.5) If the value is large, it means that the prediction effect of the leaf area task is not good. Then the optimizer will choose to train the model in a direction that is beneficial to the leaf area regression task in the next iteration, and reduce the flexibility factor of the leaf area regression task, that is, α in formula (2.5), reduce the proportion of the task loss, and appropriately adjust β and γ to meet α+β+γ=1, so that the proportion of each task loss to the total joint loss is similar; after the next round of model training is completed, the leaf area loss value decreases, the weight decreases, and the proportion of the total loss decreases. Other tasks with a higher proportion of the total loss value can be selected for training; if the loss of the genotype classification task accounts for a large proportion of the total loss, the genotype classification task will dominate the training, and the flexibility factor of the genotype classification task will be reduced, and the flexibility factor values ​​of other tasks will be adjusted; for the leaf number regression task, the same is true. After each model training, the weight parameters and flexibility factors of each task are adjusted so that in the process of continuous model training, the influence of each task on the model training reaches a balanced state; (3) Process and analyze plant phenotypes based on the trained model.

2. A multi-task learning plant phenotype processing method based on VGG16 according to claim 1, characterized in that: The specific task layer of the genotype classification task consists of a 1×1×512 fully connected layer, a 1×1×256 fully connected layer and a softmax layer, and finally outputs the classification result through the softmax layer; the specific task layer of the leaf number regression task passes through a 1×1×512 layer and a 1×1×1 fully connected layer, and outputs the regression result; the specific task layer of the leaf area regression task is constructed in the same way as the specific task layer of the leaf number regression task.

3. The multi-task learning plant phenotype processing method based on VGG16 according to claim 1, characterized in that: The softmax layer applies the Softmax activation function to map the inputs of multiple neurons between [0,1], and the sum of the mapped elements in the vector is 1. The formula for calculating its value is: In the above formula, i and j are the class labels in the multi-classification problem, and x i Represents the i-th category value in the input x value, Represents the sum of all category value indices.

4. The multi-task learning plant phenotype processing method based on VGG16 according to claim 1, characterized in that: The formula of the Relu activation function is: Relu(x)=max(0,x) In the above formula, x is the input value, and the Relu function takes the maximum value between 0 and x. Therefore, when the input is negative, the function is not activated and is not applicable to the case where all inputs are negative.

5. The multi-task learning plant phenotype processing method based on VGG16 according to claim 1, characterized in that: When learning the parameters of the weight relationship factors of the model, the weight-related parameters are combined together according to formula (2.5) to form a joint loss as a prerequisite, and then the weight-related parameters are gradually updated in the iterative rounds. Finally, a combination of a weight parameter matrix and multiple task weight-related flexible factors is learned, so that the loss function of multiple tasks reaches the minimum value, and the influence of each task on the model training is in a balanced state.

6. A multi-task learning plant phenotype processing device based on VGG16, characterized in that include: Model building module, used to build a plant phenotypic processing and analysis model. The constructed model uses the VGG16 network as a hard parameter shared network for multi-task learning. The last maximum pooling layer and all subsequent fully connected layers of the original VGG16 are removed, and task-related fully connected layers are added. They are applied to three specific task layers respectively. Each specific task layer is composed of fully connected layers, and each specific task layer completes its own specific feature extraction task. The three specific task layers are: genotype classification task, leaf number regression task, and leaf area regression task. After the plant phenotype image is input into the model, the corresponding feature vector is extracted through the basic shared network VGG16 and the specific task fully connected layer of each task. The genotype classification task uses the softmax layer to multi-classify the feature vector to obtain the probability of each category. The leaf number regression task and the leaf area regression task apply the Relu activation function in the specific task fully connected layer to obtain the final prediction result. The model training module is used to automatically learn and train the constructed plant phenotyping and analysis model. It uses task-dependent uncertainty to weigh the loss function of each specific task layer to balance the loss of regression and classification tasks. It uses flexible factors to connect different tasks and automatically optimize the weights of different tasks so that each task can be optimally trained. The regression and classification task losses are derived from the maximum Gaussian likelihood that maximizes the task-dependent uncertainty: (2.1) Likelihood function for regression task The input is x, the true label is y, the parameter matrix is ​​W, and the output is f W (x), for the regression task, its probability is defined as the Gaussian likelihood with the output as the mean, that is: p(y|f W (x))=N(f W (x),σ 2 )(2.1) In the Gaussian likelihood shown in formula (2.1), σ is the observation noise parameter of the model; based on this Gaussian likelihood, the logarithmic likelihood of the regression task in multi-task joint learning can be obtained: (2.2) Likelihood function for classification tasks In the classification task, a standard softmax layer is used, and the Gaussian likelihood of its probability is: p(y|f W (x))=Softmax(f W (x))(2.3) In formula (2.3), the noise parameter σ is introduced as a scaling factor. This parameter is related to the uncertainty of the distribution and determines the flatness of the discrete distribution. The logarithmic likelihood of the classification task is finally obtained as shown in the following formula (2.4), where c represents the number of classifications; (2.3) Multi-task joint loss According to the above formula, adaptive weights based on loss are constructed for different tasks, and different tasks are connected through flexible factors to balance the influence of each task. Under the condition of maximizing the log likelihood, that is, minimizing the negative log likelihood, the joint loss of multiple tasks is constructed: In formula (2.5), y1, y2, and y3 are the true labels of the leaf area regression task, leaf number regression task, and genotype classification task respectively; correspondingly, σ1, σ2, and σ3 are the noise parameters of the leaf area regression task, leaf number regression task, and genotype classification task respectively; L1(W) = || y1-f W (x)|| 2 , represents the Euclidean distance between the leaf area regression subtask output and the true label y1; L2(W) represents the Euclidean distance between the leaf number regression subtask output and the true label y2; L3(W) = -log(Softmax(f W (x))) is the cross entropy loss for the classification subtask; At the same time, the flexible factors of the leaf area regression task, leaf number regression task, and genotype classification task are α, β, and γ, respectively. The values ​​of the three flexible factors fluctuate within the range of (0, 1) and satisfy α+β+γ=1. This allows the model to continuously learn and adjust the loss of each subtask to minimize the multi-task loss during the training process. At the same time, it can also control the impact of each task on the model to achieve a balanced state. During the model training process, the model optimizer will automatically update the network parameters, including the flexible factor values ​​of each task, according to the gradient information of the network back propagation, in order to reduce the calculated value of the loss function; if after the previous round of iterative training, the joint loss value of the model is calculated, the leaf area loss accounts for a large proportion of the total loss, that is, in formula (2.5) If the value is large, it means that the prediction effect of the leaf area task is not good. Then the optimizer will choose to train the model in a direction that is beneficial to the leaf area regression task in the next iteration, and reduce the flexibility factor of the leaf area regression task, that is, α in formula (2.5), reduce the proportion of the task loss, and appropriately adjust β and γ to meet α+β+γ=1, so that the proportion of each task loss to the total joint loss is similar; after the next round of model training is completed, the leaf area loss value decreases, the weight decreases, and the proportion of the total loss decreases. Other tasks with a higher proportion of the total loss value can be selected for training; if the loss of the genotype classification task accounts for a large proportion of the total loss, the genotype classification task will dominate the training, and the flexibility factor of the genotype classification task will be reduced, and the flexibility factor values ​​of other tasks will be adjusted; for the leaf number regression task, the same is true. After each model training, the weight parameters and flexibility factors of each task are adjusted so that in the process of continuous model training, the influence of each task on the model training reaches a balanced state; The image recognition module is used to receive the plant image to be recognized, and use the trained model to recognize and analyze the plant image to be recognized.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the plant phenotyping method according to any one of claims 1 to 5 is implemented.

Images