Automatic labeling method and device for crop image

Abstract

The invention provides a crop image automatic labeling method and device, and relates to the technical field of deep learning. The method comprises the following steps: acquiring a training set, a test set and a verification set of crop images; on the basis of the training set, according to the unfreezing sequence of the neural network model from a high layer to a low layer, in the ith round of training, freezing parameter training of each network layer before the jth group of network layers, training parameters in the jth group of network layers, and adjusting parameters of network layers after the jth group of network layers; after the parameter training of the jth group of network layers and the subsequent network layers is completed, starting to train the parameters of the (j-1) th group of network layers and adjusting the parameters of the network layers subsequent to the (j-1) th group of network layers; and after the parameters of all the network layers in the ith round are trained, if the model performance indexes verified based on the test set and the verification set meet preset conditions, labeling each crop image in the to-be-labeled set based on the trained neural network model. Based on the scheme, the accuracy of crop image automatic labeling can be improved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a method and device for automatically labeling crop images. Background Art

[0002] With the development of agricultural information technology, data-driven agricultural applications have become an important tool for improving production efficiency, optimizing resource utilization and improving agricultural decision-making.

[0003] Typically, crop image datasets can be used as basic data for training and validating models in agricultural applications. Generally speaking, different types of crops have significant differences in morphology, color, structure, and phenotypic characteristics, and a large number of crops perform differently in different growth periods or under different growth environments. Crop image datasets usually need to cover crop images from multiple regions and seasons, so it is difficult to annotate agricultural images.

[0004] The commonly used method at present is manual labeling. However, in the manual labeling process, there are subjective differences between different people and the labeling efficiency is low. The long manual labeling time can easily lead to low labeling accuracy. Summary of the invention

[0005] The purpose of the present invention is to provide a method and device for automatically labeling crop images in order to improve the accuracy of labeling.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides an automatic annotation method for crop images, the method comprising: obtaining a training set, a test set and a validation set of crop images; the training set and the test set include manually annotated position information of each crop in the crop image and label information of each crop; based on the training set, in accordance with the unfreezing order of the neural network model from high level to low level, in the i-th round of training, freezing the parameter training of each network layer before the j-th group of network layers, training the parameters in the j-th group of network layers, and adjusting the parameters of the network layers after the j-th group of network layers; the j-th group of network layers includes at least one network layer in the neural network model except the input layer and the output layer. network layer, i and j are both integers greater than or equal to 1; after the parameter training of the jth group of network layers and subsequent network layers is completed, the parameters of the j-1th group of network layers are trained, and the parameters of the network layers after the j-1th group of network layers are adjusted; after the parameter training of all network layers in the i-th round is completed, the model performance indicators of automatic labeling of crop images are verified based on the learned characteristics of each crop, the weight value of the characteristics of each crop, the test set and the validation set; if the model performance indicators verified based on the test set and the validation set meet the preset conditions, each crop image in the to-be-labeled set is labeled based on the trained neural network model.

[0008] In a second aspect, an embodiment of the present invention provides an automatic annotation device for crop images, which includes: a data acquisition module, a training module, a verification module and an automatic annotation module; the data acquisition module is used to acquire a training set, a test set and a verification set of crop images; the training set and the test set include manually annotated position information of each crop in the crop image and label information of each crop; the training module is used to freeze the parameter training of each network layer before the jth group of network layers in the i-th round of training according to the unfreezing order of the neural network model from high level to low level based on the training set, train the parameters in the jth group of network layers, and adjust the parameters of the network layers after the jth group of network layers; the jth group of network layers includes all network layers in the neural network model except At least one network layer in the network layers other than the input layer and the output layer, i and j are both integers greater than or equal to 1; and after the parameter training of the j-th group of network layers and subsequent network layers is completed, the parameters of the j-1-th group of network layers are trained, and the parameters of the network layers after the j-1-th group of network layers are adjusted; a verification module is used to verify the model performance indicators of automatically annotating crop images based on the learned characteristics of each crop, the weight values ​​of the characteristics of each crop, the test set and the verification set after the parameter training of all network layers in the i-th round is completed; an automatic annotation module is used to annotate each crop image in the to-be-annotated set based on the trained neural network model if the model performance indicators verified based on the test set and the verification set meet the preset conditions.

[0009] According to a third aspect of an embodiment of the present invention, a computer device is provided, including: a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any of the above methods for automatically labeling crop images are implemented.

[0010] According to a fourth aspect of the embodiments of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the automatic labeling method for crop images as described in any one of the above items are implemented.

[0011] The automatic annotation method of crop images provided by an embodiment of the present invention first obtains a training set, a test set and a validation set of crop images; then, based on the training set, in the order of unfreezing the neural network model from high level to low level, in the i-th round of training, freezes the parameter training of each network layer before the j-th group of network layers, trains the parameters in the j-th group of network layers, and adjusts the parameters of the network layers after the j-th group of network layers; secondly, after the parameter training of the j-th group of network layers and the subsequent network layers is completed, starts to train the parameters of the j-1-th group of network layers, and adjusts the parameters of the network layers after the j-1-th group of network layers; then, after the parameter training of all network layers in the i-th round is completed, based on the learned features of each crop, the weight values ​​of the features of each crop, the test set and the validation set, the model performance index of the automatic annotation of crop images is verified; finally, if the model performance indexes verified based on the test set and the validation set meet the preset conditions, then annotates each crop image in the set to be annotated based on the trained neural network model. Through the method of hierarchical fine-tuning, the local network layer will be continuously iterated during the model training process, so that the parameters of each layer from the high layer to the low layer can reach the global optimal. Unfreezing the layer by layer can make the model gradually adapt to the characteristics of the recognition task, and can avoid large changes in model weight values ​​and unstable training due to one-time training of all layers. The model can transition to the new data distribution more smoothly. The accuracy is higher than that of the directly trained model, and the annotation accuracy is high. For different types of plant image data, it can avoid the problem of overfitting and the complexity of the model is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A schematic flow chart of an automatic annotation method for crop images provided in an embodiment of the present invention.

[0013] Figure 2 A logical schematic diagram of model fine-tuning provided in an embodiment of the present invention.

[0014] Figure 3 A schematic diagram of a layer-level fine-tuning process of a neural network model provided in an embodiment of the present invention.

[0015] Figure 4A logical schematic diagram of incremental training provided by an embodiment of the present invention.

[0016] Figure 5 A schematic structural diagram of an automatic crop image labeling device is provided in accordance with an embodiment of the present invention.

[0017] Figure 6 A hardware structure diagram of a computer device in which the automatic annotation system for crop images provided by an embodiment of the present invention is located. DETAILED DESCRIPTION

[0018] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0019] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "an", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0020] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0021] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0022] Figure 1 A schematic diagram of a process flow of an automatic annotation method for crop images provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown in , the following S101 to S105 may be included:

[0023] S101, obtaining a training set, a test set, and a validation set of crop images.

[0024] The crop images in the training set and the test set include the position information of each crop in the crop image and the label information of each crop. The position information is used to indicate the position of the target box of the labeled crop in the crop image. The label information indicates the crop type and crop life stage.

[0025] Exemplarily, the position information may be coordinate information of the target frame, and the shape of the target frame may be set as required, which is not specifically limited in the embodiment of the present invention.

[0026] It should be noted that during the model training process, the label information of the manually labeled training set and test set includes the crop type and the crop life stage. Optionally, after the model training is completed, the automatic labeling may only include the crop type in the label information, or may include the crop type and the crop life stage, which is not specifically limited in the embodiments of the present invention.

[0027] It can be understood that the crop type may include a crop type and sub-types under the crop type, for example, flower-orchid type 1-mid-term.

[0028] Optionally, before training a neural network model based on manually annotated crop images, the automatic annotation system for crop images may first perform image preprocessing on the crop images used in the neural network model training process, and the image preprocessing process includes: a manual processing process and a model preprocessing process. The manual processing process includes manually selecting a representative crop image set from the acquired crop image data set, and dividing the selected crop image set into a training set for crop image annotation training, a test set for crop image annotation training, and a validation set for crop image annotation training. The model preprocessing process includes data cleaning, image denoising, and image enhancement of the images in the training set. After image preprocessing, label information is manually annotated on the crop images in the training set and the test set using an annotation tool.

[0029] For example, manual labeling can be performed using labeling tools such as CVAT (Computer Vision Annotation Tool), ScaLabel, MakeSense, and LabelImg.

[0030] It should be noted that during the training process, multiple types of crops can be trained at the same time. For example, images of flowers and images of trees can be grouped as different types of images into crop types and the model can be trained at the same time.

[0031] S102. Based on the training set, in the unfreezing order of the neural network model from high layer to low layer, in the i-th round of training, freeze the parameter training of each network layer before the j-th group of network layers, train the parameters in the j-th group of network layers, and adjust the parameters of the network layers after the j-th group of network layers.

[0032] The j-th group of network layers includes at least one network layer in the network layers other than the input layer and the output layer in the neural network model, and i and j are both integers greater than or equal to 1.

[0033] Exemplarily, the neural network model used in the embodiments of the present invention may be a convolutional neural network model based on deep learning. For example, CNN (Convolutional Neural Networks), ResNet (Residual Neural Network), Efficient Net, YOLO (You Only Look Once) series, Fast R-CNN (Fast Regional Convolutional Neural Network), SSD (Single Shot MultiBox Detector), Mask R-CNN (Mask Region-based Convolutional Neural Network), etc.

[0034] It can be understood that the order of the neural network model from the input layer to the output layer is the order from the low-level network to the high-level network. Among the above-mentioned various types of neural network models, most of the neural network models require parameter training except the input layer and the output layer. The training of the traditional model is to conduct global training from the low layer to the high layer to determine the performance of the overall training of this round. If the performance is not good, the parameters of all layers are re-trained globally. In an embodiment of the present invention, the parameter training of the low-level network is frozen first, and the parameter training is unfrozen from the high-level network to the low-level network layer by layer, that is, the parameter training of the high-level local network layer is first performed, and the weight values ​​of the parameters of the currently trained local network layer and the weight values ​​of the parameters of the trained network layer are comprehensively adjusted. After the parameter training of the local network layer in this round, the lower network layer is unfrozen for parameter training.

[0035] S103: After the parameter training of the jth group of network layers and subsequent network layers is completed, start training the parameters of the j-1th group of network layers, and adjust the parameters of the network layers subsequent to the j-1th group of network layers.

[0036] Specifically, in the training of the neural network model, the model training can be guided by minimizing the loss function. When the jth group of network layers is trained, the loss value of the parameters obtained by the training of the jth group of network layers is determined for automatic labeling. If the loss value is greater than the preset loss value, it can be determined that the learned crop characteristics, the weight values ​​of the crop characteristics, the hyperparameters, etc. do not meet the loss constraints, and the loss value of the jth group of network layers can be back-propagated to each network layer of the lower layer of the jth group of network layers to adjust the loss. If the loss value is less than or equal to the preset loss value, it can be determined that the learned crop characteristics, the weight values ​​of the crop characteristics, the hyperparameters, etc. meet the loss constraints, and the training of the parameters at the lower level can be started.

[0037] It can be understood that in the embodiment of the present invention, the neural network model can be fine-tuned at the hierarchical level, starting from the last layer of the network, first freezing the low-level feature extraction layer of the neural network model, first training the parameters of the high-level feature extraction layer of the neural network model, and gradually unfreezing and training from the high-level to the low-level, so that the neural network model can fully learn the high-level features in the crop image, while ensuring the stability of the low-level features, which can improve the stability and accuracy of the neural network model when learning crop features. After the parameter training of each group of network layers is completed, the parameter training of the next group of network layers is gradually unfrozen, and the weight values ​​of the high-level features and the low-level features can be jointly optimized during the iterative training process, which can improve the effectiveness of ensuring hierarchical fine-tuning during model evaluation.

[0038] S104, after the parameters of all network layers in the i-th round are trained, based on the learned features of each crop, the weight values ​​of the features of each crop, the test set and the validation set, the model performance indicators of the automatic annotation of crop images are verified.

[0039] In the embodiment of the present invention, the neural network model will traverse each layer of the model after each round of training (all layers are trained), that is, each layer of the model is used. Since each network layer has a different function, during the training process, each round of training enables the model to learn the characteristics of the crop image automatic labeling task as much as possible (unfreeze layer by layer to achieve local optimum to global optimum). After multiple rounds of training, the parameters of each layer of the model can reach the global optimum as much as possible, so that the performance of the trained model is optimized, so that the automatic labeling based on the trained neural network model is faster and more accurate.

[0040] Specifically, in an embodiment of the present invention, after all layers of each round are unfrozen, the overall model can be tuned, the data between the layers can be comprehensively optimized to make the data transition smooth, check the average loss value of the overall model for data training, calculate the change in the loss value of the last p rounds, and measure it with the variance. When it is less than a certain fixed value, it can be considered that the loss has stabilized. At the same time, the P (precision) curve, PR (Precision-Recall Curve) curve and F1 Score (i.e., the above-mentioned model performance indicators) of the model training will be combined to carry out the next self-labeling training strategy. If all reach the preset values, the layer fine-tuning task is terminated, and you can choose to use the model directly to label the data to be labeled.

[0041] For example, the difference between the trained neural network model and the manual annotation of crop images in the test set can be verified to determine the training accuracy of the neural network model. The trained neural network model in each round can be applied to the agricultural image data in the test set for annotation, and the agricultural images in the test set can be detected. The coordinate position corresponding to the target box with a confidence greater than the preset confidence can be output, and the model annotation results can be compared with the target box in the manually annotated label information to evaluate the performance of the trained neural network model. After the model evaluation, if the model does not meet the training end conditions, the learning rate and batch size of the model can be adjusted through Bayesian optimization to perform model hyperparameter tuning; the neural network model can monitor the performance of the model in the training set and test set, analyze the error distribution of the neural network model for each round of training on the data, determine the learning status of the crop characteristics of the trained neural network model, and complete the test set verification.

[0042] S105: If the model performance indicators verified based on the test set and the validation set meet the preset conditions, annotate each image in the set to be annotated based on the trained neural network model.

[0043] It should be noted that the images in the validation set may be crop sample images that have not undergone the above-mentioned image preprocessing and have not been manually labeled.

[0044] Specifically, if the model performance indicators verified by the test set meet the preset conditions, and then the model performance indicators verified based on the validation set meet the preset conditions, and the model performance indicators verified by the test set and the validation set meet the preset conditions, automatic labeling can be started.

[0045] For example, based on the validation set and the test set, by verifying the performance indicators of the model, the accuracy and robustness of learning can be evaluated, and the performance bottleneck can be automatically adjusted. When the mAP (mean Average Precision) of the model self-evaluation is greater than or equal to 98%, the model can enter the automatic labeling stage.

[0046] It can be understood that in an embodiment of the present invention, a completed round of global training may include: forward propagation → calculation of loss → back propagation → parameter update. Among them, the forward propagation stage: starting from the input layer, the input data is passed forward layer by layer through each network layer until the output layer. Each layer passes the current feature map (information mainly including weight values, biases, gradients, activation values, etc.) to the next layer, and the initial freezing of the lower layer network will keep the relevant parameters of the lower layer network unchanged, and pass it to the next layer with the initial value until the unfrozen layer is reached, and the relevant parameters of the unfrozen layer are updated (calculating gradients, etc. at high levels and updating parameters). At the same time, the training process will also have a back propagation stage, starting from the output layer, the gradient of the loss function for each layer parameter is calculated, and it is passed back to the lower layer at one time. When performing layer-by-layer fine-tuning, by freezing the lower layer parameters, the gradient stops passing when it reaches the frozen layer, thereby ensuring that the frozen layer is not updated, and the control parameters are only updated at high levels. After each round of unfreezing the new layer, back propagation will automatically calculate the gradient of the newly unfrozen layer, so that these layers can perform parameter updates and weight value optimization.

[0047] Specifically, after multiple rounds of optimization of the self-learning mechanism based on the above learning process, large-scale automatic annotation of unlabeled crop images can be started. The annotation system automatically generates the target box and the corresponding target box information, and generates a file (such as JSON, XML, YOLO or CSV format) containing the target box coordinates and the corresponding confidence level based on the results of the automatic annotation. Filter out data of specific types or specific conditions from the labeled data, construct different subsets of data sets (such as training sets, test sets, and validation sets) according to different proportions, and further organize them into standardized data sets for subsequent research and application.

[0048] Figure 2 A logical schematic diagram of a model fine-tuning provided by an embodiment of the present invention, such as Figure 2 As shown in , select the pre-selected model, freeze the low-level feature extraction layer of the model, gradually unfreeze the previous layers, fully learn the high-level features, and unfreeze the weight values ​​of each layer of the multi-layer fine-tuning at the same time to jointly optimize the high-level features and low-level features. Get the preliminary fine-tuning model, detect the test set data, output relevant information, and then compare it with the manual annotation results to evaluate the network model. Adjust the model learning rate through Bayesian optimization, perform hyperparameter tuning, analyze the error distribution of the model to the data, determine whether the annotation effect is good (determined by the above performance indicators) and whether the error has a small impact on the annotation results (determined by the above performance indicators). If the requirements are met, the expected model is obtained and the model fine-tuning is completed. If the requirements are not met, the low-level feature extraction layer of the model is re-frozen to start the model iterative training based on hierarchical fine-tuning.

[0049] In the automatic labeling method of crop images provided by the embodiment of the present invention, firstly, a training set, a test set and a validation set of crop images are obtained; then, based on the training set, in the order of unfreezing the neural network model from high level to low level, in the i-th round of training, the parameter training of each network layer before the j-th group of network layers is frozen, the parameters in the j-th group of network layers are trained, and the parameters of the network layers after the j-th group of network layers are adjusted; secondly, after the parameter training of the j-th group of network layers and the subsequent network layers is completed, the parameters of the j-1-th group of network layers are trained, and the parameters of the network layers after the j-1-th group of network layers are adjusted; then, after the parameter training of all network layers in the i-th round is completed, based on the learned features of each crop, the weight values ​​of the features of each crop, the test set and the validation set, the model performance index of the automatic labeling of crop images is verified; finally, if the model performance indexes verified based on the test set and the validation set meet the preset conditions, each crop image in the to-be-labeled set is labeled based on the trained neural network model. Through the method of hierarchical fine-tuning, the local network layer will be continuously iterated during the model training process, so that the parameters of each layer from the high layer to the low layer can reach the global optimal. Unfreezing the layer by layer can make the model gradually adapt to the characteristics of the recognition task, and can avoid large changes in model weight values ​​and unstable training due to one-time training of all layers. The model can transition to the new data distribution more smoothly. The accuracy is higher than that of the directly trained model, and the annotation accuracy is high. For different types of plant image data, it can avoid the problem of overfitting and the complexity of the model is low.

[0050] Optionally, in the automatic annotation method for crop images provided in the embodiment of the present invention, the above S102 may specifically include the following S102a:

[0051] S102a, if the loss value of the kth round of the jth group of network layers does not meet the loss constraint condition, then continue to freeze the parameter training of each network layer before the jth group of network layers, perform the k+1th round of parameter training of the jth group of network layers, and adjust the parameters of the network layers after the jth group of network layers.

[0052] Here, k is an integer greater than 1.

[0053] It can be understood that in the embodiment of the present invention, the parameter training of the network layer is gradually unfrozen, that is, the local training of the high-level features of the agricultural crop image is gradually transformed into the global training of the high-level features and low-level features of the agricultural crop image. In the process of parameter training of the local network layer starting from the high-level network, if the loss of crop labeling based on the parameters of the network layer trained in this round is greater than the preset loss, the parameters of the network layer trained in this round do not meet the iteration stop condition, and the parameters of the network layer trained in this round can continue to be iterated for the next round of training, and the parameters of all network layers that have been trained before the network layer trained in this round are adjusted (fine-tuned) at the same time in the next round of training.

[0054] It should be noted that the high-level features (semantic features) of crop images refer to the features obtained from the deeper layers of the neural network model. The high-level features of crop images contain rich semantic information and can characterize information such as the type of crop objects in the crop images. Specifically, the high-level features of crop images reflect the high-level semantic information of crop images, contain the key information required to identify different crop objects, and can be used for crop image classification tasks. The low-level features of crop images contain more detailed information and location information, and are used for the task of locating crops in crop images. For example, the low-level features of crop images can capture the contours, edges, colors, and textures of crop images, and the low-level features of crop images can be used to accurately locate crops in crop images.

[0055] For example, during the training thawing process, the crop image annotation system checks the change in the loss value during the freezing strategy training process. If it has been reduced to less than 80% of the initial loss value of the freezing strategy, the next network layer is thawn, and so on.

[0056] Based on this scheme, the training of the relevant parameters of the low-level network can be frozen in the preliminary stage, and the training of the parameters of a group of network layers can be gradually unfrozen from the high level based on the loss value. When training at the high level, a higher learning rate can be used for training, so that the semantic information in the crop image can be fully learned. In the intermediate stage, the high-level features can be fine-tuned while training, so that the model can adapt to the data distribution of the crop image automatic labeling task, and the parameters of the unfrozen layers can be continuously updated. In the final stage, the parameters of the entire model can be fine-tuned, and the parameters of all layers of the neural network can be updated. Since the adjustment of the parameters of each layer is automatically updated according to the features learned after training, the high-level features can be quickly adapted to the task requirements. Through gradual adjustment, the low-level features and high-level features can work together to achieve rapid joint optimization. In this way, the relevant parameters of the model that meet the requirements can be quickly obtained.

[0057] Optionally, in the automatic annotation method for crop images provided in the embodiment of the present invention, the above S103 may specifically include the following S103a:

[0058] S103a. If the loss value of the kth round of the jth group of network layers meets the loss constraint condition, continue to freeze the parameter training of each network layer before the j-1th group of network layers, start training the parameters of the j-1th group of network layers, and adjust the parameters of the network layers after the j-1th group of network layers.

[0059] It can be understood that in each round of overall training, when iterative training is performed on a group of network layers, if the loss value in the kth round satisfies the loss constraint, it can be determined that the parameters of the group of network layers have converged in the kth round of training, and the local training is completed, and then the training of the parameters before the j-2th group of network layers can be continued to be frozen, and the training of the parameters of the j-1th group of network layers can be started to be unfrozen.

[0060] It should be noted that when training the parameters of the j-th group of network layers, it is necessary to fine-tune the parameters of the network layers after the j-th group of network layers at the same time to ensure the stability of high-level features, that is, to ensure the accuracy of semantic information in the image.

[0061] Based on this scheme, it is possible to determine whether to unfreeze the new network layer for parameter training based on the loss constraint condition, thereby ensuring that the loss of the determined parameter weight value is minimized during labeling, reducing the probability of labeling errors and improving the accuracy of automatic labeling of crop images.

[0062] It should be noted that in the actual training process, some types of crop training samples are relatively few. When it is determined that the sample ratio does not meet the preset ratio, the neural network model can automatically expand the sample images of the type whose sample number is less than the preset threshold. For example, based on the crop images of this type, automatic image cropping, scaling, flipping and other operations are performed to expand one crop image into multiple crop images, so that the ratio of the expanded sample images of this type of crop and the sample images of other crop types meets the preset ratio.

[0063] Optionally, in the automatic labeling method for crop images provided in the embodiment of the present invention, the label information includes the crop type and the crop life stage, and after the above S105, the following S106 may be further included:

[0064] S106. If the accuracy of the model performance indicator verified by the verification set corresponding to the first category of crops in the i-th round of training is less than the preset accuracy, then add training images of the first category of crops, and continue to perform the i+1-th round of training based on the training set after adding the training images of the first category of crops.

[0065] It should be noted that when the proportion of training samples is unbalanced, for example, the amount of crop data under type 1 is small, after the training of the neural network model is completed, there may be insufficient feature learning for the crop data under type 1, and the crop features that effectively distinguish type 1 crops from other types of crops are not learned, resulting in low overall labeling accuracy.

[0066] Specifically, the accuracy of the annotation of the trained neural network model is verified through the validation set to verify whether the neural network model obtained in the i-th round of training has learned the characteristics of the crop data. The performance of the neural network model in the training set is determined based on the verification results of each type of crop in the validation set, and the data of crop images of types whose characteristics are not fully learned can be expanded.

[0067] Optionally, the newly added training images of the first type of crops may include at least one of the following types:

[0068] (1) Images resampled from the first category of crop images in the historical training set;

[0069] (2) images extracted from the set of crop images to be labeled in the first category;

[0070] The neural network model can also extract images from the to-be-labeled set of the first category of crops, annotate the extracted images with pseudo-labels, add the crop image data with the added pseudo-labels to the training set for retraining through a playback mechanism, continue to update the model parameters in a gradual thawing manner, and annotate the crop images in the to-be-labeled set of the first category of crops based on the updated neural network model.

[0071] It should be noted that a self-learning mechanism is provided in the embodiment of the present invention. The neural network model extracts crop images from the set to be labeled, and performs iterative training through automatic label generation and model prediction confidence, which can effectively expand crop features in incremental data. During training, the neural network model generates pseudo labels for the prediction results of unlabeled data, that is, the neural network model generates preliminary predictions for the data to be labeled, and when the confidence is higher than a certain threshold, these predictions are added to the training as pseudo labels.

[0072] (3) Sample images of the first-category crop images of more scenes acquired from the outside, for example, images of the first-category crop at different growth stages, different lighting conditions, and different occlusion conditions.

[0073] For example, the crop life stages annotated in the training images of the first category of crops after adding different growth stages are different from the crop life stages annotated in the training images of the first category of crops before adding.

[0074] Specifically, the training images of the newly added first type of crops and the training images in the historical training set are mixed to perform the i+1th round of training, which can ensure the uniform distribution of different feature data.

[0075] For example, the loss function of the model can be adjusted in the (i+1)th round of training based on the adjustment rule indicated by formula (1), which can reduce the loss of original feature information while learning new features.

[0076]

[0077] Among them, L task represents the standard loss for the current task, λ represents the regularization coefficient, and λ is used to control the balance between new features and old features. represents the L2 norm of all parameters.

[0078] By constraining the size of model parameters, drastic changes in parameters can be avoided and the coexistence of new and old features in the model parameter space can be ensured.

[0079] It can be understood that during the i+1th round of training, continuing to repeat the layer fine-tuning steps in the above steps can ensure the balance of the learned new features and the original features in the entire network.

[0080] Optionally, in each set of training, new data can be introduced in batches. For example, when training the first set of network layers, the first batch of new data is introduced, and when training the second set of network layers, the second batch of data that is not repeated with the first batch is introduced. The model learns the characteristics of the new data by fine-tuning layer by layer. The ratio of new and old data can be set, such as using 80% of the original data combined with 20% of the new data, to prevent the model from being overly biased towards the new data. Through incremental training, new data is passed to the parameters of each layer layer by layer to ensure the integration of new and old knowledge.

[0081] Based on this scheme, in the automatic labeling method of crop images provided by the embodiment of the present invention, when training the neural network model, for crop images whose features are not fully learned, training images for this type of crop can be supplemented according to the performance of the neural network model in the training set, the model can be updated by resampling the original data set, or the model can be retrained and updated by extracting a small number of images from the set to be labeled and introducing pseudo labels, or more scene samples can be introduced from the outside to update the model. For example, crop images at different growth stages and different lighting conditions are introduced to enrich the training samples of the model for this type of crop, so that the characteristics of each type of crop can be fully learned in the model, the accuracy of automatic labeling of this type of crop is improved, the workload of manual re-labeling of this type of crop can be reduced in the later stage, and the efficiency and accuracy of labeling of this type of crop can be improved.

[0082] Optionally, in the automatic annotation method for crop images provided by the embodiment of the present invention, after the above S104, incremental training may be started during the training process, and the above method may further include the following S107 to S109:

[0083] S107: After verifying the model performance index based on the target data set, mark the target image in the target data set as an incremental training sample image.

[0084] The target images include at least one type of images among crop images that fail to be labeled and crop images that succeed in being labeled and whose confidence is lower than a first confidence, and the target data set includes at least one of a test set and a validation set.

[0085] Specifically, the labeling system can output the confidence of each automatically labeled target box during the model performance verification process of model training. The neural network model marks the corresponding target image as an incremental training sample image and stores it in the first folder for the target box with relatively low confidence; determines whether to label the crop image through NMS (Non-Maximum Suppression); for example, if the threshold of NMS is set to 0.6, then when the neural network model labels the crop image, if the NMS of the detected crop image is less than or equal to 0.6, no label is added; if the NMS of the crop image is greater than 0.6, a label is added, and the crop image without a label is added to the first folder. Through manual review, the image data in the first folder manually labeled is the verification data that meets the requirements of the training data set.

[0086] It should be noted that in an embodiment of the present invention, if the first image in the first folder is not labeled, the target frame and label are manually labeled; if the first image is labeled, it is manually determined whether the label of the first image is accurately labeled, and if the label is accurately labeled, the target frame position is adjusted; if the label of the first image is incorrectly labeled, the label and target frame position of the first image are manually adjusted.

[0087] S108. After the i-th round of training is completed, if the incremental training conditions are met, the incremental training is started in the i+1-th round of training, and the review images of the incremental training crop sample images marked from the 1st round to the i-th round are added to the training set to continue training, and the weight values ​​of each crop feature are re-determined.

[0088] It should be noted that, in this solution, the review image is an image obtained by manually re-labeling the target image.

[0089] S109: Automatically adjust the weight value of each crop feature based on the re-determined weight value of each crop feature and the weight value of the parameter in the (i+1)th round of training.

[0090] Exemplarily, the feature weight value may be automatically adjusted based on the adjustment rule shown in the following formula (2).

[0091] L adaptive =ω t *L new +(1-ω t )*L replay Formula (2)

[0092] Among them, ωt Represents the weight value dynamically adjusted based on time t, which can be set according to the characteristic learning speed of the new data and the learning rate decay setting, L new Represents the re-determined loss value, L replay Represents the original loss value, L adaptive Represents the dynamically adjusted loss value.

[0093] Based on this scheme, a feedback channel can be automatically generated based on the annotation results of at least one item in the test set and the validation set and the actual annotation results. The crop image annotation system can automatically identify erroneous crop images and low-confidence crop images, and mark such crop images as key learning crop images. Such data can be added back to the training set to start incremental training. In the incremental training, features with higher weight values ​​among the extracted crop features are selected, and the weight values ​​of each crop feature used for the annotation task are automatically adjusted.

[0094] Optionally, in the automatic annotation method for crop images provided in the embodiment of the present invention, after the above S105, online self-learning (online incremental training) may be performed in the automatic annotation process of the crop images, which may include the following S110 to S112:

[0095] S110, adding the target image generated in the labeling process to the online self-learning incremental sample set.

[0096] The self-learning incremental samples include: failed labeling samples generated during the automatic labeling process and crop images that are successfully labeled and whose confidence level is lower than a first confidence level threshold.

[0097] It is understandable that when automatic labeling is performed based on a trained neural network model, there will be crop images that fail to be labeled and crop images that succeed in labeling but have a low confidence level in the labeling results.

[0098] It should be noted that, in the embodiment of the present invention, before performing online incremental training, the crop images in the online self-learning incremental sample set need to be manually re-labeled.

[0099] The neural network model can output the target image generated during the automatic labeling process to the user and instruct the user to manually label. After the user completes the manual labeling, online incremental training can be performed based on the manually labeled target image to re-optimize the model parameters.

[0100] S111, adding the images and manually labeled information in the manually labeled online self-learning enhanced sample set to the training set, starting the online incremental training, and adjusting the parameter weight values ​​of each layer of the neural network model online.

[0101] Optionally, when automatically annotating a batch of images, online incremental training may be started. This may be started once or multiple times and may be set according to requirements. This is not specifically limited in the embodiments of the present invention.

[0102] S112. Based on the adjusted parameter weight values ​​of each layer of the neural network model, continue to label the remaining images in the set to be labeled.

[0103] Exemplarily, the above-mentioned S112 may be performed when the first condition of online self-learning is satisfied.

[0104] For example, the first condition includes at least one of the following conditions 1-1 to 1-3:

[0105] Condition 1-1: The number of target images generated by automatic annotation is greater than or equal to a first number.

[0106] For example, when performing a labeling task, online incremental training can be started every time 200 target images are generated.

[0107] Condition 1-2: a ratio of the number of target images generated by automatic annotation to the number of images for which annotation has been performed is greater than or equal to a first value.

[0108] For example, if 10% of the images for which the annotation task has been performed are target images, an online incremental training can be started.

[0109] Condition 1-3: the ratio of the number of images that have been labeled to the total number of images in the original set to be labeled is greater than or equal to a second value.

[0110] For example, when 50% of the labeling task is completed, online incremental training is started once, and when 75% of the automatic labeling task is completed, online incremental training is started again.

[0111] Condition 1-3: The time for executing automatic marking is greater than the first preset time.

[0112] For example, incremental training is started every hour after the labeling task is executed.

[0113] Based on this solution, during the automatic labeling process, online incremental training can be started based on images that failed to be labeled and images whose confidence in the labeling results is lower than the confidence threshold. Such images can be manually re-labeled and then incremental training can be performed on such manually labeled images. This allows model optimization to be performed based on automatically labeled crop images, thereby improving the accuracy of the model's automatic labeling.

[0114] Optionally, in the automatic annotation method for crop images provided in the embodiment of the present invention, after the above S105, the following S113 may be further included:

[0115] S113. In the labeling process, the crop images whose confidence level is greater than the second confidence level in the labeling results are added to the historical training set, and retraining is performed based on the added training set to adjust the weight values ​​of the parameters of each layer of the neural network model.

[0116] Exemplarily, the above-mentioned S114 may be performed when the second condition of online self-learning is satisfied.

[0117] The second condition includes at least one of the following conditions 2-1 to 2-3:

[0118] Condition 2-1: The total number of images for which the trained neural network model has performed labeling tasks is greater than or equal to the third number.

[0119] Condition 2-2: The total number of target images automatically labeled based on the trained neural network model is greater than or equal to the fourth number.

[0120] Condition 2-3: The time for executing automatic marking is greater than the first preset time.

[0121] Based on this scheme, in the process of automatic labeling of crop images based on a neural network model trained with a historical training set, crop images with higher confidence in the labeling results can be added to the historical training set, and online incremental training can be started, which can further optimize the performance of the neural network model for automatic labeling of crop images.

[0122] Optionally, in the automatic annotation method for crop images provided in the embodiment of the present invention, after the above S105, the following S114 to S116 may be further included:

[0123] S114. In the case of a newly added training data set, if the label information in the newly added training data set indicates that the crop type is an untrained target subtype in the trained second crop type, based on the first data volume of the newly added training data set, historical training data corresponding to the trained subtype in the second crop type of the second data volume is extracted according to a preset ratio.

[0124] For example, the data volume ratio of the new training data set and the historical training data set is 8:2.

[0125] It can be understood that after training large types of crop images, when adding sub-types of crops for labeling, if the labeling is performed directly based on the previously trained model, the sub-type of crops may be easily labeled as other sub-types of crops. For example, the features of orchids of types 1 to 10 have been learned, and a new type 11 orchid needs to be automatically labeled. At this time, if automatic labeling is performed based on the previously trained neural network model, it is easy to be labeled as types 1 to 10, or even may not be recognized as orchids; if training is performed directly on type 11 orchids, the model may forget the features of orchids of types 1 to 10, which will lead to a decrease in the accuracy of orchid labeling in the later stage.

[0126] S115. Mix the newly added training data set and the extracted historical training data to form an updated training data set.

[0127] S116. Based on the updated training data set and the parameter weight values ​​of each layer of the neural network model, retrain to obtain the crop characteristics of the target subtype and the parameter weight values ​​of each layer of the neural network model.

[0128] It should be noted that, in an embodiment of the present invention, each time a new type of training data enters the model, historical training data can be randomly inserted to supplement it, and a playback mechanism can be used to periodically play back part of the historical training data to ensure that the original features are not lost or forgotten when training the newly added crop type image data.

[0129] Specifically, the loss function of the above formula (1) can be used to adjust the balance between new and old features.

[0130] Based on this scheme, after the trained neural network model is obtained based on the historical training data, when training the new type of crop image data, the large type of the newly added crop image can be determined based on the manually annotated annotation information of the newly added training set. If the large type is the same and the small type is different, the playback mechanism can be used for model training when training the newly added type of crop image, that is, the historical training data of crops related to the type can be randomly inserted into the new training set to supplement the data and mix the new and old data. During training, the new and old features can be balanced by adjusting the loss function to make the feature extraction continuous, which can prevent the model from losing the old data to extract the features of other types of crops when receiving new image data for training, and ensure that the model can retain the obtained features and learn new features at the same time. The latest model output results are compared with the manually annotated results, the differences are analyzed and the training strategy of the model is adjusted. Finally, the new type of crop data is annotated based on the optimized model, which improves the generalization ability of the model and avoids overfitting.

[0131] Example:

[0132] Figure 3 A schematic diagram of a layer fine-tuning process of a neural network model provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown in the figure, the YOLOv7 network includes the following layers from low-level network to high-level network: input layer, CBS, pooling layer, residual block, up sample layer, feature fusion layer, detection head and output layer. CBS (Convolutional Block with Squeeze-and-Excitation) includes: convolutional layer, batch normalization layer and activation function layer.

[0133] In an embodiment of the present invention, the training of the parameters of the network layer can be gradually unfrozen in the order from the high-level network to the low-level network. Among them, the crop image data in the training set is input into the YOLOv7 network through the input layer of the YOLOv7 network. The CBS module of the YOLOv7 network extracts the basic features of the crop image data in the training set. Specifically, the CBS module is a basic module in the YOLOv7 network architecture, wherein the convolution operation is performed through the convolution layer of the YOLOv7 network to extract local features in the crop image, the batch normalization layer of the YOLOv7 network is used to accelerate model training, stabilize model convergence, and introduce nonlinear features through the ReLU (Rectified Linear Unit) activation function of the YOLOv7 network activation function layer. The pooling layer of the YOLOv7 network downsamples the image data in spatial latitude to reduce the size of the feature map, retain important features, and reduce the amount of calculation of the model. The residual block (ResNet Block) of the YOLOv7 network uses residual connection operations to effectively avoid the gradient vanishing problem in the deep network, which can enable the model to train a deeper network to capture more complex features. The upsampling layer of the YOLOv7 network upsamples the feature map to restore the size of the reduced feature map in the pooling layer and the convolution layer, thereby improving the detection resolution of the model. The feature fusion layer of the YOLOv7 network fuses features at different levels so that the model can comprehensively utilize high-level semantic features and low-level detail features. The feature fusion layer of the YOLOv7 network includes FAN (Feature Aggregation Network) and PAN (Path Aggregation Network), which are commonly used architectures for multi-scale feature fusion, which can enhance the model's ability to recognize multi-scale targets. In the embodiment of the present invention, it is used for the detection of plant species of various sizes and shapes in agricultural scenes. After feature fusion, the detection head of the YOLOv7 network predicts the final target (including bounding box regression tasks and classification tasks), and outputs the type and annotation position of the target. The output layer of the YOLOv7 network generates the final output result of the model.

[0134] Specifically, in the embodiment of the present invention, the low-level network of the YOLOv7 network includes: a convolution layer, a pooling layer and a residual layer, which are used to extract basic low-level features in the crop image. The high-level network of the YOLOv7 network includes an upsampling layer, a feature fusion layer and a monitoring head, which are used to extract abstract and high-level features.

[0135] The model parameters of each layer of the YOLOv7 network can be trained layer by layer in the order of unfreezing from high to low layers. When the jth group of network layers are trained, the parameters of the layers before the jth group of network layers are not trained. After the parameters of the jth group of network layers converge, the parameters of the j-1th group of network layers are trained. For example, when training the detection head, the CBS, pooling layer, residual block, upsampling layer and feature fusion layer before the detection head are frozen. The frozen network layer will transfer the extracted feature map to the high-level network in sequence until the detection head, and start training the parameters of the detection head. After the parameter training of the detection head is completed, the feature fusion layer is unfrozen. The frozen network layer will transfer the extracted feature map to the high-level network in sequence until the feature fusion layer, train the parameters of the feature fusion layer, and adjust the parameter values ​​of the trained feature fusion layer and the detection head until the parameter values ​​of all network layers are adjusted after the CBS training is completed.

[0136] Figure 4 A logical diagram of incremental training provided by an embodiment of the present invention, such as Figure 3 As shown in the figure, the annotation results of the test set and the validation set are compared with the actual annotation results; the model automatically generates a feedback channel; marks the incremental learning data: incorrectly labeled and low-confidence samples; obtains samples in more environmental scenarios from the outside; adds these two types of samples to the training set and starts incremental training; automatically adjusts the model weight value and the threshold of the evaluation model performance indicator; evaluates the model accuracy and robustness; determines whether the mAP is greater than 98%. If it is greater than 98%, the expected model is obtained and the model fine-tuning is completed; if it is not greater than 98%, the incremental samples are re-obtained to start incremental training.

[0137] Corresponding to the above-mentioned method embodiments, the present invention also provides embodiments of a device and a terminal to which the device is applied.

[0138] Figure 5 A schematic diagram of a structure of an automatic annotation device for crop images is provided in an embodiment of the present invention. Figure 5As shown in the figure, the automatic annotation device 500 for crop images includes: a data acquisition module 501, a training module 502, a verification module 503 and an automatic annotation module 504; the data acquisition module 501 is used to obtain a training set, a test set and a verification set of crop images; the training set and the test set include the position information of each crop in the crop image and the label information of each crop that are manually annotated; the training module 502 is used to freeze the parameter training of each network layer before the jth group of network layers in the i-th round of training according to the unfreezing order of the neural network model from high level to low level based on the training set, train the parameters in the jth group of network layers, and adjust the parameters of the network layers after the jth group of network layers; the jth group of network layers includes all network layers in the neural network model except the input layer and the output layer. At least one network layer in the network layers other than the outgoing layer, i and j are both integers greater than or equal to 1; and after the parameter training of the j-th group of network layers and subsequent network layers is completed, the parameters of the j-1-th group of network layers are trained, and the parameters of the network layers after the j-1-th group of network layers are adjusted; a verification module 503 is used to verify the model performance index of automatically annotating crop images based on the learned characteristics of each crop, the weight value of the characteristics of each crop, the test set and the verification set after the parameter training of all network layers in the i-th round is completed; an automatic annotation module 504 is used to annotate each crop image in the to-be-annotated set based on the trained neural network model if the model performance indicators verified based on the test set and the verification set meet the preset conditions.

[0139] Optionally, the training module is specifically used for: if the loss value of the kth round of the jth group of network layers does not meet the loss constraint condition, then continue to freeze the parameter training of each network layer before the jth group of network layers, perform the k+1th round of parameter training of the jth group of network layers, and adjust the parameters of the network layers after the jth group of network layers, where k is an integer greater than 1.

[0140] Optionally, the training module is specifically used to: if the loss value of the kth round of the jth group of network layers does not meet the loss constraint condition, continue to freeze the parameter training of each network layer before the j-1th group of network layers, start training the parameters of the j-1th group of network layers, and adjust the parameters of the network layers after the j-1th group of network layers.

[0141] Optionally, the label information includes crop type and crop life stage; the acquisition module 501 is also used to add training images of the first category of crops if the accuracy of the model performance indicator verified by the verification set corresponding to the first category of crops in the i-th round of training is less than a preset accuracy; the training module is also used to continue to perform the i+1th round of training based on the training set after adding the training images of the first category of crops.

[0142] Optionally, the automatic labeling device for crop images also includes: a labeling module; a labeling module, used to label the target image in the target data set as an incremental training crop sample image after verifying the model performance index based on the target data set; a training module, further used to start incremental training in the (i+1) round of training after the i-th round of training is completed, if the incremental training conditions are met, add the incremental training crop sample images marked from the 1st round to the i-th round to the training set to continue training, and redetermine the weight value of each crop feature; and automatically adjust the weight value of each crop feature based on the redetermined weight value of each crop feature and the weight value of the parameter of the (i+1) round of training; wherein the target image includes at least one type of image from the crop image that failed to be labeled and the crop image that was successfully labeled and the confidence level is lower than the first confidence level, and the target data set includes at least one of a test set and a validation set.

[0143] Optionally, the marking module is also used to add the target image generated in the labeling process to the online self-learning incremental sample set; the acquisition module is also used to add the images in the manually re-labeled online self-learning incremental sample set and the manually labeled information to the training set, and the training module is also used to start online incremental training and adjust the parameter weight values ​​of each layer of the neural network model online; the automatic labeling module is also used to continue labeling the remaining images in the labeling set based on the adjusted parameter weight values ​​of each layer of the neural network model; wherein the self-learning incremental samples include: failed labeling samples generated in the automatic labeling process and crop images that are successfully labeled and have a confidence level lower than a first confidence threshold.

[0144] Optionally, the automatic labeling device for crop images also includes: an adding module; an adding module, used to add crop images with a confidence level greater than a second confidence level in the labeling results to a historical training set during the labeling process; and a training module, further used to retrain based on the added training set to adjust the parameter weight values ​​of each layer of the neural network model.

[0145] Optionally, the automatic labeling device for crop images also includes: a data mixing module; an acquisition module, which is also used to, in the case of a newly added training data set, if the label information in the newly added training data set indicates that the crop type is an untrained target subtype in the trained second crop type, then based on the first data amount of the newly added training data set, extract historical training data corresponding to the trained subtype in the second crop type of the second data amount according to a preset ratio; the data mixing module is used to mix the newly added training data set and the extracted historical training data into an updated training data set; the training module is also used to retrain and acquire crop characteristics of the target subtype and parameter weight values ​​of each layer of the neural network model based on the updated training data set and the parameter weight values ​​of each layer of the neural network model.

[0146] The automatic crop image labeling device provided by the embodiment of the present invention first obtains a training set, a test set and a verification set of crop images; then, based on the training set and in the unfreezing order of the neural network model from high level to low level, in the i-th round of training, freezes the parameter training of each network layer before the j-th group of network layers, trains the parameters in the j-th group of network layers, and adjusts the parameters of the network layers after the j-th group of network layers; secondly, after the parameter training of the j-th group of network layers and the subsequent network layers is completed, starts to train the parameters of the j-1-th group of network layers, and adjusts the parameters of the network layers after the j-1-th group of network layers; then, after the parameter training of all network layers in the i-th round is completed, based on the learned features of each crop, the weight values ​​of the features of each crop, the test set and the verification set, the model performance index of the automatic crop image labeling is verified; finally, if the model performance indexes verified based on the test set and the verification set meet the preset conditions, then each crop image in the to-be-labeled set is labeled based on the trained neural network model. Through the method of hierarchical fine-tuning, the local network layer will be continuously iterated during the model training process, so that the parameters of each layer from the high layer to the low layer can reach the global optimal. Unfreezing the layer by layer can make the model gradually adapt to the characteristics of the recognition task, and can avoid large changes in model weight values ​​and unstable training due to one-time training of all layers. The model can transition to the new data distribution more smoothly. The accuracy is higher than that of the directly trained model, and the annotation accuracy is high. For different types of plant image data, it can avoid the problem of overfitting and the complexity of the model is low.

[0147] The embodiments of the crop image automatic labeling method of the present invention can be applied to a computer device, such as a server or a terminal device. The method embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the processor of the crop image automatic labeling device reads the corresponding computer program instructions in the non-volatile memory into the memory and runs them. From the hardware level, if Figure 6 FIG. 1 is a hardware structure diagram of a computer device in which the automatic annotation system for crop images provided by an embodiment of the present invention is located. Figure 6 In addition to the processor 610, memory 630, network interface 620, and non-volatile memory 640 shown, the server or electronic device where the device 631 is located in the embodiment may also include other hardware according to the actual function of the computer device, which will not be described in detail.

[0148] Correspondingly, the present invention also provides an automatic labeling device for crop images, which includes a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to: execute the steps of the automatic labeling method for crop images executed by the above-mentioned controller.

[0149] In one embodiment, a computer device is provided, including: a memory and a processor, wherein the memory stores a computer program, and the processor implements any step of the above-mentioned automatic crop image labeling method when executing the computer program.

[0150] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any step in the above-mentioned method for automatically labeling crop images can be implemented.

[0151] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0152] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Those of ordinary skill in the art can understand and implement it without paying creative labor.

[0153] The above describes specific embodiments of the present invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0154] Those skilled in the art will readily appreciate other embodiments of the invention after considering the specification and practicing the invention claimed herein. The invention is intended to cover any variations, uses or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not claimed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0155] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

[0156] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for automatic annotation of crop images, characterized in that: The method comprises: Acquire a training set, a test set and a validation set of crop images; the training set and the test set include manually annotated position information of each crop in the crop image and label information of each crop; Based on the training set, in the order of unfreezing the neural network model from high layer to low layer, in the i-th round of training, freeze the parameter training of each network layer before the j-th group of network layers, train the parameters in the j-th group of network layers, and adjust the parameters of the network layers after the j-th group of network layers; the j-th group of network layers includes at least one network layer in the network layers other than the input layer and the output layer in the neural network model, and i and j are both integers greater than or equal to 1; After the parameter training of the j-th group of network layers and subsequent network layers is completed, start training the parameters of the j-1-th group of network layers, and adjust the parameters of the network layers subsequent to the j-1-th group of network layers; After the parameters of all network layers in the i-th round are trained, based on the learned features of each crop, the weight values ​​of the features of each crop, the test set and the validation set, verifying the model performance index of the automatic annotation of crop images; If the model performance indicators verified based on the test set and the validation set both meet preset conditions, each crop image in the to-be-annotated set is annotated based on the trained neural network model.

2. The method according to claim 1, characterized in that: The freezing of parameter training of each network layer before the jth group of network layers, training of parameters in the jth group of network layers, and adjustment of parameters of network layers after the jth group of network layers include: If the loss value of the kth round of the jth group of network layers does not meet the loss constraint condition, then continue to freeze the parameter training of each network layer before the jth group of network layers, perform the k+1th round of parameter training of the jth group of network layers, and adjust the parameters of the network layers after the jth group of network layers, where k is an integer greater than 1.

3. The method according to claim 2, characterized in that After the parameter training of the j-th group and subsequent network layers is completed, starting to train the parameters of the j-1-th group of network layers, and adjusting the parameters of the network layers subsequent to the j-1-th group of network layers, comprises: If the loss value of the kth round of the jth group of network layers does not meet the loss constraint condition, continue to freeze the parameter training of each network layer before the j-1th group of network layers, start training the parameters of the j-1th group of network layers, and adjust the parameters of the network layers after the j-1th group of network layers.

4. The method according to claim 1, characterized in that: The tag information includes the crop type and the crop life stage, and the method further includes: If the accuracy of the model performance indicator verified by the verification set corresponding to the first category of crops in the i-th round of training is less than the preset accuracy, then the training images of the first category of crops are added, and the i+1-th round of training is continued based on the training set after the training images of the first category of crops are added.

5. The method according to claim 1, characterized in that The method further comprises: After verifying the model performance indicator based on the target data set, marking the target image in the target data set as an incremental training crop sample image; After the i-th round of training is completed, if the incremental training condition is met, the incremental training is started in the i+1-th round of training, and the incremental training crop sample images marked from the 1st round to the i-th round are added to the training set to continue training, and the weight value of each crop feature is re-determined; Automatically adjusting the weight values ​​of the various crop features based on the re-determined weight values ​​of the various crop features and the weight values ​​of the parameters of the (i+1)th round of training; The target images include at least one type of images among crop images that fail to be labeled and crop images that succeed in being labeled and whose confidence is lower than a first confidence, and the target data set includes at least one of the test set and the validation set.

6. The method according to any one of claims 1 to 5, characterized in that After labeling each image in the set to be labeled, the method further includes: Add the target image generated during the annotation process to the online self-learning incremental sample set; Adding the manually re-labeled images and manually labeled information in the online self-learning incremental sample set to the training set, starting online incremental training, and adjusting the weight values ​​of the parameters of each layer of the neural network model online; Based on the adjusted parameter weight values ​​of each layer of the neural network model, continue to label the remaining images in the set to be labeled; The self-learning incremental samples include: failed labeling samples generated in the automatic labeling process and crop images that are successfully labeled and whose confidence is lower than a first confidence threshold.

7. The method according to claim 6, characterized in that After labeling each image in the set to be labeled, the method further includes: During the labeling process, the crop images with confidence greater than the second confidence in the labeling results are added to the historical training set, and retraining is performed based on the added training set to adjust the weight values ​​of the parameters of each layer of the neural network model.

8. The method according to claim 7, characterized in that: After labeling each image in the set to be labeled, the method further includes: In the case of a newly added training data set, if the label information in the newly added training data set indicates that the crop type is an untrained target subtype in the trained second crop type, then based on the first data volume of the newly added training data set, historical training data corresponding to the trained subtype in the second crop type of the second data volume are extracted according to a preset ratio; Mixing the newly added training data set and the extracted historical training data into an updated training data set; Based on the updated training data set and the parameter weight values ​​of each layer of the neural network model, retraining is performed to obtain the crop characteristics of the target subtype and the parameter weight values ​​of each layer of the neural network model.

9. An automatic annotation device for crop images, characterized in that: The automatic annotation device for crop images comprises: a data acquisition module, a training module, a verification module and an automatic annotation module; The data acquisition module is used to acquire a training set, a test set and a validation set of crop images; the training set and the test set include manually annotated position information of each crop in the crop image and label information of each crop; The training module is used to freeze the parameter training of each network layer before the j-th group of network layers in the i-th round of training according to the unfreezing order of the neural network model from high level to low level based on the training set, train the parameters in the j-th group of network layers, and adjust the parameters of the network layers after the j-th group of network layers; the j-th group of network layers includes at least one network layer among the network layers other than the input layer and the output layer in the neural network model, and i and j are both integers greater than or equal to 1; and after the parameter training of the j-th group of network layers and subsequent network layers is completed, start training the parameters of the j-1-th group of network layers, and adjust the parameters of the network layers after the j-1-th group of network layers; The verification module is used to verify the model performance index of the automatic crop image annotation based on the learned characteristics of each crop, the weight value of the characteristics of each crop, the test set and the verification set after the parameter training of all network layers in the i-th round is completed; The automatic labeling module is used to label each crop image in the to-be-labeled set based on the trained neural network model if the model performance indicators verified based on the test set and the validation set meet preset conditions.

10. A computer device comprising: A memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the steps of the method according to any one of claims 1 to 9 when executing the computer program.

Images