A method and device for automatically labeling crop images

Abstract

The present invention provides a method and device for automatically labeling crop images, relating to the field of deep learning technology. The method includes: obtaining a training set, a test set, and a validation set of crop images; freezing the parameters of each network layer before the jth group of network layers in the i-th round of training according to the unfreezing order of the neural network model from high to low layers based on the training set, training the parameters in the jth group of network layers and adjusting the parameters of the network layers after the jth group of network layers; after completing the parameter training of the jth group of network layers and subsequent network layers, training the parameters of the j-1st group of network layers and adjusting the parameters of the network layers after the j-1st group of network layers; after completing the parameter training of all network layers in the i-th round, if the model performance indicators verified based on the test set and the validation set meet preset conditions, labeling each crop image in the to-be-labeled set based on the trained neural network model. Based on this solution, the accuracy of automatic crop image labeling can be improved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a method and device for automatically labeling crop images. Background Art

[0002] With the development of agricultural information technology, data-driven agricultural applications have become an important tool for improving production efficiency, optimizing resource utilization and enhancing agricultural decision-making.

[0003] Crop image datasets typically serve as the foundation for training and validating models in agricultural applications. Different crop species typically exhibit significant differences in morphology, color, structure, and phenotypic characteristics, and many crops exhibit varying performance during different growth periods or under different growing environments. Crop image datasets typically need to cover crop images from multiple regions and seasons, making agricultural image annotation challenging.

[0004] The currently commonly used method is manual labeling. However, during the manual labeling process, there are subjective differences between different people and the labeling efficiency is low. The long manual labeling time can easily lead to low labeling accuracy. Summary of the Invention

[0005] The purpose of the present invention is to provide a method and device for automatically labeling crop images to improve the accuracy of labeling in order to solve the problems in the prior art.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides an automatic annotation method for crop images, the method comprising: obtaining a training set, a test set, and a validation set of crop images; the training set and the test set include manually annotated position information of each crop in the crop image and label information of each crop; based on the training set and in accordance with the unfreezing order of the neural network model from high to low layers, in the i-th round of training, freezing the parameters of each network layer before the j-th group of network layers, training the parameters in the j-th group of network layers, and adjusting the parameters of the network layers after the j-th group of network layers; the j-th group of network layers includes at least one network layer in the neural network model other than the input layer and the output layer. Network layer, i and j are both integers greater than or equal to 1; after the parameter training of the jth group of network layers and subsequent network layers is completed, the parameters of the j-1th group of network layers are trained, and the parameters of the network layers after the j-1th group of network layers are adjusted; after the parameter training of all network layers in the i-th round is completed, the model performance indicators of automatic labeling of crop images are verified based on the learned characteristics of each crop, the weight values ​​of the characteristics of each crop, the test set and the validation set; if the model performance indicators verified based on the test set and the validation set meet the preset conditions, then each crop image in the to-be-labeled set is labeled based on the trained neural network model.

[0008] In a second aspect, an embodiment of the present invention provides an automatic annotation device for crop images, which includes: a data acquisition module, a training module, a verification module and an automatic annotation module; the data acquisition module is used to acquire a training set, a test set and a verification set of crop images; the training set and the test set include manually annotated position information of each crop in the crop image and label information of each crop; the training module is used to freeze the parameters of each network layer before the jth group of network layers in the i-th round of training according to the unfreezing order of the neural network model from high to low layers based on the training set, train the parameters in the jth group of network layers, and adjust the parameters of the network layers after the jth group of network layers; the jth group of network layers includes all the layers in the neural network model except the jth group of network layers. At least one network layer in the network layers other than the input layer and the output layer, i and j are both integers greater than or equal to 1; and after the parameter training of the j-th group of network layers and subsequent network layers is completed, the parameters of the j-1-th group of network layers are trained, and the parameters of the network layers after the j-1-th group of network layers are adjusted; a verification module is used to verify the model performance indicators of the automatic labeling of crop images based on the learned characteristics of each crop, the weight values ​​of the characteristics of each crop, the test set and the validation set after the parameter training of all network layers in the i-th round is completed; the automatic labeling module is used to label each crop image in the to-be-labeled set based on the trained neural network model if the model performance indicators verified based on the test set and the validation set meet the preset conditions.

[0009] According to a third aspect of an embodiment of the present invention, a computer device is provided, comprising: a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any of the above methods for automatically labeling crop images are implemented.

[0010] According to a fourth aspect of the embodiments of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the automatic crop image labeling method as described in any one of the above items are implemented.

[0011] An automatic crop image labeling method provided by an embodiment of the present invention first obtains a training set, a test set, and a validation set of crop images; then, based on the training set, in the order of unfreezing the neural network model from high to low layers, in the i-th round of training, freezes the parameter training of each network layer before the j-th group of network layers, trains the parameters in the j-th group of network layers, and adjusts the parameters of the network layers after the j-th group of network layers; secondly, after the parameter training of the j-th group of network layers and subsequent network layers is completed, starts training the parameters of the j-1-th group of network layers, and adjusts the parameters of the network layers after the j-1-th group of network layers; then, after the parameter training of all network layers in the i-th round is completed, verifies the performance index of the model for automatically labeling crop images based on the learned features of each crop, the weight values ​​of the features of each crop, the test set, and the validation set; finally, if the model performance indexes verified based on the test set and the validation set both meet preset conditions, then labels each crop image in the to-be-labeled set based on the trained neural network model. Through the method of hierarchical fine-tuning, the local network layer will undergo continuous iterations during the model training process, so that the parameters of each layer of the model from high to low layers can reach the global optimal level. Unfreezing the model layer by layer can gradually adapt to the characteristics of the recognition task, and can avoid large changes in model weight values ​​and unstable training due to one-time training of all layers. It can make the model transition to the new data distribution more smoothly. Compared with the directly trained model, the accuracy of the model is high, the annotation accuracy is high, and for different types of plant image data, the problem of overfitting can be avoided, and the complexity of the model is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A schematic flow chart of a method for automatically labeling crop images provided by an embodiment of the present invention.

[0013] Figure 2 A logical diagram of model fine-tuning provided by an embodiment of the present invention.

[0014] Figure 3 A schematic diagram of the layer-level fine-tuning process of a neural network model provided by an embodiment of the present invention.

[0015] Figure 4A logical diagram of incremental training provided by an embodiment of the present invention.

[0016] Figure 5 A schematic structural diagram of an automatic crop image labeling device is provided in accordance with an embodiment of the present invention.

[0017] Figure 6 A hardware structure diagram of the computer device in which the automatic crop image annotation system provided by an embodiment of the present invention is located. DETAILED DESCRIPTION

[0018] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent like or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0019] The terms used in this invention are for the purpose of describing specific embodiments only and are not intended to limit the invention. The singular forms "a," "an," and "the" used in this invention and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0020] It should be understood that although the terms "first," "second," "third," etc. may be used in the present invention to describe various information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present invention. Depending on the context, the term "if" as used herein may be interpreted as "when," "when," or "in response to determining."

[0021] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0022] Figure 1 A schematic diagram of a process flow of an automatic annotation method for crop images provided by an embodiment of the present invention is shown as follows: Figure 1 As shown in , the following S101 to S105 may be included:

[0023] S101. Obtain a training set, a test set, and a validation set of crop images.

[0024] The crop images in the training and test sets include manually annotated crop location information and label information for each crop. The location information indicates the location of the target box of the annotated crop in the crop image. The label information indicates the crop type and crop life stage.

[0025] For example, the position information may be coordinate information of the target frame, and the shape of the target frame may be set as required, which is not specifically limited in the embodiment of the present invention.

[0026] It should be noted that during model training, the label information for the training and test sets manually annotated includes crop type and crop life stage. Optionally, after model training is complete, the automatic labeling can include only crop type or both crop type and crop life stage, though this is not specifically limited in this embodiment of the present invention.

[0027] It is understood that the crop type may include a crop type and subtypes under the crop type, for example, flower-orchid type 1-mid-season.

[0028] Optionally, before training a neural network model based on manually annotated crop images, the automatic crop image annotation system may first perform image preprocessing on the crop images used in the neural network model training process, and the image preprocessing process includes: a manual processing process and a model preprocessing process. The manual processing process includes manually selecting a representative crop image set from the acquired crop image dataset, and dividing the selected crop image set into a training set for crop image annotation training, a test set for crop image annotation training, and a validation set for crop image annotation training. The model preprocessing process includes data cleaning, image denoising, and image enhancement on the images in the training set. After image preprocessing, labeling tools are manually used to annotate label information on the crop images in the training set and the test set.

[0029] For example, manual labeling can be performed using labeling tools such as CVAT (Computer Vision Annotation Tool), ScaLabel, MakeSense, and LabelImg.

[0030] It should be noted that during the training process, multiple types of crops can be trained at the same time. For example, images of flowers and images of trees can be grouped into crop types as different types of images and the model can be trained at the same time.

[0031] S102. Based on the training set and in the unfreezing order of the neural network model from high layer to low layer, in the i-th round of training, freeze the parameter training of each network layer before the j-th group of network layers, train the parameters in the j-th group of network layers, and adjust the parameters of the network layers after the j-th group of network layers.

[0032] The j-th group of network layers includes at least one network layer in the network layers other than the input layer and the output layer in the neural network model, and i and j are both integers greater than or equal to 1.

[0033] Exemplarily, the neural network model used in the embodiments of the present invention may be a convolutional neural network model based on deep learning. For example, CNN (Convolutional Neural Networks), ResNet (Residual Neural Network), EfficientNet (Efficient Network), YOLO (You Only Look Once) series, Fast R-CNN (Fast Regional Convolutional Neural Network), SSD (Single Shot MultiBox Detector), Mask R-CNN (Mask Region-based Convolutional Neural Network), etc.

[0034] It can be understood that the order of the neural network model from the input layer to the output layer is the order from the low-level network to the high-level network. Among the various types of neural network models mentioned above, most of the neural network models require parameter training except for the input layer and the output layer. The training of the traditional model is to conduct global training from the low layer to the high layer to determine the performance of the overall training of this round. If the performance is not good, the parameters of all layers are re-trained globally. In an embodiment of the present invention, the parameter training of the low-level network is frozen first, and the parameter training is unfrozen from the high-level network to the low-level network layer by layer, that is, the parameter training of the high-level local network layer is performed first, and the weight values ​​of the parameters of the currently trained local network layer and the weight values ​​of the parameters of the trained network layer are comprehensively adjusted. After the parameter training of the local network layer in this round, the lower network layer is unfrozen for parameter training.

[0035] S103 . After the parameter training of the jth group of network layers and subsequent network layers is completed, start training the parameters of the j-1th group of network layers, and adjust the parameters of the network layers subsequent to the j-1th group of network layers.

[0036] Specifically, in the training of the neural network model, the model training can be guided by minimizing the loss function. When the j-th group of network layers is trained, the loss value of the parameters obtained by training the j-th group of network layers is determined for automatic labeling. If the loss value is greater than the preset loss value, it can be determined that the learned crop characteristics, the weight values ​​of the crop characteristics, the hyperparameters, etc. do not meet the loss constraints. The loss value of the j-th group of network layers can be back-propagated to the various network layers of the lower layers of the j-th group of network layers to adjust the loss. If the loss value is less than or equal to the preset loss value, it can be determined that the learned crop characteristics, the weight values ​​of the crop characteristics, the hyperparameters, etc. meet the loss constraints, and the parameter training of the lower layers can be started.

[0037] It can be understood that in an embodiment of the present invention, the neural network model can be fine-tuned at the hierarchical level, starting from the last layer of the network. The low-level feature extraction layer of the neural network model is first frozen, and the parameters of the high-level feature extraction layer of the neural network model are first trained. The model is gradually unfrozen and trained from the high level to the low level. This allows the neural network model to fully learn the high-level features in the crop image while ensuring the stability of the low-level features, thereby improving the stability and accuracy of the neural network model when learning crop features. After the parameter training of each group of network layers is completed, the parameter training of the next group of network layers is gradually unfrozen. The weight values ​​of the high-level features and the low-level features can be jointly optimized during the iterative training process, thereby improving the effectiveness of ensuring hierarchical fine-tuning during model evaluation.

[0038] S104. After the parameters of all network layers in the i-th round are trained, the model performance indicators of the automatic crop image labeling are verified based on the learned characteristics of each crop, the weight values ​​of each crop characteristic, the test set and the validation set.

[0039] In an embodiment of the present invention, the neural network model will traverse each layer of the model after each training round (all layers are trained), that is, each layer of the model will be used. Since each network layer has a different function, during the training process, each round of training enables the model to learn the characteristics of the crop image automatic labeling task as much as possible (unfreezing layer by layer to achieve local optimality to global optimality). After multiple rounds of training, the parameters of each layer of the model can reach the global optimality as much as possible, so that the performance of the trained model is optimized, thereby achieving faster and more accurate automatic labeling based on the trained neural network model.

[0040] Specifically, in an embodiment of the present invention, after all layers in each round are unfrozen, the overall model can be tuned, and the data between each layer can be comprehensively optimized to make the data transition smooth. The average loss value of the overall model for data training is checked, and the change in loss value in the last p rounds is calculated and measured by variance. When it is less than a certain fixed value, it can be considered that the loss has stabilized. At the same time, the P (precision) curve, PR (Precision-Recall Curve) curve and F1 Score (i.e., the above-mentioned model performance indicators) of the model training will be combined to carry out the next self-labeling training strategy. If all reach the preset values, the hierarchical fine-tuning task is terminated, and you can choose to use the model directly to label the data to be labeled.

[0041] For example, the difference between the trained neural network model's labeling of crop images in the test set and manual labeling can be verified to determine the training accuracy of the neural network model. The trained neural network model can be applied to the agricultural image data in the test set for labeling. The agricultural images in the test set can be inspected and the coordinate positions corresponding to the target boxes with a confidence level greater than a preset confidence level can be output. The model labeling results can be compared with the target boxes in the manually labeled label information to evaluate the performance of the trained neural network model. After model evaluation, if the model does not meet the training end conditions, the model's learning rate and batch size can be adjusted through Bayesian optimization to perform model hyperparameter tuning. The neural network model can monitor the model's performance in the training and test sets, analyze the error distribution of the neural network model for each round of training on the data, determine the learning status of the crop characteristics of the trained neural network model, and complete the test set verification.

[0042] S105: If the model performance indicators verified based on the test set and the validation set meet the preset conditions, each image in the set to be labeled is labeled based on the trained neural network model.

[0043] It should be noted that the images in the validation set may be crop sample images that have not undergone the above-mentioned image preprocessing and have not been manually labeled.

[0044] Specifically, if the model performance indicators verified by the test set all meet the preset conditions, and then the model performance indicators verified based on the validation set all meet the preset conditions, and the model performance indicators verified by the test set and validation set all meet the preset conditions, then automatic labeling can be started.

[0045] For example, based on the validation set and test set, by verifying the performance indicators of the model, the accuracy and robustness of learning can be evaluated, and the performance bottleneck can be automatically adjusted. When the model's self-assessed mAP (mean Average Precision) is greater than or equal to 98%, the model can enter the automatic labeling stage.

[0046] It can be understood that in an embodiment of the present invention, a completed round of global training may include: forward propagation → calculation of loss → back propagation → parameter update. Among them, the forward propagation stage: starting from the input layer, the input data is passed forward layer by layer through each network layer until the output layer. Each layer passes the current feature map (information mainly including weight values, biases, gradients, activation values, etc.) to the next layer, and the initial freezing of the lower layer network will keep the relevant parameters of the lower layer network unchanged and pass them to the next layer with the initial values ​​until the unfrozen layer is reached, and the relevant parameters of the unfrozen layer are updated (gradients, etc. are calculated at the higher layer and parameters are updated). At the same time, the training process will also have a back propagation stage, starting from the output layer, the gradient of the loss function for each layer parameter is calculated and passed back to the lower layer at once. When performing layer-by-layer fine-tuning, by freezing the lower layer parameters, the gradient stops when it reaches the frozen layer, thereby ensuring that the frozen layer is not updated and the control parameters are only updated at the higher layer. After each round of unfreezing the new layer, back propagation will automatically calculate the gradient of the newly unfrozen layer, so that these layers can perform parameter updates and weight value optimization.

[0047] Specifically, after multiple rounds of optimization using the self-learning mechanism based on the aforementioned learning process, large-scale automatic annotation of unlabeled crop images can begin. The annotation system automatically generates target boxes and corresponding target box information. Based on the automatic annotation results, a file containing the target box coordinates and corresponding confidence levels (such as in JSON, XML, YOLO, or CSV format) is generated. Specific types or conditions of data are filtered out from the annotated data, and different subsets of datasets (such as training sets, test sets, and validation sets) are constructed according to different proportions. These are further organized into standardized datasets for subsequent research and application.

[0048] Figure 2 A logical diagram of a model fine-tuning provided by an embodiment of the present invention, such as Figure 2 As shown in , a pre-selected model is selected, the low-level feature extraction layer of the model is frozen, and the previous layers are gradually unfrozen to fully learn the high-level features. At the same time, the weight values ​​of each layer of the multi-layer fine-tuning are unfrozen to jointly optimize the high-level features and low-level features. A preliminary fine-tuned model is obtained, the test set data is tested, and relevant information is output. This is then compared with the manual annotation results to evaluate the network model. The model learning rate is adjusted through Bayesian optimization, hyperparameters are tuned, and the error distribution of the model on the data is analyzed to determine whether the annotation effect is good (determined by the above performance indicators) and whether the error has a small impact on the annotation results (determined by the above performance indicators). If the requirements are met, the desired model is obtained and the model fine-tuning is completed. If the requirements are not met, the low-level feature extraction layer of the model is re-frozen to start iterative model training based on layer-by-layer fine-tuning.

[0049] In the automatic labeling method for crop images provided by an embodiment of the present invention, a training set, a test set, and a validation set of crop images are first obtained; then, based on the training set, in the order of unfreezing the neural network model from high to low layers, in the i-th round of training, the parameter training of each network layer before the j-th group of network layers is frozen, the parameters in the j-th group of network layers are trained, and the parameters of the network layers after the j-th group of network layers are adjusted; secondly, after the parameter training of the j-th group of network layers and subsequent network layers is completed, the parameter training of the j-1-th group of network layers is started, and the parameters of the network layers after the j-1-th group of network layers are adjusted; then, after the parameter training of all network layers in the i-th round is completed, the performance index of the model for automatically labeling crop images is verified based on the learned features of each crop, the weight values ​​of the features of each crop, the test set, and the validation set; finally, if the model performance indexes verified based on the test set and the validation set both meet preset conditions, each crop image in the to-be-labeled set is labeled based on the trained neural network model. Through the method of hierarchical fine-tuning, the local network layer will undergo continuous iterations during the model training process, so that the parameters of each layer of the model from high to low layers can reach the global optimal level. Unfreezing the model layer by layer can gradually adapt to the characteristics of the recognition task, and can avoid large changes in model weight values ​​and unstable training due to one-time training of all layers. It can make the model transition to the new data distribution more smoothly. Compared with the directly trained model, the accuracy of the model is high, the annotation accuracy is high, and for different types of plant image data, the problem of overfitting can be avoided, and the complexity of the model is low.

[0050] Optionally, in the automatic crop image labeling method provided by the embodiment of the present invention, the above-mentioned S102 may specifically include the following S102a:

[0051] S102a. If the loss value of the kth round of the jth group of network layers does not meet the loss constraint condition, then continue to freeze the parameter training of each network layer before the jth group of network layers, perform the k+1th round of parameter training of the jth group of network layers, and adjust the parameters of the network layers after the jth group of network layers.

[0052] Here, k is an integer greater than 1.

[0053] It can be understood that in an embodiment of the present invention, by gradually unfreezing the parameter training of the network layer, that is, the local training of the high-level features of the agricultural crop image is gradually transformed into the global training of the high-level features and low-level features of the agricultural crop image. In the process of parameter training of the local network layer starting from the high-level network, if the loss of crop labeling based on the parameters of the network layer trained in this round is greater than the preset loss, the parameters of the network layer trained in this round do not meet the iteration stop condition, and the parameters of the network layer trained in this round can be continued to be iteratively trained in the next round. In the next round of training, the parameters of all network layers that have been trained before the network layer trained in this round are adjusted (fine-tuned) at the same time.

[0054] It should be noted that the high-level features (semantic features) of crop images refer to the features obtained from the deeper layers of the neural network model. The high-level features of crop images contain rich semantic information and can characterize information such as the type of crop objects in the crop images. Specifically, the high-level features of crop images reflect the high-level semantic information of crop images and contain the key information required to identify different crop objects. They can be used for crop image classification tasks. The low-level features of crop images contain more detailed information and location information and are used for the task of locating crops in crop images. For example, the low-level features of crop images can capture the contours, edges, colors, and textures of crop images. The low-level features of crop images can be used to accurately locate crops in crop images.

[0055] For example, during the training and unfreezing process, the crop image annotation system checks the change in the loss value during the training of the freezing strategy. If it has decreased to less than 80% of the initial loss value of the freezing strategy, the next network layer is unfrozen, and so on.

[0056] Based on this solution, the training of relevant parameters of the low-level network can be frozen in the initial stage. Based on the loss value, the training of parameters of a group of network layers can be gradually unfrozen from the high level. When training the high-level layers, a higher learning rate can be used for training to fully learn the semantic information in the crop images. In the intermediate stage, by fine-tuning the high-level features while training, the model can be adapted to the data distribution of the crop image automatic labeling task, and the parameters of the unfrozen layers can be continuously updated. In the final stage, the parameters of the entire model can be fine-tuned to update the parameters of all layers of the neural network. Since the adjustment of the parameters of each layer is automatically updated according to the features learned after training, the high-level features can be quickly adapted to the task requirements. Through gradual adjustment, the low-level features and high-level features can work together to achieve rapid joint optimization. In this way, the relevant parameters of the model that meets the requirements can be quickly obtained.

[0057] Optionally, in the automatic crop image labeling method provided by the embodiment of the present invention, the above-mentioned S103 may specifically include the following S103a:

[0058] S103a. If the loss value of the kth round of the jth group of network layers meets the loss constraint condition, continue freezing the parameter training of each network layer before the j-1th group of network layers, start training the parameters of the j-1th group of network layers, and adjust the parameters of the network layers after the j-1th group of network layers.

[0059] It can be understood that in each round of overall training, when iterative training is performed on a group of network layers, if the loss value in the kth round meets the loss constraint condition, it can be determined that the parameters of this group of network layers have converged in the kth round of training, and the local training is completed. Then, the training of the parameters before the j-2th group of network layers can be continued, and the training of the parameters of the j-1th group of network layers can be unfrozen.

[0060] It should be noted that when training the parameters of the j-th group of network layers, it is necessary to fine-tune the parameters of the network layers after the j-th group of network layers at the same time to ensure the stability of high-level features, that is, to ensure the accuracy of semantic information in the image.

[0061] Based on this scheme, it is possible to determine whether to unfreeze the new network layer for parameter training based on the loss constraint condition, thereby ensuring that the loss of the determined parameter weight value is minimized when labeling, reducing the probability of labeling errors and improving the accuracy of automatic crop image labeling.

[0062] It should be noted that during actual training, some crop types have fewer training samples. If the sample ratio does not meet a preset ratio, the neural network model can automatically expand sample images for crops with fewer than a preset threshold. For example, the model can automatically crop, scale, and flip images of that crop type, expanding a single crop image into multiple crop images so that the ratio of expanded sample images of that crop type to sample images of other crop types meets a preset ratio.

[0063] Optionally, in the automatic crop image labeling method provided in the embodiment of the present invention, the label information includes the crop type and the crop life stage. Then, after the above-mentioned S105, the following S106 may be further included:

[0064] S106. If the accuracy of the model performance indicator verified by the validation set corresponding to the first type of crop in the i-th round of training is less than the preset accuracy, then additional training images of the first type of crop are added, and the i+1-th round of training is continued based on the training set after the additional training images of the first type of crop are added.

[0065] It should be noted that when the proportion of training samples is unbalanced, for example, the amount of crop data under type 1 is small, the neural network model may not have learned enough features for the crop data under type 1 after training, and may not have learned the crop features that effectively distinguish type 1 crops from other types of crops, resulting in low overall labeling accuracy.

[0066] Specifically, the validation set is used to verify the accuracy of the trained neural network model's annotations, thereby verifying whether the neural network model from the i-th round of training has learned the characteristics of the crop data. Based on the verification results for each crop type in the validation set, the neural network model's performance in the training set is determined. Data augmentation can be performed on crop images for which characteristics have not been fully learned.

[0067] Optionally, the newly added first-category crop training images may include at least one of the following categories:

[0068] (1) Images that are resampled from the first category of crop images in the historical training set;

[0069] (2) images extracted from the set of crop images to be labeled in the first category;

[0070] The neural network model can also extract images from the to-be-labeled set of the first category of crops, annotate the extracted images with pseudo-labels, add the crop image data with added pseudo-labels to the training set through a playback mechanism for retraining, continue to update the model parameters in a gradual unfreezing manner, and annotate the crop images in the to-be-labeled set of the first category of crops based on the updated neural network model.

[0071] It should be noted that the embodiments of the present invention provide a self-learning mechanism. The neural network model extracts crop images from the unlabeled dataset and iteratively trains through automatic label generation and model prediction confidence, effectively expanding crop features in incremental data. During training, the neural network model generates pseudo-labels based on the prediction results of the unlabeled data. Specifically, the neural network model generates preliminary predictions for the unlabeled data. When the confidence level exceeds a certain threshold, these predictions are incorporated into the training as pseudo-labels.

[0072] (3) Sample images of the first-category crop images of more scenes acquired from the outside, for example, images of the first-category crop at different growth stages, different lighting conditions, and different occlusion conditions.

[0073] For example, the crop life stages annotated in the training images of the first category of crops after adding different growth stages are different from the crop life stages annotated in the training images of the first category of crops before adding them.

[0074] Specifically, the training images of the newly added first type of crops are mixed with the training images in the historical training set, and the i+1th round of training is performed to ensure the uniform distribution of different feature data.

[0075] For example, the loss function of the model can be adjusted in the (i+1)th round of training based on the adjustment rule indicated by formula (1), which can reduce the loss of original feature information while learning new features.

[0076]

[0077] Among them, L task Represents the standard loss for the current task, λ represents the regularization coefficient, and λ is used to control the balance between new features and old features. represents the L2 norm of all parameters.

[0078] By constraining the size of model parameters, drastic changes in parameters are avoided and the coexistence of new and old features in the model parameter space is ensured.

[0079] It can be understood that during the i+1th round of training, continuing to repeat the layer fine-tuning steps in the above steps can ensure that the learned new features and the original features are balanced in the entire network.

[0080] Optionally, new data can be introduced in batches during each training session. For example, when training the first set of network layers, the first batch of new data is introduced, and when training the second set of network layers, a second batch of data that does not overlap with the first batch is introduced. The model learns the characteristics of the new data through layer-by-layer fine-tuning. The ratio of new and old data can be set, such as using 80% of the original data and 20% of the new data, to prevent the model from being overly biased towards the new data. Through incremental training, the new data is passed to the parameters of each layer layer by layer, ensuring the integration of new and old knowledge.

[0081] Based on this solution, in the automatic labeling method for crop images provided by an embodiment of the present invention, when training a neural network model, for crop images whose features have not been fully learned, additional training images for this type of crop can be added based on the performance of the neural network model in the training set. The model can be updated by resampling the original data set, or the model can be retrained and updated by extracting a small number of images from the set to be labeled and introducing pseudo-labels, or the model can be updated by introducing samples from more scenes from the outside. For example, crop images at different growth stages and under different lighting conditions can be introduced to enrich the training samples for the model for this type of crop, so that the model can fully learn the features of each type of crop, improve the accuracy of automatic labeling of this type of crop, reduce the workload of manual re-labeling of this type of crop in the later stage, and improve the efficiency and accuracy of labeling of this type of crop.

[0082] Optionally, in the automatic crop image annotation method provided by the embodiment of the present invention, after the above-mentioned S104, incremental training may be started during the training process, and the above-mentioned method may further include the following S107 to S109:

[0083] S107: After verifying the model performance indicators based on the target dataset, mark the target images in the target dataset as incremental training sample images.

[0084] The target images include at least one type of images selected from crop images that failed to be labeled and crop images that were successfully labeled and whose confidence level is lower than a first confidence level, and the target dataset includes at least one of a test set and a validation set.

[0085] Specifically, the labeling system can output the confidence of each automatically labeled target box during the model performance verification process of model training. The neural network model marks the corresponding target image as an incremental training sample image for the target box with relatively low confidence and stores it in the first folder; NMS (Non-Maximum Suppression) is used to determine whether to label the crop image; for example, if the NMS threshold is set to 0.6, then when the neural network model labels the crop image, if the NMS of the detected crop image is less than or equal to 0.6, the label is not labeled; if the NMS of the crop image is greater than 0.6, the label is labeled, and the unlabeled crop image is added to the first folder. Through manual review, the image data in the manually labeled first folder is verified data that meets the requirements of the training data set.

[0086] It should be noted that in an embodiment of the present invention, if the first image in the first folder is not labeled, the target frame and label are manually labeled; if the first image is labeled, it is manually determined whether the label of the first image is accurately labeled. If the label is accurately labeled, the position of the target frame is adjusted; if the label of the first image is incorrect, the label and target frame position of the first image are manually adjusted.

[0087] S108. After the i-th round of training is completed, if the incremental training conditions are met, incremental training is started in the i+1-th round of training, and the review images of the incremental training crop sample images marked from the 1st round to the i-th round are added to the training set to continue training, and the weight values ​​of each crop feature are re-determined.

[0088] It should be noted that, in this solution, the review image is an image obtained by manually re-labeling the target image.

[0089] S109 , automatically adjusting the weight values ​​of the various crop features based on the re-determined weight values ​​of the various crop features and the weight values ​​of the parameters in the (i+1)th round of training.

[0090] Exemplarily, the feature weight value may be automatically adjusted based on the adjustment rule shown in the following formula (2).

[0091] L adaptive =ω t *L new +(1-ω t )*L replay Formula (2)

[0092] Among them, ωt Represents the weight value dynamically adjusted based on time t, which can be set according to the characteristic learning speed of the new data and the learning rate attenuation. new Represents the re-determined loss value, L replay Represents the original loss value, L adaptive Represents the loss value after dynamic adjustment.

[0093] Based on this scheme, a feedback channel can be automatically generated based on the annotation results of at least one item in the test set and the validation set and the actual annotation results. The crop image annotation system can automatically identify erroneous crop images and low-confidence crop images, mark such crop images as key learning crop images, and add this type of data back to the training set to start incremental training. In the incremental training, the features with higher weight values ​​among the extracted crop features are selected, and the weight values ​​of each crop feature used for the annotation task are automatically adjusted.

[0094] Optionally, in the automatic crop image annotation method provided in the embodiment of the present invention, after the above-mentioned S105, online self-learning (online incremental training) may be performed during the automatic crop image annotation process, which may include the following S110 to S112:

[0095] S110: Add the target image generated during the labeling process to the online self-learning incremental sample set.

[0096] The self-learning incremental samples include: failed labeling samples generated during the automatic labeling process and crop images that are successfully labeled and have a confidence level lower than a first confidence threshold.

[0097] It is understandable that when automatic labeling is performed based on a trained neural network model, there will be crop images that fail to be labeled and crop images that succeed but have low confidence in the labeling results.

[0098] It should be noted that, in the embodiment of the present invention, before performing online incremental training, the crop images in the online self-learning incremental sample set need to be manually re-labeled.

[0099] The neural network model can output the target image generated during the automatic labeling process to the user, instructing the user to manually label it. After the user completes the manual labeling, online incremental training can be performed based on the manually labeled target image to re-optimize the model parameters.

[0100] S111. Add the images and manually labeled information in the manually labeled online self-learning enhanced sample set to the training set, start online incremental training, and adjust the parameter weight values ​​of each layer of the neural network model online.

[0101] Optionally, when automatically labeling a batch of images, online incremental training can be started. It can be started once or multiple times and can be set according to needs. This embodiment of the present invention does not specifically limit this.

[0102] S112. Continue labeling the remaining images in the labeling set based on the adjusted parameter weight values ​​of each layer of the neural network model.

[0103] Exemplarily, the above-mentioned S112 may be performed when the first condition of online self-learning is satisfied.

[0104] For example, the first condition includes at least one of the following conditions 1-1 to 1-4:

[0105] Condition 1-1: The number of target images generated by automatic annotation is greater than or equal to a first number.

[0106] For example, when performing a labeling task, online incremental training can be started every time 200 target images are generated.

[0107] Condition 1-2: a ratio of the number of target images generated by automatic annotation to the number of images for which annotation has been performed is greater than or equal to a first value.

[0108] For example, if 10% of the images for which the annotation task has been performed are target images, an online incremental training can be started.

[0109] Condition 1-3: the ratio of the number of labeled images to the total number of images in the original set to be labeled is greater than or equal to a second value.

[0110] For example, online incremental training is started once when the labeling task is 50% completed, and online incremental training is started again when the automatic labeling task is 75% completed.

[0111] Condition 1-4: The time for executing automatic marking is greater than the first preset time.

[0112] For example, incremental training is started every hour after the labeling task is executed.

[0113] Based on this solution, during the automatic labeling process, online incremental training can be started based on images that failed to be labeled and images whose confidence in the labeling results is lower than the confidence threshold. By manually re-labeling such images, the manually labeled images of this type are then incrementally trained. This allows model optimization based on automatically labeled crop images to improve the accuracy of the model's automatic labeling.

[0114] Optionally, in the automatic crop image annotation method provided in the embodiment of the present invention, after the above-mentioned S105, the following S113 may be further included:

[0115] S113. During the labeling process, the crop images with a confidence level greater than the second confidence level in the labeling results are added to the historical training set, and retraining is performed based on the added training set to adjust the parameter weight values ​​of each layer of the neural network model.

[0116] Exemplarily, the above-mentioned S114 may be performed when the second condition of online self-learning is satisfied.

[0117] The second condition includes at least one of the following conditions 2-1 to 2-3:

[0118] Condition 2-1: The total number of images for which the trained neural network model has performed labeling tasks is greater than or equal to a third number.

[0119] Condition 2-2: The total number of target images automatically labeled based on the trained neural network model is greater than or equal to the fourth number.

[0120] Condition 2-3: The time for executing automatic marking is greater than the first preset time.

[0121] Based on this scheme, in the process of automatic crop image labeling using a neural network model trained based on a historical training set, crop images with higher confidence in the labeling results can be added to the historical training set, and online incremental training can be started, which can further optimize the performance of the neural network model for automatic crop image labeling.

[0122] Optionally, in the automatic crop image annotation method provided in the embodiment of the present invention, after the above-mentioned S105, the following S114 to S116 may be further included:

[0123] S114. In the case of a newly added training data set, if the label information in the newly added training data set indicates that the crop type is an untrained target subtype in the trained second crop type, then based on the first data volume of the newly added training data set, historical training data corresponding to the trained subtype in the second crop type in the second data volume is extracted according to a preset ratio.

[0124] For example, the data volume ratio of the new training dataset and the historical training dataset is 8:2.

[0125] It can be understood that after training large types of crop images, new subtypes of crops are added for labeling. If the labeling is performed directly based on the previously trained model, the subtype of crops may be easily labeled as other subtypes of crops. For example, the features of orchids of types 1 to 10 have been learned, and new orchids of type 11 need to be automatically labeled. At this time, if automatic labeling is performed based on the previously trained neural network model, it is easy to label them as types 1 to 10, or even may not be recognized as orchids; if training is performed directly on orchids of type 11, the model may forget the features of orchids of types 1 to 10, which will lead to a decrease in the accuracy of orchid labeling in the later stage.

[0126] S115 , mixing the newly added training data set and the extracted historical training data into an updated training data set.

[0127] S116. Based on the updated training data set and the parameter weight values ​​of each layer of the neural network model, retrain to obtain the crop characteristics of the target subtype and the parameter weight values ​​of each layer of the neural network model.

[0128] It should be noted that, in an embodiment of the present invention, each time a new type of training data enters the model, historical training data can be randomly inserted to supplement it, and a playback mechanism can be used to regularly replay part of the historical training data to ensure that the original features are not lost or forgotten when training the newly added crop type image data.

[0129] Specifically, the loss function of the above formula (1) can be combined to adjust the balance between new and old features.

[0130] Based on this solution, after obtaining a trained neural network model based on historical training data, when training on new crop image data, the major type of the newly added crop images can be determined based on the manually annotated information in the new training set. If the major type is the same but the minor type is different, a replay mechanism can be used for model training on the newly added crop images. This randomly inserts historical training data of crops of the same type into the new training set to supplement the data and mix the old and new data. During training, the loss function is adjusted to balance the new and old features, ensuring that feature extraction is continuous. This prevents the model from losing features extracted from old data when training on new image data, ensuring that the model retains existing features while learning new ones. The latest model output is compared with the manually annotated results to analyze the differences and adjust the model training strategy. Finally, the new crop data type is annotated based on the optimized model, improving the model's generalization ability and avoiding overfitting.

[0131] Example:

[0132] Figure 3 A schematic diagram of a layer fine-tuning process of a neural network model provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown in Figure 1, the YOLOv7 network, from low to high layers, consists of the following: input layer, CBS, pooling layer, residual block, upsampling layer, feature fusion layer, detection head, and output layer. The CBS (Convolutional Block with Squeeze-and-Excitation) includes the following: convolutional layer, batch normalization layer, and activation function layer.

[0133] In an embodiment of the present invention, the training of the parameters of the network layer can be gradually unfrozen in the order from the high-level network to the low-level network. The crop image data in the training set is input into the YOLOv7 network through the input layer of the YOLOv7 network. The CBS module of the YOLOv7 network extracts the basic features of the crop image data in the training set. Specifically, the CBS module is a basic module in the YOLOv7 network architecture, wherein the convolution operation is performed by the convolution layer of the YOLOv7 network to extract local features in the crop image, the batch normalization layer of the YOLOv7 network is used to accelerate model training and stabilize model convergence, and the ReLU (Rectified Linear Unit) activation function of the YOLOv7 network activation function layer is used to introduce nonlinear features. The pooling layer of the YOLOv7 network downsamples the image data in spatial latitude to reduce the size of the feature map, retain important features, and reduce the amount of calculation of the model. The YOLOv7 network's ResNet blocks utilize residual connections to effectively avoid the vanishing gradient problem in deep networks, enabling deeper model training to capture more complex features. The YOLOv7 network's upsampling layer upsamples feature maps to restore the size of the feature maps reduced in the pooling and convolution layers, thereby improving the model's detection resolution. The YOLOv7 network's feature fusion layer fuses features from different levels, allowing the model to comprehensively utilize high-level semantic features and low-level detail features. The YOLOv7 network's feature fusion layer includes FAN (Feature Aggregation Network) and PAN (Path Aggregation Network), commonly used architectures for multi-scale feature fusion. These architectures can enhance the model's ability to recognize objects at multiple scales. In this embodiment of the present invention, they are used to detect plant species of varying sizes and shapes in agricultural scenarios. After feature fusion, the YOLOv7 network's detection head predicts the final target (including bounding box regression and classification tasks) and outputs the target's type and annotated location. The YOLOv7 network's output layer generates the model's final output.

[0134] Specifically, in this embodiment of the present invention, the YOLOv7 network's low-level network includes convolutional layers, pooling layers, and residual layers, which are used to extract basic, low-level features from crop images. The YOLOv7 network's high-level network includes upsampling layers, feature fusion layers, and a detection head, which are used to extract abstract, high-level features.

[0135] The model parameters of each layer of the YOLOv7 network can be trained layer by layer in the order of unfreezing from high to low layers. When training the jth group of network layers, the parameters of the layers before the jth group of network layers are not trained. After the parameters of the jth group of network layers converge, the parameters of the j-1th group of network layers are trained. For example, when training the detection head, the CBS, pooling layer, residual block, upsampling layer, and feature fusion layer before the detection head are frozen. The frozen network layer will transfer the extracted feature maps to the high-level network in sequence until the detection head, and the parameters of the detection head will be trained. After the parameter training of the detection head is completed, the feature fusion layer will be unfrozen. The frozen network layer will transfer the extracted feature maps to the high-level network in sequence until the feature fusion layer, and the parameters of the feature fusion layer will be trained. The parameter values ​​of the trained feature fusion layer and the detection head will be adjusted until the parameter values ​​of all network layers are adjusted after the CBS training is completed.

[0136] Figure 4 A logical diagram of incremental training provided by an embodiment of the present invention, such as Figure 3 As shown in , the labeling results of the test set and validation set are compared with the actual labeling results; the model automatically generates a feedback channel; marks the incremental learning data: incorrectly labeled and low-confidence samples; obtains samples from more environmental scenarios from the outside; adds these two types of samples to the training set and starts incremental training; automatically adjusts the model weight value and the evaluation model performance indicator threshold; evaluates the model accuracy and robustness; determines whether the mAP is greater than 98%. If it is greater than 98%, obtain the expected model and complete model fine-tuning; if it is not greater than 98%, re-acquire incremental samples to start incremental training.

[0137] Corresponding to the aforementioned method embodiments, the present invention also provides embodiments of a device and a terminal to which the device is applied.

[0138] Figure 5 The present invention provides a structural diagram of an automatic crop image annotation device. Figure 5As shown in FIG, the automatic annotation device 500 for crop images includes: a data acquisition module 501, a training module 502, a verification module 503 and an automatic annotation module 504; the data acquisition module 501 is used to obtain a training set, a test set and a verification set of crop images; the training set and the test set include the position information of each crop in the crop image and the label information of each crop manually annotated; the training module 502 is used to freeze the parameters of each network layer before the j-th group of network layers in the i-th round of training according to the unfreezing order of the neural network model from high to low layers based on the training set, train the parameters of the j-th group of network layers, and adjust the parameters of the network layers after the j-th group of network layers; the j-th group of network layers includes all network layers in the neural network model except the input layer and the output layer. At least one network layer in the network layers other than the outgoing layer, i and j are both integers greater than or equal to 1; and after the parameter training of the j-th group of network layers and subsequent network layers is completed, the parameters of the j-1-th group of network layers are trained, and the parameters of the network layers after the j-1-th group of network layers are adjusted; a verification module 503 is used to verify the model performance indicators of the automatic annotation of crop images based on the learned characteristics of each crop, the weight value of the characteristics of each crop, the test set and the validation set after the parameter training of all network layers in the i-th round is completed; an automatic annotation module 504 is used to annotate each crop image in the to-be-annotated set based on the trained neural network model if the model performance indicators verified based on the test set and the validation set meet the preset conditions.

[0139] Optionally, the training module is specifically used to: if the loss value of the kth round of the jth group of network layers does not meet the loss constraint condition, then continue to freeze the parameter training of each network layer before the jth group of network layers, perform the k+1th round of parameter training of the jth group of network layers, and adjust the parameters of the network layers after the jth group of network layers, where k is an integer greater than 1.

[0140] Optionally, the training module is specifically used to: if the loss value of the kth round of the jth group of network layers meets the loss constraint condition, continue freezing the parameter training of each network layer before the j-1th group of network layers, start training the parameters of the j-1th group of network layers, and adjust the parameters of the network layers after the j-1th group of network layers.

[0141] Optionally, the label information includes crop type and crop life stage; the data acquisition module 501 is further used to add training images of the first category of crops if the accuracy of the model performance indicator verified by the verification set corresponding to the first category of crops in the i-th round of training is less than a preset accuracy; the training module is further used to continue to perform the i+1-th round of training based on the training set after adding the training images of the first category of crops.

[0142] Optionally, the automatic labeling device for crop images also includes: a labeling module; a labeling module, which is used to label the target image in the target data set as an incremental training crop sample image after verifying the model performance index based on the target data set; a training module, which is also used to start incremental training in the (i+1) round of training after the i-th round of training is completed, if the incremental training conditions are met, and add the incremental training crop sample images marked from the 1st round to the i-th round to the training set to continue training, and re-determine the weight value of each crop feature; and automatically adjust the weight value of each crop feature based on the re-determined weight value of each crop feature and the weight value of the parameter of the (i+1) round of training; wherein the target image includes at least one type of image from the crop image that failed to be labeled and the crop image that was successfully labeled and the confidence level is lower than the first confidence level, and the target data set includes at least one of a test set and a validation set.

[0143] Optionally, the labeling module is also used to add the target image generated in the labeling process to the online self-learning incremental sample set; the data acquisition module is also used to add the images and manual labeling information in the manually re-labeled online self-learning incremental sample set to the training set, and the training module is also used to start online incremental training and adjust the parameter weight values ​​of each layer of the neural network model online; the automatic labeling module is also used to continue labeling the remaining images in the labeling set based on the adjusted parameter weight values ​​of each layer of the neural network model; wherein the self-learning incremental samples include: labeling failure samples generated in the automatic labeling process and crop images that are successfully labeled and have a confidence level lower than a first confidence threshold.

[0144] Optionally, the automatic labeling device for crop images also includes: an adding module; the adding module is used to add crop images with a confidence level greater than a second confidence level in the labeling results to a historical training set during the labeling process; and the training module is also used to retrain based on the added training set and adjust the parameter weight values ​​of each layer of the neural network model.

[0145] Optionally, the automatic labeling device for crop images also includes: a data mixing module; a data acquisition module, which is also used to, in the case of a newly added training data set, if the label information in the newly added training data set indicates that the crop type is an untrained target subtype in the trained second crop type, then based on the first data amount of the newly added training data set, extract historical training data corresponding to the trained subtype in the second crop type of the second data amount according to a preset ratio; the data mixing module is used to mix the newly added training data set and the extracted historical training data into an updated training data set; the training module is also used to retrain and obtain the crop characteristics of the target subtype and the parameter weight values ​​of each layer of the neural network model based on the updated training data set and the parameter weight values ​​of each layer of the neural network model.

[0146] An automatic crop image labeling device provided by an embodiment of the present invention first obtains a training set, a test set, and a validation set of crop images; then, based on the training set, in the order of unfreezing the neural network model from high to low layers, in the i-th round of training, freezes the parameter training of each network layer before the j-th group of network layers, trains the parameters in the j-th group of network layers, and adjusts the parameters of the network layers after the j-th group of network layers; secondly, after the parameter training of the j-th group of network layers and subsequent network layers is completed, starts training the parameters of the j-1-th group of network layers, and adjusts the parameters of the network layers after the j-1-th group of network layers; then, after the parameter training of all network layers in the i-th round is completed, verifies the model performance index of the automatic crop image labeling based on the learned features of each crop, the weight values ​​of the features of each crop, the test set, and the validation set; finally, if the model performance indexes verified based on the test set and the validation set both meet preset conditions, then labels each crop image in the to-be-labeled set based on the trained neural network model. Through the method of hierarchical fine-tuning, the local network layer will undergo continuous iterations during the model training process, so that the parameters of each layer of the model from high to low layers can reach the global optimal level. Unfreezing the model layer by layer can gradually adapt to the characteristics of the recognition task, and can avoid large changes in model weight values ​​and unstable training due to one-time training of all layers. It can make the model transition to the new data distribution more smoothly. Compared with the directly trained model, the accuracy of the model is high, the annotation accuracy is high, and for different types of plant image data, the problem of overfitting can be avoided, and the complexity of the model is low.

[0147] The embodiment of the automatic annotation method of crop images of the present invention can be applied to a computer device, such as a server or a terminal device. The method embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the automatic annotation device of crop images in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, if Figure 6 The figure is a hardware structure diagram of the computer device where the automatic annotation system of crop images provided by the embodiment of the present invention is located, except Figure 6 In addition to the processor 610, memory 630, network interface 620, and non-volatile memory 640 shown, the server or electronic device where the device 631 is located in the embodiment may also include other hardware according to the actual function of the computer device, which will not be described in detail.

[0148] Accordingly, the present invention also provides an automatic labeling device for crop images, which includes a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: execute the steps of the automatic labeling method for crop images executed by the above-mentioned controller.

[0149] In one embodiment, a computer device is provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor implements any step of the above-mentioned method for automatically labeling crop images when executing the computer program.

[0150] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any step of the above-mentioned method for automatically labeling crop images can be implemented.

[0151] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0152] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present invention. Those of ordinary skill in the art can understand and implement it without paying any creative work.

[0153] The foregoing description describes specific embodiments of the present invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0154] Other embodiments of the present invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention claimed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow from the general principles of the invention and include common knowledge or customary techniques in the art not claimed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the invention being indicated by the following claims.

[0155] It should be understood that the present invention is not limited to the exact construction described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

[0156] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for automatic annotation of crop images, characterized in that: The method comprises: Acquire a training set, a test set, and a validation set of crop images; the training set and the test set include manually annotated location information of each crop in the crop image and label information of each crop, wherein the label information includes crop type and crop life stage; Based on the training set and in the order of unfreezing the neural network model from high layer to low layer, in the i-th round of training, freezing the parameter training of each network layer before the j-th group of network layers, training the parameters in the j-th group of network layers, and adjusting the parameters of the network layers after the j-th group of network layers; the j-th group of network layers includes at least one network layer among the network layers other than the input layer and the output layer in the neural network model, and i and j are both integers greater than or equal to 1; After the parameter training of the j-th group of network layers and subsequent network layers is completed, starting to train the parameters of the j-1-th group of network layers, and adjusting the parameters of the network layers subsequent to the j-1-th group of network layers; After the parameters of all network layers in the i-th round are trained, based on the learned features of each crop, the weight values ​​of the features of each crop, the test set and the validation set, verifying the performance indicators of the model for automatically labeling crop images; If the model performance indicators verified based on the test set and the validation set both meet preset conditions, annotating each crop image in the to-be-annotated set based on the trained neural network model; The target image generated during the labeling process is added to the online self-learning incremental sample set; the manually re-labeled images and manually labeled information in the online self-learning incremental sample set are added to the training set, online incremental training is initiated, and the weight values ​​of the parameters of each layer of the neural network model are adjusted online; based on the adjusted weight values ​​of the parameters of each layer of the neural network model, the remaining images in the to-be-labeled set are further labeled; wherein the self-learning incremental samples include: the failed labeling samples generated during the automatic labeling process and the crop images that are successfully labeled with a confidence level lower than a first confidence threshold; During the labeling process, crop images with a confidence level greater than a second confidence level in the labeling results are added to a historical training set, and retraining is performed based on the added training set to adjust the weight values ​​of the parameters of each layer of the neural network model; In the case of a newly added training data set, if the label information in the newly added training data set indicates that the crop type is an untrained target subtype in the trained second crop type, then based on the first data volume of the newly added training data set, historical training data corresponding to the trained subtype in the second crop type of the second data volume are extracted according to a preset ratio; the newly added training data set and the extracted historical training data are mixed to form an updated training data set; based on the updated training data set and the parameter weight values ​​of each layer of the neural network model, the crop characteristics of the target subtype and the parameter weight values ​​of each layer of the neural network model are retrained to obtain.

2. The method according to claim 1, characterized in that The freezing of parameter training of each network layer before the j-th group of network layers, training of parameters in the j-th group of network layers, and adjustment of parameters of network layers after the j-th group of network layers include: If the loss value of the kth round of the jth group of network layers does not meet the loss constraint condition, then continue to freeze the parameter training of each network layer before the jth group of network layers, perform the k+1th round of parameter training of the jth group of network layers, and adjust the parameters of the network layers after the jth group of network layers, where k is an integer greater than 1.

3. The method according to claim 2, characterized in that After the parameter training of the jth group and subsequent network layers is completed, starting to train the parameters of the j-1th group of network layers and adjusting the parameters of the network layers subsequent to the j-1th group of network layers includes: If the loss value of the kth round of the jth group of network layers meets the loss constraint condition, continue to freeze the parameter training of each network layer before the j-1th group of network layers, start training the parameters of the j-1th group of network layers, and adjust the parameters of the network layers after the j-1th group of network layers.

4. The method according to claim 1, wherein The method further comprises: If the accuracy of the model performance indicator verified by the validation set corresponding to the first category of crops in the i-th round of training is less than the preset accuracy, then training images of the first category of crops are added, and the i+1-th round of training is continued based on the training set after the training images of the first category of crops are added.

5. The method according to claim 1, wherein The method further comprises: After verifying the model performance indicator based on the target dataset, marking the target image in the target dataset as an incremental training crop sample image; After the i-th round of training is completed, if the incremental training conditions are met, incremental training is started in the i+1-th round of training, and the incremental training crop sample images marked from the 1st round to the i-th round are added to the training set to continue training, and the weight value of each crop feature is re-determined; Automatically adjusting the weight values ​​of the various crop features based on the re-determined weight values ​​of the various crop features and the weight values ​​of the parameters trained in the (i+1)th round; The target images include at least one type of images from among crop images that failed to be labeled and crop images that were successfully labeled with a confidence level lower than a first confidence level, and the target dataset includes at least one of the test set and the validation set.

6. An automatic annotation device for crop images, characterized in that: The automatic annotation device for crop images includes: a data acquisition module, a training module, a verification module, an automatic annotation module, a marking module, an adding module and a data mixing module; The data acquisition module is used to acquire a training set, a test set, and a validation set of crop images; the training set and the test set include manually annotated position information of each crop in the crop image and label information of each crop, wherein the label information includes crop type and crop life stage; The training module is configured to, based on the training set and in an unfreezing order from high to low layers of the neural network model, freeze the parameter training of each network layer before the j-th group of network layers in the i-th round of training, train the parameters in the j-th group of network layers, and adjust the parameters of the network layers after the j-th group of network layers; the j-th group of network layers includes at least one network layer among the network layers other than the input layer and the output layer in the neural network model, and i and j are both integers greater than or equal to 1; and after the parameter training of the j-th group of network layers and subsequent network layers is completed, start training the parameters of the j-1-th group of network layers and adjust the parameters of the network layers after the j-1-th group of network layers; The verification module is configured to verify the performance indicators of the model for automatically labeling crop images based on the learned characteristics of each crop, the weight values ​​of the characteristics of each crop, the test set, and the validation set after the parameters of all network layers in the i-th round are trained; The automatic labeling module is configured to label each crop image in the to-be-labeled set based on the trained neural network model if the model performance indicators verified based on the test set and the validation set both meet preset conditions; The labeling module is further configured to add the target image generated during the labeling process to the online self-learning incremental sample set; the data acquisition module is further configured to add the manually relabeled images and manually labeled information in the online self-learning incremental sample set to the training set; the training module is further configured to initiate online incremental training and online adjust the parameter weights of each layer of the neural network model; the automatic labeling module is further configured to continue labeling the remaining images in the to-be-labeled set based on the adjusted parameter weights of each layer of the neural network model; wherein the self-learning incremental samples include: failed labeling samples generated during the automatic labeling process and crop images that were successfully labeled with a confidence level lower than a first confidence threshold; The adding module is used to add the crop images with a confidence level greater than the second confidence level in the annotation results to the historical training set during the annotation process; the training module is further used to retrain based on the added training set to adjust the parameter weight values ​​of each layer of the neural network model; The data acquisition module is used to, in the case of a newly added training data set, extract historical training data corresponding to the trained subtype of the second crop type in a second data volume according to a preset ratio based on the first data volume of the newly added training data set if the label information in the newly added training data set indicates that the crop type is an untrained target subtype of the trained second crop type; the data mixing module is used to mix the newly added training data set and the extracted historical training data into an updated training data set; the training module is also used to retrain and obtain the crop characteristics of the target subtype and the parameter weight values ​​of each layer of the neural network model.

7. A computer device comprising: A memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the steps of the method according to any one of claims 1 to 5 when executing the computer program.

Images