An unsupervised cross-domain plant organ counting method based on minimizing feature differences
Through the combination of cyclic generative adversarial network and Bayesian loss function, the problems of large feature differences and unstable density map quality in cross-domain plant organ counts are solved, and more accurate plant organ counts are achieved, avoiding interference from Gaussian nuclear function .
Patent Information
- Application Number
- CN202210825100.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-13
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-07-13
AI Technical Summary
The existing density map counting method has problems such as large feature differences and poor adversarial learning effects in cross-domain plant organ counting, and the quality of the density map is unstable based on Gaussian kernel functions.
Adopting an unsupervised cross-domain method based on a circular generative adversarial network, by constructing improved generators and domain judges, using spatial attention modules and channel attention modules, converting source domain data into pseudo-target domain data, and building a density graph estimation model , use Bayesian loss function to generate density graphs to avoid the use of Gaussian kernel functions.
It effectively narrows the characteristic differences between source and target domain data, improves the consistency of cross-domain data mapping, enhances counting accuracy, and avoids interference problems caused by density map generation.
Smart Images

Figure CN115049933B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of plant organ counting, and in particular to an unsupervised cross-domain plant organ counting method based on minimizing feature differences. Background Art
[0002] Leaf counting can be used to assess the growth stage and health of a plant. Other counting tasks for plant phenotyping include counting the number of ears in cereal crops to estimate grain yield, and counting flowers to guide pruning and thinning processes that directly affect fruit yield and quality.
[0003] In the counting of plant organs in images, the current methods include direct regression for counting. This method does not make full use of the spatial information of the target and the counting effect is often poor. Tewodros et al. proposed a cross-domain plant organ density map counting method. This method uses a network model similar to U-Net as a density map generator to complete the crossover between two sets of data sets in different fields, where the data set with point annotation information is called the source domain data and the data set without annotation information is called the target domain data. The encoder (downsampling) part of the network model is used as a feature extractor to extract image features, and the decoder (upsampling) upsamples the extracted image features to generate a density map for counting. The domain discriminator is used to classify which domain the features extracted by the feature extractor belong to, and the gradient reverse layer (GRL) method is combined to complete the adversarial process between the decoder (upsampling) and the domain discriminator, so that the difference between the features extracted from the target domain data and the features extracted from the source domain data is minimized.
[0004] There are two problems with the current density map counting method: First, due to the large difference in the characteristics of the source domain data and the target domain data, relying solely on the gradient reversal (GRL) method to make the domain discriminator and decoder (upsampling) perform adversarial learning cannot guarantee that the source domain data and the target domain data can be mapped to the same space, resulting in poor counting results for some images. Second, this method relies on the Gaussian kernel function to generate the corresponding Ground-Truth density map for point annotation. The quality of the density map will directly affect the counting effect. Due to interference from factors such as target overlap, occlusion, and light in the image itself, the quality of the generated Ground-Truth density map cannot be guaranteed. Summary of the invention
[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide an unsupervised cross-domain plant organ counting method based on minimizing feature differences. The purpose is to propose an unsupervised cross-domain plant organ counting method, which effectively utilizes the existing data sets with labeled information to complete the task of counting organs in the target data set without the need for additional manual labeling.
[0006] To achieve the above object, the present invention provides the following technical solution: an unsupervised cross-domain plant organ counting method based on minimizing feature differences, characterized in that it comprises the following steps:
[0007] S1: Building an improved recurrent generative adversarial network generator;
[0008] S2: Convert the source domain dataset into pseudo data with small feature differences from the target domain dataset;
[0009] S3: Construct density map estimation model;
[0010] S4: Use the built density map estimation model to train the data containing the pseudo dataset and the source domain dataset;
[0011] S5: Use the trained density map estimation model to test on the target domain dataset.
[0012] Furthermore, the step S1 comprises:
[0013] S101: Define two existing agricultural image datasets in different fields. The one with annotation information is called source domain X, and the one without annotation information is called target domain Y.
[0014] S102: Setting a cyclic generative adversarial network, in which two resnet9s with added spatial attention modules and channel attention modules (CBAM) are constructed as generators G and F of the cyclic generative adversarial network and two domain judgers Dx and Dy;
[0015] S103: Generator G implements migration from X to Y, and generator F implements migration from Y to X;
[0016] S104: Use the Adam optimizer to optimize the parameters of the improved Cycle GAN, setting the learning rate to 2×10 -4 , the batchsize is set to 4, and the final generator is obtained through 100 epochs of iterative training.
[0017] Furthermore, in step S2, the cyclic generative adversarial network obtained in step S1 is combined with a source domain dataset, and the source domain dataset is obtained by the generator G trained in step S1 to obtain a pseudo target domain dataset.
[0018] Furthermore, it is characterized in that step S3 comprises:
[0019] S301: The density map estimation model removes the last pooling layer and linear layer of vgg-19 and adds three convolutional layers as output layers. The output density map is 1 / 8 the size of the original image.
[0020] S302: Use Adam optimizer to optimize the model, batch size is set to 10, and learning rate is 10 -4 , after 100 epochs of iterative training, the final model is obtained.
[0021] Furthermore, in step S4, a new training set is obtained by mixing the pseudo data set obtained in step S2 and the source domain data set, and the training process is completed by the model obtained in step S3.
[0022] Furthermore, in step S102, two domain judgers Dx and Dy are constructed, including the ability of training the judger Dx(Dy) to distinguish between pseudo data and real data generated by the generator F(G), and calculating the probability score of the judger Dx(Dy) to judge the category of pseudo data generated by the generator F(G) (whether it is pseudo data or real data).
[0023] Furthermore, in step S301, a density map is generated using the Bayesian loss that only requires point annotation information, the prior probability of each pixel belonging to each target is solved by the likelihood function, and then the expectation is calculated in combination with the density map output by the model, and then the loss is calculated using the defined Bayesian loss function.
[0024] Compared with the prior art, the present invention has the following beneficial effects:
[0025] Improve the existing unsupervised cross-domain plant organ counting method, reduce the feature differences between source domain data and target domain data, and ensure that the model can better map the data of the two domains into the same space.
[0026] The present invention solves the prior probability that each pixel belongs to each target through the likelihood function, then calculates the expectation in combination with the density map output by the model, and then uses the defined Bayesian loss function to calculate the loss; thus avoiding the problem that the density map converted from point annotations using the Gaussian kernel function is strongly interfered by other factors. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.
[0028] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.
[0029] Figure 1 This is a flow chart of an unsupervised cross-domain plant organ counting method based on minimizing feature differences in the present invention.
[0030] Figure 2 This is a schematic diagram of an unsupervised cross-domain plant organ counting method based on minimizing feature differences according to the present invention.
[0031] Figure 3 The present invention provides an unsupervised cross-domain plant organ counting method based on minimizing feature differences in images from two different domains.
[0032] Figure 4 This is the detailed structure of a generator for an unsupervised cross-domain plant organ counting method based on minimizing feature differences in the present invention.
[0033] Figure 5 This is the detailed structure of the residual block of an unsupervised cross-domain plant organ counting method based on minimizing feature differences in the present invention.
[0034] Figure 6 The detailed structure of the domain discriminator of the unsupervised cross-domain plant organ counting method based on minimizing feature differences in the present invention.
[0035] Figure 7 The present invention discloses an unsupervised cross-domain plant organ counting method based on minimizing feature differences (a) source domain image and (b) target domain data image.
[0036] Figure 8 The present invention is a flow chart of a density map estimation method for an unsupervised cross-domain plant organ counting method based on minimizing feature differences.
[0037] Fig. 9 This is an unsupervised cross-domain plant organ counting method based on minimizing feature differences in the present invention. (a) The original image of the target domain and (b) the obtained density map. DETAILED DESCRIPTION
[0038] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.
[0039] Example 1
[0040] An unsupervised cross-domain plant organ counting method based on minimizing feature differences, including:
[0041] S1 builds an improved generator of the cyclic generative adversarial network, such as Figure 2 As shown in the dashed box 1, the two generators G, the two generators F, the two domain judges Dx, and the two domain judges Dy share parameters; the solid line with an arrow indicates the test phase, and the dashed line with an arrow only exists in the training phase; f_y(x) is a pseudo target domain image obtained by passing the source domain image x(y) through the generator G(F), and rec_x(y) is a pseudo source domain image obtained by passing the pseudo target domain image f_y(x) through the generator F(G);
[0042] It is known that there are two agricultural image datasets in different fields, one with annotation information is called source domain X, and the other without annotation information is called target domain Y. Figure 3 As shown, (a) the source domain X is taken from CVPP 2017LCC, and (b) the target domain Y is taken from KOMATSUNA.
[0043] The cyclic generative adversarial network includes two resnet9s with added spatial attention modules and channel attention modules (CBAM) as generators G and F of the cyclic generative adversarial network; two domain judgers Dx and Dy: (1) train the judger Dx (Dy) to distinguish the ability of pseudo data generated by the generator F (G) and real data; (2) calculate the probability score of the judger Dx (Dy) to judge the category of pseudo data generated by the generator F (G) (whether it is pseudo data or real data); three loss functions The mean square error (MSE) loss is used for back propagation to improve the judgment ability of the discriminator and the transfer ability of the generator; is the mean absolute error (MAE) loss, which is back-propagated to improve the transfer ability of the generator.
[0044] The detailed structure of the generator is as follows Figure 4 As demonstrated, the detailed structure of the residual block is as follows Figure 5 shown. Figure 4 Middle and Figure 5 Image enhancement is ReflectionPad2d, regularization is InstanceNorm, activation function 1 is Leaky ReLU activation function, activation function 2 is ReLU activation function, and activation function 3 is Tanh activation function; the parentheses represent batchsize, data length and width, and number of channels respectively.
[0045] Generator G implements the migration from X to Y, and generator F implements the migration from Y to X. The detailed structure of the generator is as follows Figure 3As shown, it is mainly divided into a decoder and an encoder structure. The decoder uses convolution, IN regularization, and Leaky ReLU activation function. In addition, image enhancement (ReflectionPad2d) is used to make the image symmetrical along the edges, thereby increasing the image resolution. The Residual block module is used in the two structural link areas. The default is 9 repeated modules to restore and enhance the data. The encoder part uses deconvolution, IN normalization, and ReLU activation function to restore the image size. Finally, the image resolution is improved through ReflectionPad2d, and the image is restored to its original size through convolution, effectively resolving the object edge information. The domain discriminator Dx is used to judge the generation ability of F, and the domain discriminator Dy is used to judge the generation ability of G. The detailed structure of the domain discriminator is as follows Figure 6 Its loss is shown in formulas (1), (2) and (3).
[0046]
[0047]
[0048]
[0049] Among them, G and F represent generators, D X and D y represents the discriminator, x is an image from the source domain dataset X, and y is an image from the target domain dataset Y.
[0050] The Adam optimizer is used to optimize the parameters of the improved Cycle GAN. The learning rate is set to 2×10 -4 , the batchsize is set to 4, and the final generator is obtained through 100 epochs of iterative training.
[0051] S2: Use the cyclic generative adversarial network obtained in step S1 to combine with the source domain dataset, and convert the source domain dataset into a pseudo dataset with small feature differences from the target domain dataset; the source domain dataset is converted into a pseudo target domain dataset through the generator G trained in step S1. The result is as follows: Figure 7 As shown, (a) source domain image from CVPPP data, (b) pseudo KOMATSUNA target domain data image obtained by the generator.
[0052] S3: Construct a density map estimation model, Figure 8 As shown in the figure, 512×512×3 represents the length and width of the image and the number of channels, and (512, 256, 3, 1) represents the input channel, output channel, convolution kernel size, and step size respectively.
[0053] The density map estimation model removes the last pooling layer and linear layer of vgg-19, and adds three convolutional layers as the output layer. The output density map is 1 / 8 of the size of the original image. Unlike the traditional density map estimation method, the present invention does not use a Gaussian kernel function to generate a density map, but uses a Bayesian loss that only requires point annotation information. The prior probability of each pixel belonging to each target is solved by the likelihood function, and then the expectation is calculated in combination with the density map output by the model, and then the loss is calculated using the defined Bayesian loss function. It can be seen from formulas (4) to (7).
[0054]
[0055]
[0056] Let c n is with z n The total number of related is the pixel x m Right n Contribution. n The expectations are:
[0057]
[0058] Where m is the number of pixels and n is the number of targets. Bayesian loss can be defined as:
[0059]
[0060] The model is optimized using the Adam optimizer. The batch size is set to 10 and the learning rate is 10. -4 After 100 epochs of iterative training, the present invention obtains the final model.
[0061] S4: Use the density map estimation model built in step S3 to train the data containing the pseudo data set and the source domain data set;
[0062] The pseudo dataset obtained in step S2 is mixed with the source domain dataset to obtain a new training set, and the training process is completed by the model obtained in step S3. Figure 1 As shown in the dotted box 2.
[0063] S5: Use the density map estimation model trained in step S4 to test on the target domain dataset. The results are as follows: Fig. 9 As shown, it can be seen from the figure that the generated density map shows accurate plant organ counts without interference.
[0064] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.
Claims
1. An unsupervised cross-domain plant organ counting method based on minimizing feature differences, It is characterized in that The following steps are involved: S1: Building an improved recurrent generative adversarial network generator; S2: Convert the source domain dataset into pseudo data with small feature differences from the target domain dataset; S3: Construct density map estimation model; S4: Use the built density map estimation model to train the data containing the pseudo dataset and the source domain dataset; S5: Use the trained density map estimation model to test on the target domain dataset; The step S1 comprises: S101: Define two existing agricultural image datasets in different fields. The one with annotation information is called source domain X, and the one without annotation information is called target domain Y. S102: Set up a cyclic generative adversarial network, in which two resnet9s with added spatial attention modules and channel attention modules (CBAM) are constructed as generators G, F and two domain judgers of the cyclic generative adversarial network Dx、Dy ; S103: Generator G accomplish X arrive Y Migration, Generator F accomplish Y arrive X migrate; S104: Use the Adam optimizer to optimize the improved Cycle GAN parameters and set the learning rate to , the batchsize is set to 4, and the final generator is obtained through 100 epochs of iterative training; In step S2, the cyclic generative adversarial network obtained in step S1 is combined with the source domain dataset, and the source domain dataset is trained by the generator in step S1. G Get a pseudo target domain dataset; The step S3 comprises: S301: The density map estimation model removes the last pooling layer and linear layer of vgg-19 and adds three convolutional layers as output layers. The output density map is 1 / 8 the size of the original image. S302: Use Adam optimizer to optimize the model, batch size is set to 10, and learning rate is 10 -4 , after 100 epochs of iterative training, the final model is obtained.
2. According to claim 1, an unsupervised cross-domain plant organ counting method based on minimizing feature differences, It is characterized in that In step S4, a new training set is obtained by mixing the pseudo data set obtained in step S2 and the source domain data set, and the training process is completed by the model obtained in step S3.
3. According to claim 1, an unsupervised cross-domain plant organ counting method based on minimizing feature differences, It is characterized in that In step S102, two domain judgers are constructed Dx、Dy , including the ability of training the judge Dx(Dy) to distinguish between pseudo data and real data generated by the generator F(G), and calculating the probability score of the judge Dx(Dy) to judge the category of pseudo data generated by the generator F(G) (whether it is pseudo data or real data).
4. According to claim 1, an unsupervised cross-domain plant organ counting method based on minimizing feature differences, It is characterized in that In step S301, a density map is generated using the Bayesian loss that only requires point annotation information, the prior probability of each pixel belonging to each target is solved by the likelihood function, and then the expectation is calculated in combination with the density map output by the model, and then the loss is calculated using the defined Bayesian loss function.
