A method for anomaly detection in large-scale plant growth images by incorporating relative distance

Abstract

This invention discloses a large-scale plant growth image anomaly detection method incorporating relative distance. It employs supervised learning and recognition of a large number of continuously acquired similar images to autonomously extract anomalous images. A pre-trained ResNet and VAE are fused using a plant temporal image comparison method incorporating relative distance. The PDE extracts the phenological stages of the plants, using this as prior knowledge to establish image pairs with distance hierarchy. These pairs are then trained in a Siamese network using a comparative loss metric based on classification distance. The trained encoder effectively extracts semantic information from plant images. The VAE encoder converts the input data into a low-dimensional latent representation, randomly samples a vector from the latent space, and the decoder decodes this vector into reconstructed data. The presence of anomalies is determined by comparing the differences between the original and reconstructed data. This fusion method exhibits semantic consistency, making the latent representations of plant images more similar. Simultaneously, it demonstrates higher accuracy and robustness in anomaly detection.

Description

Technical Field

[0001] This invention relates to the field of anomaly detection technology for massive plant images, and specifically to a method for anomaly detection of large-scale plant growth images that incorporates relative distance. Background Technology

[0002] Anomaly detection has garnered widespread attention across various disciplines and is particularly important for detecting image anomalies in low-quality images. Low-quality images include those with improper exposure, out-of-focus images, and images occluded by foreign objects. By analyzing and comparing the differences between a given image and normal images or training data, anomalies or anomalous regions can be identified. Such anomaly detection methods can automatically detect images that deviate from the norm and promptly remove these low-quality images, thus ensuring the quality of both the final and training datasets. Compared to manually sifting through a large number of images, computer vision-based anomaly detection methods save manpower and provide accuracy comparable to manual screening.

[0003] Current research on anomaly detection based on supervised learning largely requires a large amount of labeled data, and the accuracy of detection results is low when the proportion of anomalous images is small. Furthermore, supervised learning methods assume that the sample distribution of training and testing data is the same, while in real-world applications, the data distribution may change over time or be affected by external conditions, thus impacting the performance of the trained model on new data. Therefore, researchers are increasingly inclined to use semi-supervised or unsupervised learning methods to detect and filter images of poor quality. However, existing methods often fail to properly handle data with small variations in time-series images, easily leading to overfitting. To overcome these limitations, it is necessary to develop methods for extracting key plant phenotypic features applicable to complex backgrounds, to discover regions with small variations in the data and remove anomalous data, while better capturing sequence dependencies. In addition, developing effective anomalous image processing strategies is crucial to ensure data quality and the accuracy of model training. These improvements can not only save manpower but also improve the accuracy of anomaly detection. Anomaly detection research in agriculture has largely focused on plant diseases and pests, with little attention paid to anomalies in image quality itself. Summary of the Invention

[0004] The purpose of this invention is to propose a large-scale cherry growth image anomaly detection method (LCG-AD, Embedded Prior Distance) that introduces relative distance. This method can detect anomalies such as poor image quality in massive images. This innovative method not only simplifies the workflow and saves manpower and time, but is also applicable to the detection of time-series images throughout the complete growth cycle of plants. It can overcome the problems of existing vision-based methods in terms of applicability and cost.

[0005] To achieve the above objectives, a method for detecting anomalies in large-scale plant growth images by incorporating relative distance is provided, comprising:

[0006] Acquire multiple time-series images of plant growth;

[0007] Phenological period image pairs were constructed based on multiple time-series images;

[0008] The LCG-AD network was used to perform preliminary feature extraction on phenological period image pairs, and the trained LCG-AD network was obtained.

[0009] The trained LCG-AD network is used to identify abnormal image sequences.

[0010] Furthermore, the method for constructing phenological period image pairs is as follows:

[0011] Obtain n image sequences of the target plant. In the selected sequences, budding is taken as the baseline time day0. The start time and duration of different phenological stages are obtained by manually interpreting the image sequences. The average value of each time point represents the phenological stage of the n image sequences, and has a relatively fixed annual growth cycle.

[0012] Extracting the required images requires starting from images divided into phenological periods. Although the average phenological period of randomly selected images can approximate the overall phenological period, the exact times of the phenological period boundaries between images from different time sequences are not the same. Furthermore, since phenological period changes are a gradual process, images at the end of one phenological period and the beginning of the next may contain similar semantic information. Therefore, to achieve automatic and accurate extraction of images from different phenological periods and to maximize the semantic differences between images from different phenological periods, images near the phenological period boundaries are discarded, and only images with a time distance from the critical point are selected. This method effectively reduces the difficulty and workload of manual image selection and improves the semantic diversity of the images.

[0013] The remaining image sequences after discarding are paired up, i.e., Cartesian product operation is performed; for each image pair, based on the two image sequences and phenological period information, they are recorded as the same sequence at the same time ([1,0,0,0]), the same sequence at different times ([0,1,0,0]), different sequences at the same time ([0,0,1,0]), and different sequences at different times ([0,0,0,1]), and are marked and saved using One-hot encoding.

[0014] Furthermore, the LCG-AD network is used to perform preliminary feature extraction on the phenological period image pairs, specifically as follows:

[0015] Image x and image y from the image pair are input into the feature extraction part of the LCG-AD network for pre-training, and then image enhancement processing is performed to obtain image v1 and image v2. Image v1 and image v2 are then fed into the encoder to extract features, resulting in feature vectors h1 and h2. Then, feature vectors h1 and h2 are fed into the projection head of a small neural network containing two ReLU layers and two BN layers to obtain feature vectors z1 and z2.

[0016] Furthermore, the image enhancement processing method is as follows: random cropping and flipping, random brightness, random contrast, random saturation, random hue, and random black and white data enhancement of the image.

[0017] Furthermore, the contrastive loss is obtained using the classification distance: the classification distances of different image pairs are mapped onto a fully connected layer, and the relative distances of different types of image pairs and the actual distances of feature vectors are fused using the classification distances to obtain the contrastive loss; specifically, the Euclidean distance e obtained by subtracting feature vector z1 and feature vector z2 is directly input into the fully connected layer, and then o is obtained, i.e.:

[0018]

[0019] t = eW

[0020] o = Softmax(t)

[0021] In the formula, W is the weight matrix of the fully connected layer, and t and o are feature vectors with the same dimension as the label.

[0022] Furthermore, the error between the category information in the feature vector o and the relative distance information in the label categories of the image pair is obtained through cross-entropy. For any image pair, we have:

[0023]

[0024] In the formula, a jRepresenting labels; i, j = 1, 2, 3, 4;

[0025] The LCG-AD network is trained multiple times, and the loss is repeatedly backpropagated to update the parameters.

[0026] Furthermore, the LCG-AD network is used to identify abnormal image sequences, specifically as follows:

[0027] Image sequences are input into the trained LCG-AD network, where an encoder extracts features to obtain a one-dimensional feature vector. This one-dimensional feature vector is then input into the network's anomaly detection module. In the anomaly detection module, the image data first passes through an input layer, which passes the one-dimensional feature vector to the next layer to obtain a vector m. This vector m is then processed by an encoder to generate a low-dimensional vector, which includes two ReLU activation functions. The encoder progressively maps the image data to the latent space, capturing the important features of the image and expressing the essential features of the original image in a lower-dimensional space, thus obtaining the latent variable g. Two independent fully connected layers are used to output the mean vector μ and the variance vector σ in the latent space, respectively.

[0028] Furthermore, to ensure that the variance vector σ is positive, the original values ​​output by the encoder are subjected to an index-taking transformation, and the dimensions of these two vectors are the same as the dimensions of the latent space, as shown in the following formula:

[0029] μ = Encoder(a)

[0030] log(μ 2 = Encoder(a)

[0031] g~N(μ,σ 2 )

[0032] Where N(μ,σ) 2 It follows a normal distribution;

[0033] A latent variable vector is sampled from the normal distribution represented by the mean and variance of the encoder output: This sampling process uses reparameterization to transform the sampling process into a differentiable operation, that is, by sampling a random noise vector (usually following a normal distribution with a mean of 0 and a variance of 1) from the normal distribution, and then transforming it into a latent variable with the correct mean and variance through linear transformation and bias addition.

[0034] Furthermore, the obtained latent variables are input to the decoder, which inversely converts them into a reconstructed vector m' of the original input image. The decoder also includes two ReLU activation functions to decompress from low dimension to high dimension. After passing through an output layer, a reconstructed one-dimensional feature vector is obtained. The anomaly detection module is trained through backpropagation of loss to detect abnormal images and mark them.

[0035] Furthermore, the total loss is obtained; the total loss includes the reconstruction loss (mean squared error, MSE) and the KL divergence, as shown in the following formula:

[0036]

[0037]

[0038] L total =L recon +L kl

[0039] Where N is the number of images, D is the dimension of the input image, and m ij It is the j-th dimension of the input i-th image, m' ioj It is the corresponding value of the reconstructed output image, μ j and σ j denoted as mean and variance of the encoder output, J is the dimension of the latent space, and ∈ is a small constant used for numerical stability.

[0040] The lengths of the encoder input and decoder output vectors remain unchanged. When m ≈ m', meaning the decoder output can approximately reconstruct the input, it proves that the autoencoder is in a good state. The difference between the output and input is used to measure the image information lost due to compression. Calculate the difference between the input and output, set the loss function, train the network, adjust the parameters, and reduce the total loss.

[0041] Compared with existing technologies, the technical solutions adopted in this invention have the following advantages: This invention proposes a method for detecting anomalies in large-scale plant growth images by introducing relative distance. It primarily targets anomalies that are often overlooked or uncommon, such as poor-quality images due to exposure issues or occlusion by foreign objects. This reduces the reliance on manual screening during anomaly detection and is suitable for applications requiring the processing of large numbers of images. This invention explores plant phenotyping methods and applications based on computer vision, providing technical support for smart agriculture and holding significant importance in the field of plant anomaly detection. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the LCG-AD network structure;

[0043] Figure 2Here is a flowchart of an image anomaly detection method; Specific implementation methods

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit the application; that is, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0045] This embodiment uses cherries as an example, but the method is equally applicable to other crops. The widespread application of this method will drive innovation in deep learning theory and technology in agronomy and botany, enabling in-depth research into crop growth patterns and environmental adaptability, and ultimately benefiting agricultural workers. It has significant theoretical and practical implications for detecting anomalies in poor image quality.

[0046] This embodiment provides a large-scale plant growth image anomaly detection method incorporating relative distance. It involves supervised learning and recognition of a large number of continuously collected similar images, autonomously extracting anomalous images, and fusing a ResNet and a VAE (Variational Autoencoder) pre-trained with a plant time-series image contrastive learning method (PDE) with relative distance. The PDE extracts the phenological stages of the plants, using this as prior knowledge to establish image pairs with distance hierarchy. These pairs are then trained in a Siamese network using a contrastive loss metric based on classification distance, converging the model with relative distance as the objective. The trained encoder can effectively extract semantic information from plant images. The VAE encoder converts the input data into a low-dimensional latent representation, randomly samples a vector from the latent space, and the decoder decodes this vector into reconstructed data. By comparing the differences between the original data and the reconstructed data, anomalies are determined. This fusion method exhibits semantic consistency, making the latent representations of plant images more similar. Simultaneously, it demonstrates higher accuracy and robustness in anomaly detection. Specific implementation methods include:

[0047] Step 1: Obtain multiple time-series images of plant growth;

[0048] Specifically, the multiple time-series images in this embodiment are data collected from January to June 2021 and from December 2021 to May 2022 by Hikvision iDS-2DC4223IW- / GLT(S5) and iDS-2DC2204IW(S6) cameras installed in the laboratory at Feitun Village, Siping Town, Pulandian District, Dalian City and the trellis cultivation garden in Jinpu New District, Dalian City. More than 20,000 time-series images of cherry growth were collected. The image resolution is 1920*1080, and the image format is JPG.

[0049] LCG-AD network structure as follows Figure 1 As shown, during feature extraction pre-training, 21 image sequences containing complete cherry growth images were selected from all unlabeled datasets, numbered 1 to 21, totaling 7373 images. For LCG-AD anomaly detection, 62 complete cherry growth cycles from multiple angles were selected from the unlabeled dataset at 9:00, 12:00, and 15:00 each day, totaling 7405 images. Normal and abnormal images were manually labeled, with 0 for normal and 1 for abnormal. 280 abnormal images were used to verify the accuracy of the LCG-AD method. Considering the variability of the cherry growing environment and the applicability of the research, the selected dataset included the following six anomalies: insufficient light, occlusion, incorrect angle rotation, water droplets, overexposure, out-of-focus, and blurry images. All other data were considered normal images.

[0050] Step 2: Construct phenological period image pairs based on multiple time-series images;

[0051] Specifically, the phenological stages of cherry plants, including dormancy, flowering and budding, fruit setting, and ripening, were obtained through manual interpretation of image sequences, with budding set as the baseline time. Then, the required cherry images were extracted, selecting 12 images from each phenological stage, discarding images near the boundary between phenological stages and choosing those far from the critical time points. When constructing image pairs, 3310 experimental images were extracted from 7373 images in 21 cherry growth image sequences. Pairwise image pairing yielded 56245 image pairs of the same sequence at the same time, 219597 image pairs of the same sequence at different times, 1051553 image pairs of different sequences at the same time, and 4149000 image pairs of different sequences at different times. To balance the training samples, data resampling was performed on different types of image pairs, with 100,000 or 150,000 images sampled for each type of experiment.

[0052] Step 3: Use the LCG-AD network to perform preliminary feature extraction on the phenological period image pairs and obtain the trained LCG-AD network;

[0053] Specifically, the constructed image pairs are input into the LCG-AD network, first passing through the feature extraction part and then pre-trained using the PDE method. During training, classification distance is used as the contrastive loss function to obtain a trained encoder. This encoder can then be transferred downstream for other tasks. After the images enter the network, data augmentation techniques are implemented using the image library in TensorFlow. The input image size is 512*288, using the RMSprop optimizer with a learning rate of 0.001, a discounting factor of 0.9, 10 training iterations, and a batch size of 64 samples per training iteration. Then, a set of 62 selected time-series images covering the complete cherry growth cycle is input into the LCG-AD network again. The feature extraction part uses a ResNet-50 pre-trained with PDE as the encoder to obtain preliminary extracted feature vectors. The results of the combined ablation experiments are shown in Table 1.

[0054] Table 1. Partial schematic diagram of experimental results

[0055]

[0056]

[0057] Step 4: Identify abnormal image sequences using the trained LCG-AD network.

[0058] Specifically, the anomaly detection module consists of two parts: an Encoder and a Decoder. Since the vector received from the feature extraction part has already undergone preliminary feature extraction, the Encoder only serves to transmit data. Next, the data passes through two hidden layers, both using ReLU activation functions. The number of neurons and the number of layers in the hidden layers are (256, 128), compressing the data to a specific dimension. Then, a latent space layer is passed to obtain a lower-dimensional 32-dimensional vector containing all the data's features. At this point, the data is passed to the Decoder, which also uses ReLU activation functions in its two hidden layers, with a hidden layer dimension of (128, 256), inversely symmetrical to the Encoder. The network's loss function consists of mean squared error (MSE) and KL divergence. Both the Encoder and Decoder in the LCG-AD anomaly detection part use the Adam optimizer with a learning rate of 0.001, a discounting factor of 0.9, 100 training iterations, and a batch size of 64 samples per training iteration. After this part of the output, normal images and abnormal images are marked as 0 and 1, respectively. Before and after LCG-AD anomaly detection, three sets of local display accuracy, precision, recall, and F1-score were selected from 62 sets of sequences, and the results are shown in Table 1. In X_a_b, X represents the camera number, a represents the preset point number, and b represents the time point.

[0059] As shown in Table 2 of the experimental results, this method performs excellently in anomaly detection, effectively distinguishing between normal and abnormal data. For the anomaly detection results, the LCG-AD method improves image quality and reliability, providing more reliable semantic information and feature representations to support more accurate image analysis and processing. The obtained anomaly detection results can also be used for other downstream tasks, such as semantic segmentation and plant growth modeling. The experimental flowchart is shown below. Figure 2 As shown, this information lays the foundation for further in-depth research on cherries, helping farmers better manage farm resources, improve production efficiency, and provide better planning for the sale and distribution of agricultural products.

[0060] Table 2 Results of Method Combination

[0061] (a) Accuracy value

[0062]

[0063] (b) F1-score

[0064]

[0065] In this application, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0066] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for detecting anomalies in large-scale plant growth images by incorporating relative distance, characterized in that, include: Acquire multiple time-series images of plant growth; Phenological period image pairs were constructed based on multiple time-series images; The LCG-AD network was used to perform preliminary feature extraction on phenological period image pairs, and the trained LCG-AD network was obtained. The trained LCG-AD network is used to identify abnormal image sequences. The LCG-AD network was used to perform preliminary feature extraction on phenological period image pairs, specifically as follows: Image x and image y from the image pair are input into the feature extraction part of the LCG-AD network for pre-training, followed by image enhancement processing to obtain images v1 and v2. Images v1 and v2 are then fed into the encoder to extract features, resulting in feature vectors h1 and h2. Finally, feature vectors h1 and h2 are fed into a small neural network projection head containing two ReLU layers and two BN layers to obtain feature vectors. z 1 and eigenvectors z 2 ; Contrast loss is obtained using classification distance: The classification distances of different image pairs are mapped onto a fully connected layer. The classification distances are then used to fuse the relative distances of different types of image pairs with the actual distances of the feature vectors to obtain the contrast loss. Specifically, the feature vectors are... z 1 and eigenvectors z 2 Euclidean distance obtained by subtraction e The input is directly fed into the fully connected layer, and then... o ,Right now: In the formula, It is the weight matrix of the fully connected layer. and It is a feature vector with the same dimension as the label; Feature vectors are obtained through cross-entropy. The error between the category information and the relative distance information in the image pair's label categories is given by the following for any image pair: In the formula, Represents a label; i, j = 1, 2, 3, 4; The LCG-AD network is trained multiple times, and the loss is backpropagated continuously to update the parameters; The abnormal image sequence is identified using the trained LCG-AD network, specifically as follows: Image sequences are input into the trained LCG-AD network, where an encoder extracts features to obtain a one-dimensional feature vector. This feature vector is then input into the network's anomaly detection module. In this module, the image data first passes through an input layer, passing the feature vector to the next layer to obtain a vector m. This vector m is then processed by an encoder to generate a low-dimensional vector, which includes two ReLU activation functions. The encoder progressively maps the image data to a latent space, capturing key image features and obtaining latent variables. Two independent fully connected layers are used to output the mean vector μ and variance vector σ in the latent space, respectively. To ensure that the variance vector σ is positive, the original values ​​output by the encoder are subjected to an index-taking transformation, and the dimensions of these two vectors are the same as the dimensions of the latent space, as shown in the following formula: in, It follows a normal distribution; A latent variable vector is sampled from the normal distribution represented by the mean and variance of the encoder output: by sampling a random noise vector from the normal distribution and then transforming it into a latent variable with the correct mean and variance through linear transformation and bias addition; The obtained latent variables are input into the decoder, which inversely converts them into a reconstructed vector m' of the original input image; the decoder also includes two ReLU activation functions to decompress from low dimension to high dimension; and then passes through an output layer to obtain the reconstructed one-dimensional feature vector; The anomaly detection module is trained through backpropagation of loss, detects abnormal images, and marks them.

2. The method for detecting anomalies in large-scale plant growth images by incorporating relative distance as described in claim 1, characterized in that, The method for constructing phenological period image pairs is as follows: Obtain n image sequences of the target plant. In the selected sequences, take budding as the baseline time day0, and obtain the start time and duration of different phenological stages by manually interpreting the image sequences. The average value of each time point represents the phenological stage of the n image sequences. Discard the image sequences that are located at the boundary between phenological periods from among the n image sequences; The remaining image sequences after discarding are paired up, i.e., Cartesian product operation is performed; for each image pair, based on the two image sequences and phenological period information, they are recorded as the same sequence at the same time ([1,0,0,0]), the same sequence at different times ([0,1,0,0]), different sequences at the same time ([0,0,1,0]), and different sequences at different times ([0,0,0,1]), and are marked and saved using One-hot encoding.

3. The method for detecting anomalies in large-scale plant growth images by incorporating relative distance as described in claim 1, characterized in that, The image enhancement processing method is as follows: random cropping and flipping, random brightness, random contrast, random saturation, random hue, and random black and white data enhancement are applied to the image.

4. The method for detecting anomalies in large-scale plant growth images by incorporating relative distance as described in claim 1, characterized in that, Obtain the total loss; the total loss includes the reconstruction loss and the KL divergence, as shown in the following formula: Where N is the number of images, and D is the dimension of the input image. It is the j-th dimension of the input i-th image. It is the corresponding value of the reconstructed output image. and These are the mean and variance of the encoder output, and J is the dimension of the latent space. It is a small constant used for numerical stability.

Images