Wheat growth period identification method based on multi-modal characteristics

Abstract

The invention discloses a wheat growth period identification method based on multi-modal features, and belongs to the technical field of crop growth period identification. Image data features of a wheat growth period are acquired by using a multi-scale convolution sum attention mechanism technology, and the wheat growth period is identified by adding asymmetric multi-channel convolution. Multi-scale extraction of canopy image features is carried out and a nonlinear expression capability is increased at the same time; the attention mechanism module is introduced to enable the network to pay more attention to main effective features, interference features are filtered, the capability of extracting key information from redundant information is enhanced, the recognition accuracy is improved, the problem of canopy image area weight dispersion is solved, the obtained wheat growth period image data features are more accurate and comprehensive, and the wheat growth period image data feature extraction efficiency is improved. According to the image data features of the wheat growth period, on one hand, the image features can be directly classified to identify the growth period, and on the other hand, the image data features of the wheat growth period and serialized environmental data features can be fused; the method realizes small difference and high similarity of morphological characteristics of plant canopies in the transition period of adjacent growth periods, and also can accurately identify the growth period in which the plant canopies are located.

Description

Technical Field

[0001] The invention belongs to the technical field of crop growth period identification, and in particular relates to a wheat growth period identification method based on multimodal features. Background Art

[0002] Traditionally, the acquisition of information about the growth period of crops relies on manual observation, which is an inefficient process. With the development of deep learning image understanding technology, the research on automatic real-time identification of crop growth periods based on RGB images has received attention and has achieved certain research results, but most existing technical methods only use image feature information for identification, and their accuracy is limited. Due to factors such as field environmental growth conditions, especially in the face of adjacent periods between different growth periods, the morphological and structural characteristics of crop canopies are slightly different during the transition period and have high similarity. How to accurately identify different growth periods still faces challenges. Summary of the invention

[0003] In view of the deficiencies described in the above-mentioned prior art, the present invention provides a method for identifying the growth period of wheat based on multimodal features. By processing the image data features of the growth period of winter wheat differently, two growth period identification methods can be obtained.

[0004] The technical solution adopted by the present invention is: a method for identifying wheat growth periods based on multimodal features, the steps are:

[0005] S1, extracting wheat canopy image features using a multi-scale convolutional neural network model;

[0006] The multi-scale convolutional neural network model includes a shallow feature extraction module, a deep feature extraction module, a multi-scale feature extraction module and a feature filtering module; the input of the multi-scale convolutional neural network model is a winter wheat canopy image;

[0007] The shallow feature extraction module is used to extract shallow features of the input image to obtain a shallow feature map. The shallow features are local features of the image, which are relatively close to the input image, but contain more pixel information, mainly including fine-grained information such as color, texture, edge and corner. Since the shallow feature extraction module, also known as the shallow network, has a small receptive field and a small receptive overlap area, it can extract more detailed information;

[0008] The deep feature extraction module is used to extract deep features from the shallow feature map to obtain the deep feature map, and the number of feature channels of the deep feature extraction module is greater than the number of feature channels of the shallow feature extraction module; the deep features are global features of the image, which are very close to the output and contain more abstract semantic information, mainly including coarse-grained information. The deep feature extraction module, also known as the deep network, has an increased receptive field, an increased receptive field overlap area, and compressed image information, and can obtain information about the integrity of the image;

[0009] The multi-scale feature extraction module uses convolution kernels of different sizes to extract features from the deep feature map to obtain a multi-scale feature map; the multi-scale feature extraction module can extract more accurate feature information, obtain more image attributes, and improve the image recognition effect. The multi-scale can avoid the problem of edge information being processed due to a single network scale and fuzzy recognition, which ultimately leads to classification errors, so that the obtained features have richer spatial features and extract more wheat canopy image features;

[0010] The feature filtering module is used to eliminate interference features in the multi-scale feature map to obtain wheat canopy image features. The multi-scale feature map is used as input. For the multi-scale feature map, it first passes through three convolution layers, and then passes the convolution-processed features into the CBAM module (Convolutional Block Attention Module, hybrid attention mechanism) for feature weight allocation. The CBAM module is a hybrid attention mechanism, which connects spatial attention and channel attention in a serial manner. After that, the input features can be weighted fused. The convolution features after the CBAM module reallocates the weights are short-circuited with the multi-scale features at the time of input, and the output is the wheat canopy image feature after activation;

[0011] S2, using LSTM network model to extract environmental characteristics of wheat growth;

[0012] The LSTM network model includes two LSTM network layers and one fully connected layer; the number of hidden layers in the LSTM network layer is 2, the number of hidden layer nodes is 128, and the number of nodes in the fully connected layer is 512; wherein, LSTM is Long Short-Term Memory, a long short-term memory network.

[0013] S3, fusing wheat canopy image features with environmental features to obtain fused features;

[0014] S4, using a deep neural network to process the fusion features to identify the growth period of the wheat. The growth period of the wheat includes the seedling stage, the tillering stage, the wintering stage, the greening stage, and the jointing stage.

[0015] As a preferred embodiment of the present invention, the present invention also provides a method for identifying the growth period of the wheat by directly inputting the wheat canopy image features into a global average pooling layer for sampling after obtaining the wheat canopy image features in step S1, and inputting the wheat canopy target image feature vector into a fully connected layer for processing and then processing it through a classification function.

[0016] As a preferred solution of the present invention, the shallow feature extraction module is provided with three convolutional layers and one maximum pooling layer. Each convolutional layer structure uses Conv2d+BN+ReLU to extract shallow features from the input image of the current layer, and the convolution kernel size of the first convolutional layer is the largest in order to obtain a larger receptive field. The convolution kernel sizes of the remaining two convolutional layers are the same. The convolution kernel of the first convolutional layer is 7×7, and the convolution kernels of the remaining two convolutional layers are 3×3. When each convolutional layer is working, a padding of 1 is inserted, a step size of 1 is inserted, and maximum pooling downsampling is performed after the convolution is completed.

[0017] As a preferred solution of the present invention, the deep feature extraction module is provided with two convolutional layers, each convolutional layer adopts Conv2d+BN+ReLU method to extract deep features from the input of the current layer; and the convolution kernels of the two convolutional layers are the same size as the convolution kernels of the remaining two convolutional layers of the shallow feature extraction module.

[0018] As a preferred solution of the present invention, the multi-scale feature extraction module is provided with two Inception-A structure layers connected in series; each Inception-A structure layer includes a first convolution branch, a second convolution branch, a third convolution branch and a pooling branch, the first convolution branch, the second convolution branch and the third convolution branch are connected in parallel with the pooling branch, and the convolution kernel of the first convolution branch is the smallest; the convolution kernel size of the second convolution branch and the third convolution branch is the same, and the third convolution branch is two convolution layers connected in series; the first convolution branch, the second convolution branch and the third convolution branch are spliced ​​with the pooling branch in the channel dimension as the output of the current Inception-A structure layer. The convolution kernel of the first convolution branch is 1×1; the convolution kernel of the second convolution branch and the third convolution branch is 3×3; the pooling branch is 3×3 maximum pooling. The three convolution branches present an asymmetric structure to achieve the extraction of multi-scale wheat features.

[0019] As a preferred solution of the present invention, the input of the hybrid attention mechanism LSTM network model is the environmental data feature vector of wheat growth, and the elements of the environmental data feature vector include temperature, light, soil moisture and air humidity; the output of the hybrid attention mechanism LSTM network model is the environmental data serialization feature, and the environmental data serialization feature length is the same as the wheat canopy image feature length.

[0020] As a preferred solution of the present invention, the training data of the LSTM network model is a time series environmental data set. The time series environmental data set is obtained by merging the environmental data of the current acquisition time with the label of the wheat growth period with the image data acquisition time as the alignment anchor point. Different wheat growth periods have different labels.

[0021] The present invention utilizes multi-scale convolution and attention mechanism technology to obtain image data features of wheat growth period, and by adding asymmetric multi-channel convolution, canopy image features are extracted at multi-scale while increasing nonlinear expression capability; by introducing an attention mechanism module, the network pays more attention to main effective features, and at the same time filters interference features, enhances the ability to extract key information from redundant information, improves recognition accuracy, and solves the problem of weight dispersion in canopy image regions, so that the obtained image data features of wheat growth period are more accurate and comprehensive. Using the image data features of wheat growth period, on the one hand, the image features can be directly classified to identify the growth period, and on the other hand, the image data features of wheat growth period can be fused with serialized environmental data features, and through collaborative reasoning of multiple heterogeneous modal data, the canopy morphological features of plants with small differences and high similarities during the transition period between adjacent growth periods can also accurately identify the growth period. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 This is a recognition principle diagram of the wheat growth period recognition method based on multimodal features of the present invention.

[0024] Figure 2 This is a structural principle diagram of the multi-scale convolutional neural network model of the present invention.

[0025] Figure 3 This is a structural principle diagram of the multi-scale feature extraction module of the present invention.

[0026] Figure 4 This is a structural principle diagram of the feature filtering module of the present invention.

[0027] Figure 5 It is a structural principle diagram of the CBAM module of the present invention.

[0028] Figure 6 This is a structural principle diagram of the LSTM network model of the present invention.

[0029] Figure 7 This is the variety collection map of the wheat canopy image dataset of the present invention.

[0030] Figure 8 Schematic diagram of the image data enhancement method of the present invention.

[0031] Fig. 9 This is the accuracy training diagram of the MMN-W model of the present invention.

[0032] Fig.10 This is the loss rate training diagram of the MMN-W model of the present invention.

[0033] Fig.11 This is a schematic diagram of the method for identifying the growth period of wheat using the multi-scale convolutional neural network model of the present invention.

[0034] Fig.12 This is a comparison chart of the accuracy test between the MSN-W model of the present invention and a typical convolutional neural network.

[0035] Fig.13 This is the confusion matrix diagram of the MMN-W model of the present invention.

[0036] Fig.14 This is the confusion matrix diagram of the MSN-W model of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] The present invention provides an embodiment of a wheat growth period identification method based on multimodal features, such as Figure 1 As shown, the steps are:

[0039] S1, extracting wheat canopy image features using a multi-scale convolutional neural network model;

[0040] The multi-scale convolutional neural network model, such as Figure 2 As shown, it includes a shallow feature extraction module, a deep feature extraction module, a multi-scale feature extraction module and a feature filtering module; the input of the multi-scale convolutional neural network model is a winter wheat canopy image;

[0041] The shallow feature extraction module is used to extract shallow features of the input image to obtain a shallow feature map. The shallow features are local features of the image, which are relatively close to the input image, but contain more pixel information, mainly including fine-grained information such as color, texture, edge and corner. Since the shallow feature extraction module, also known as the shallow network, has a small receptive field and a small receptive overlap area, it can extract more detailed information;

[0042] The shallow feature extraction module is equipped with three convolutional layers and one maximum pooling layer. Each convolutional layer uses Conv2d+BN+ReLU to extract shallow features from the input image of the current layer. The convolution kernel size of the first convolutional layer is the largest in order to obtain a larger receptive field. The convolution kernel sizes of the remaining two convolutional layers are the same. The convolution kernel of the first convolutional layer is 7×7, and the convolution kernels of the remaining two convolutional layers are 3×3. When each convolutional layer is working, the padding is inserted as 1 and the step size is 1. After the convolution is completed, the maximum pooling downsampling is performed.

[0043] The deep feature extraction module is used to extract deep features from the shallow feature map to obtain the deep feature map, and the number of feature channels of the deep feature extraction module is greater than the number of feature channels of the shallow feature extraction module; the deep features are global features of the image, which are very close to the output and contain more abstract semantic information, mainly including coarse-grained information. The deep feature extraction module, also known as the deep network, has an increased receptive field, an increased receptive field overlap area, and compressed image information, and can obtain information about the integrity of the image.

[0044] The deep feature extraction module is provided with two convolutional layers, each convolutional layer extracts deep features from the input of the current layer by using the Conv2d+BN+ReLU method; and the convolution kernels of the two convolutional layers are the same size as the convolution kernels of the remaining two convolutional layers of the shallow feature extraction module, the convolution kernel size is 3×3, the padding is 1, the step size is 1, and the activation function is ReLU. After convolution processing of the two convolutional layers, maximum pooling downsampling is performed to obtain a deep feature map;

[0045] The multi-scale feature extraction module uses convolution kernels of different sizes to extract features from the deep feature map to obtain a multi-scale feature map; the multi-scale feature extraction module can extract more accurate feature information, obtain more image attributes, and improve the image recognition effect. The multi-scale can avoid the problem of edge information being processed due to a single network scale and fuzzy recognition, which ultimately leads to classification errors, so that the obtained features have richer spatial features and extract more wheat canopy image features;

[0046] In this embodiment, Figure 3As shown, the multi-scale feature extraction module is provided with two Inception-A structure layers connected in series; each Inception-A structure layer includes a first convolution branch, a second convolution branch, a third convolution branch and a pooling branch, the first convolution branch, the second convolution branch and the third convolution branch are connected in parallel with the pooling branch, and the convolution kernel of the first convolution branch is the smallest; the convolution kernel size of the second convolution branch and the third convolution branch is the same, and the third convolution branch is two convolution layers connected in series; the first convolution branch, the second convolution branch and the third convolution branch are spliced ​​with the pooling branch in the channel dimension as the output of the current Inception-A structure layer. The convolution kernel of the first convolution branch is 1×1; the convolution kernel of the second convolution branch and the third convolution branch is 3×3; and the pooling branch is 3×3 maximum pooling. The three convolution branches present an asymmetric structure to achieve the extraction of multi-scale wheat features.

[0047] The feature filtering module, such as Figure 4 As shown in the figure, it is used to eliminate the interference features in the multi-scale feature map to obtain the wheat canopy image features. The multi-scale feature map is used as input. For the multi-scale feature map, it first passes through three convolutional layers. The first two convolutional layers use the Conv2d+BN+ReLU method to process the input of the current layer. The last convolutional layer first performs convolution in the Conv2d+BN method, and then passes the convolutional processed features into the CBAM (Convolutional Block Attention Module) module for feature weight allocation. The CBAM module is a hybrid attention mechanism. The principle diagram is as follows Figure 5 As shown in the figure, after the spatial attention and channel attention are connected in series, the input features can be weighted fused, and the convolution features after the weights are redistributed by the CBAM module are short-circuited with the multi-scale features at the time of input, and finally activated with the ReLU activation function to output the wheat canopy image features; by increasing the feature weight of the image through CBAM, the feature is strengthened, which further improves the recognition ability of the model while improving the robustness of the model and enhancing the fitting ability of the model.

[0048] Because wheat growth and development is the result of the combined effects of crops, environment and management measures, the image features of its morphological structure in different growth periods are a reflection of the combined effects. In order to further improve the recognition accuracy of high-similarity image features in adjacent time periods during different growth periods of wheat, the image data features of wheat growth periods are fused with the environmental data features to construct a wheat growth period recognition model based on multimodal features. Therefore, it is necessary to extract and fuse the environmental features. The specific operations are as follows:

[0049] S2, using LSTM network model to extract environmental characteristics of wheat growth;

[0050] The LSTM network model includes two LSTM network layers and one fully connected layer; Figure 6 As shown in the figure, the number of hidden layers in the LSTM network layer is 2, and the number of hidden layer nodes is 128; the number of nodes in the fully connected layer is 512; the input of the LSTM network model is the environmental data feature vector of wheat growth, and the elements of the environmental data feature vector include temperature, light, soil moisture and air humidity; the output of the LSTM network model is the serialization feature of the environmental data, and the length of the serialization feature of the environmental data is the same as the feature length of the wheat canopy image, ensuring the equal importance of image features and environmental features.

[0051] And the training data of the LSTM network model is a time series environmental data set, which is obtained by merging the environmental data of the current acquisition time with the label of the wheat growth period with the image data acquisition time as the alignment anchor point. Different wheat growth periods have different labels. The wheat growth period includes the seedling stage, tillering stage, wintering stage, greening stage, and jointing stage. In this embodiment, the seedling stage is set to a label of 0, the tillering stage is set to a label of 1; the wintering stage is set to a label of 2; the greening stage is set to a label of 3; and the jointing stage is set to a label of 4.

[0052] S3, the wheat canopy image features and environmental features are fused to obtain fused features, and a 1×1024 multimodal feature fusion vector is obtained through Concat to achieve the complementarity of multimodal features;

[0053] S4, uses deep neural network to process the fused features to identify the growth period of wheat.

[0054] The present invention will obtain a wheat growth period recognition model based on multimodal features (MMN-W model). It is necessary to first train the wheat growth period recognition model based on multimodal features. After the training is completed, the wheat growth period to be identified can be obtained directly according to the steps of this embodiment.

[0055] During training, first establish a multimodal dataset using the above steps. The multimodal dataset is a combination of wheat morphological structure image data and time series environmental factor data, but it is necessary to perform feature representation on the wheat image data and environmental data separately.

[0056] Environmental data for wheat:

[0057] First, the environmental data collected during the wheat growth period includes data from all time periods during the wheat growth period. The various sensor data at the winter wheat planting points in the Xuchang base are used as the source of environmental data features. In order to establish the association between the image data and the environmental factor data during the wheat growth period, the image data acquisition time is used as the alignment anchor point to establish an environmental factor data set containing time series such as temperature, light, soil and air humidity. The temperature data includes the highest temperature of the day, the lowest temperature of the day, the average temperature of the day, the effective accumulated temperature during the wheat growth period, and the accumulated temperature of the week before the image acquisition; the light data includes the number of sunshine hours on the day; the soil moisture includes the soil moisture on the day; and the air humidity includes the relative humidity on the day. The following table shows the wheat environmental data collected for some time periods.

[0058] Wheat environmental data table for some periods of time

[0059]

[0060] Then, after collecting the data, the environmental factor data needs to be preprocessed. The preprocessing mainly involves linear interpolation, smoothing, and normalization to ensure the integrity of the data. MinMaxScaler(.) is used to further normalize the target variable.

[0061] Some environmental data after preprocessing are shown in the table.

[0062] Some environmental data after table preprocessing

[0063]

[0064]

[0065] Next, the preprocessed environmental data needs to be classified and labeled into five categories: seedling stage, tillering stage, overwintering stage, greening stage, and jointing stage.

[0066] The image data and environmental factor data of the wheat growth period are aligned with the image data acquisition time as the anchor point. The preprocessed environmental data at the current moment is integrated into the image label to establish an environmental factor dataset containing time series such as temperature, light, soil and air humidity. The number of images in each category and the corresponding labels are shown in the table.

[0067] Table of the correspondence between categories and labels

[0068] category Label quantity Seedling stage 0 3600 Tillering stage 1 3600 Overwintering period 2 3600 Green period 3 3600 Jointing stage 4 3600

[0069] For the image dataset:

[0070] First, we collected wheat canopy images from the experimental area of ​​Xuchang Campus of Henan Agricultural University in Xuchang City, Henan Province (113°8′E, 34°13′N). The experimental area belongs to the northern temperate continental monsoon climate zone, with an average annual temperature of 14.3°C, an average annual rainfall of 711.1 mm, and a frost-free period of 217 days. The length of the wheat experimental area is 60 m, the width is 60 m, and the total area is 3600 m 2 The wheat varieties included Yumai 49 (YM49), Zhoumai 27 (ZM27), Aikang 58 (AK58), and Xinong 509 (XN509). These varieties have universal characteristics and are commonly planted in central and eastern Henan. They are semi-winter varieties. The four wheat varieties were subjected to five different nitrogen fertilizer levels to set up control groups, namely N0 (0kg hm -2 )、N8(120kg hm -2 )、N15(225kg hm -2 )、N22(330kghm -2 ) and N29 (435kg hm -2 ). The wheat planting method at the base is grid planting in a field environment, with a planting density of about 300 plants per square meter and normal field cultivation management. The wheat data collection time is from October 2021 to June 2022. The collection equipment is Nikon 3100 (sensor CMOS, maximum aperture F / 5.6, 14.2 million pixels). During the collection, the camera uses a tripod to fix the height of 1 meter from the ground. The collection method is vertical photography, and the maximum resolution size is 4608×3072 pixels. Data collection is carried out every 3 days, and the collection is carried out from 9 to 11 am. The light is relatively stable, and all images are collected under natural light conditions.

[0071] Wheat canopy images were collected during the five growth stages of winter wheat: emergence-tillering (the first day of the emergence period to the day before tillering, collectively referred to as the emergence period), tillering-overwintering (the first day of the tillering period to the day before the overwintering period, collectively referred to as the tillering period), overwintering-greening (the first day of the overwintering period to the day before the greening period, collectively referred to as the overwintering period), greening-jointing (the first day of the greening period to the day before the jointing period, collectively referred to as the greening period), and jointing-heading (the first day of the jointing period to the day before the heading period, collectively referred to as the jointing period). The variety collection of the wheat canopy image dataset is shown in Tables 1 and 2. Figure 7 shown.

[0072] Table 2. Variety collection of wheat canopy image dataset

[0073]

[0074] Secondly, the collected images are preprocessed. The present invention selects the images of wheat in five growth stages to expand the data set by methods such as rotation, scaling and adding Gaussian noise. After the enhancement, the data set is expanded by 3 times to a total of 18,000 images. Figure 8 As shown, the specific method is as follows:

[0075] (1) Image scaling: Based on the same ratio, image scaling is performed in both vertical and horizontal directions.

[0076] (2) Random flipping: Images are randomly selected from the data and then randomly flipped.

[0077] (3) Gaussian noise: Gaussian noise is added to the image to enhance the diversity of the data, thereby improving the robustness and generalization ability of the training model.

[0078] Again, the preprocessed image set is set. The image set is composed of wheat canopy images collected between October 2021 and June 2022. From the obtained wheat canopy image data, 18,000 images of wheat in five stages, namely, seedling-tillering, tillering-overwintering, overwintering-greening, greening-jointing, and jointing-heading, are screened out.

[0079] The dataset was randomly divided into a training set and a test set in a ratio of 4:1. The number of images in each wheat growth period is shown in the table.

[0080] Table 2. Sample situation of wheat dataset

[0081] Wheat growing period Training set / sheet Test set / sheet Total / sheet Seedling-tillering 2880 720 3600 Tillering-overwintering 2880 720 3600 Overwintering-Returning to Green 2880 720 3600 Greening-Jointing 2880 720 3600 Jointing-heading 2880 720 3600

[0082] The time series dataset and the constructed image dataset were fused to form a multimodal dataset, which was used to train the wheat growth period recognition model with multimodal features.

[0083] The training process is to pass N sample images and their corresponding environmental data to the wheat growth period recognition model with multimodal features each time, extract features separately and then fuse them, use multi-layer perceptron MLP to further fuse the fused features, and then output the probability of each growth period through the softmax layer, and do cross entropy loss with the label. Through the gradient descent algorithm, the loss is fed back to the model, the gradient of each layer in the model is calculated, and then the weight is updated. After a lot of training, the output results of the wheat growth period recognition model with multimodal features are close to the label results. When the loss curve or the accuracy curve tends to be flat, as shown in the model accuracy training process diagram, Fig. 9 As shown and Fig.10 The model loss rate training process shown in the figure shows that the model has learned the sample distribution and achieved the correct extraction of data features.

[0084] The present invention also provides another embodiment of a method for identifying the growth period of wheat, such as Fig.11 As shown, the multi-scale convolutional neural network recognition model MSN-W performs classification and recognition. The main steps are:

[0085] S1, extracting wheat canopy image features based on a multi-scale convolutional neural network model;

[0086] The multi-scale convolutional neural network model, such as Figure 2 As shown, it includes a shallow feature extraction module, a deep feature extraction module, a multi-scale feature extraction module and a feature filtering module; the input of the multi-scale convolutional neural network model is a winter wheat canopy image;

[0087] The shallow feature extraction module is used to extract shallow features of the input image to obtain a shallow feature map. The shallow features are local features of the image, which are relatively close to the input image, but contain more pixel information, mainly including fine-grained information such as color, texture, edge and corner. Since the shallow feature extraction module, also known as the shallow network, has a small receptive field and a small receptive overlap area, it can extract more detailed information;

[0088] The shallow feature extraction module is equipped with three convolutional layers and one maximum pooling layer. Each convolutional layer uses Conv2d+BN+ReLU to extract shallow features from the input image of the current layer. The convolution kernel size of the first convolutional layer is the largest in order to obtain a larger receptive field. The convolution kernel sizes of the remaining two convolutional layers are the same. The convolution kernel of the first convolutional layer is 7×7, and the convolution kernels of the remaining two convolutional layers are 3×3. When each convolutional layer is working, the padding is inserted as 1 and the step size is 1. After the convolution is completed, the maximum pooling downsampling is performed.

[0089] The deep feature extraction module is used to extract deep features from the shallow feature map to obtain the deep feature map, and the number of feature channels of the deep feature extraction module is greater than the number of feature channels of the shallow feature extraction module; the deep features are global features of the image, which are very close to the output and contain more abstract semantic information, mainly including coarse-grained information. The deep feature extraction module, also known as the deep network, has an increased receptive field, an increased receptive field overlap area, and compressed image information, and can obtain information about the integrity of the image.

[0090] The deep feature extraction module is provided with two convolutional layers, each convolutional layer extracts deep features from the input of the current layer by using the Conv2d+BN+ReLU method; and the convolution kernels of the two convolutional layers are the same size as the convolution kernels of the remaining two convolutional layers of the shallow feature extraction module, the convolution kernel size is 3×3, the padding is 1, the step size is 1, and the activation function is ReLU. After convolution processing of the two convolutional layers, maximum pooling downsampling is performed to obtain a deep feature map;

[0091] The multi-scale feature extraction module uses convolution kernels of different sizes to extract features from the deep feature map to obtain a multi-scale feature map; the multi-scale feature extraction module can extract more accurate feature information, obtain more image attributes, and improve the image recognition effect. The multi-scale can avoid the problem of edge information being processed due to a single network scale and fuzzy recognition, which ultimately leads to classification errors, so that the obtained features have richer spatial features and extract more wheat canopy image features;

[0092] In this embodiment, Figure 3 As shown, the multi-scale feature extraction module is provided with two Inception-A structure layers connected in series; each Inception-A structure layer includes a first convolution branch, a second convolution branch, a third convolution branch and a pooling branch, the first convolution branch, the second convolution branch and the third convolution branch are connected in parallel with the pooling branch, and the convolution kernel of the first convolution branch is the smallest; the convolution kernel size of the second convolution branch and the third convolution branch is the same, and the third convolution branch is two convolution layers connected in series; the first convolution branch, the second convolution branch and the third convolution branch are spliced ​​with the pooling branch in the channel dimension as the output of the current Inception-A structure layer. The convolution kernel of the first convolution branch is 1×1; the convolution kernel of the second convolution branch and the third convolution branch is 3×3; and the pooling branch is 3×3 maximum pooling. The three convolution branches present an asymmetric structure to achieve the extraction of multi-scale wheat features.

[0093] The feature filtering module, such as Figure 4 As shown in the figure, it is used to eliminate the interference features in the multi-scale feature map to obtain the wheat canopy image features. The multi-scale feature map is used as input. For the multi-scale feature map, it first passes through three convolutional layers. The first two convolutional layers use the Conv2d+BN+ReLU method to process the input of the current layer. The last convolutional layer first performs convolution in the Conv2d+BN method, and then passes the convolutional processed features into the CBAM (Convolutional Block Attention Module) module for feature weight allocation. The CBAM module is a hybrid attention mechanism. The principle diagram is as follows Figure 5As shown in the figure, after the spatial attention and channel attention are connected in series, the input features can be weighted fused, and the convolution features after the weights are redistributed by the CBAM module are short-circuited with the multi-scale features at the time of input, and finally activated with the ReLU activation function to output the wheat canopy image features; by increasing the feature weight of the image through CBAM, the feature is strengthened, which further improves the recognition ability of the model while improving the robustness of the model and enhancing the fitting ability of the model.

[0094] S2 ‘ After obtaining the wheat canopy image features in step S1, the wheat canopy image features are directly input into a global average pooling layer for sampling to obtain the target image feature vector of the wheat canopy. The target image feature vector of the wheat canopy is input into a fully connected layer (MLP layer) for processing and then processed by a softmax classification function to identify the growth period of the wheat.

[0095] The model parameters used are shown in the table:

[0096] Table model detailed parameters

[0097] Input Layer Network Layer Output size Parameter quantity 224×224×3 Shallow feature extraction module 54×54×64 28896 54×54×64 Deep feature extraction module 26×26×128 221952 26×26×128 Multi-scale feature extraction module 26×26×192 922112 26×26×192 Feature filtering 26×26×512 2954404 26×26×512 Fully connected layer 1×1×512 65664

[0098] The image dataset used to train the model is:

[0099] First, we collected wheat canopy images from the experimental area of ​​Xuchang Campus of Henan Agricultural University in Xuchang City, Henan Province (113°8′E, 34°13′N). The experimental area belongs to the northern temperate continental monsoon climate zone, with an average annual temperature of 14.3°C, an average annual rainfall of 711.1 mm, and a frost-free period of 217 days. The length of the wheat experimental area is 60 m, the width is 60 m, and the total area is 3600 m 2 The wheat varieties included Yumai 49 (YM49), Zhoumai 27 (ZM27), Aikang 58 (AK58), and Xinong 509 (XN509). These varieties have universal characteristics and are commonly planted in central and eastern Henan. They are semi-winter varieties. The four wheat varieties were subjected to five different nitrogen fertilizer levels to set up control groups, namely N0 (0kg hm -2 )、N8(120kg hm -2 )、N15(225kg hm -2 )、N22(330kghm -2 ) and N29 (435kg hm -2). The wheat planting method at the base is grid planting in a field environment, with a planting density of about 300 plants per square meter and normal field cultivation management. The wheat data collection time is from October 2021 to June 2022. The collection equipment is Nikon 3100 (sensor CMOS, maximum aperture F / 5.6, 14.2 million pixels). During the collection, the camera uses a tripod to fix the height of 1 meter from the ground. The collection method is vertical photography, and the maximum resolution size is 4608×3072 pixels. Data collection is carried out every 3 days, and the collection is carried out from 9 to 11 am. The light is relatively stable, and all images are collected under natural light conditions.

[0100] Wheat canopy images were collected during the five growth stages of winter wheat: emergence-tillering (the first day of the emergence period to the day before tillering, collectively referred to as the emergence period), tillering-overwintering (the first day of the tillering period to the day before the overwintering period, collectively referred to as the tillering period), overwintering-greening (the first day of the overwintering period to the day before the greening period, collectively referred to as the overwintering period), greening-jointing (the first day of the greening period to the day before the jointing period, collectively referred to as the greening period), and jointing-heading (the first day of the jointing period to the day before the heading period, collectively referred to as the jointing period). The variety collection of the wheat canopy image dataset is shown in Tables 1 and 2. Figure 7 shown.

[0101] Table 2. Variety collection of wheat canopy image dataset

[0102]

[0103] Secondly, the collected images are preprocessed. The present invention selects the images of wheat in five growth stages to expand the data set by methods such as rotation, scaling and adding Gaussian noise. After the enhancement, the data set is expanded by 3 times to a total of 18,000 images. Figure 8 As shown, the specific method is as follows:

[0104] (1) Image scaling: Based on the same ratio, image scaling is performed in both vertical and horizontal directions.

[0105] (2) Random flipping: Images are randomly selected from the data and then randomly flipped.

[0106] (3) Gaussian noise: Gaussian noise is added to the image to enhance the diversity of the data, thereby improving the robustness and generalization ability of the training model.

[0107] Again, the preprocessed image set is set. The image set is composed of wheat canopy images collected between October 2021 and June 2022. From the obtained wheat canopy image data, 18,000 images of wheat in five stages, namely, seedling-tillering, tillering-overwintering, overwintering-greening, greening-jointing, and jointing-heading, are screened out.

[0108] The dataset was randomly divided into a training set and a test set in a ratio of 4:1. The number of images in each wheat growth period is shown in the table.

[0109] Table 2. Sample situation of wheat dataset

[0110] Wheat growing period Training set / sheet Test set / sheet Total / sheet Seedling-tillering 2880 720 3600 Tillering-overwintering 2880 720 3600 Overwintering-Returning to Green 2880 720 3600 Greening-Jointing 2880 720 3600 Jointing-heading 2880 720 3600

[0111] During training, because the training is based on deep learning tasks, GPU acceleration is required. The main hardware configuration is the CPU is 11th Gen Intel(R)Core(TM)i5-11400H@2.70GHz 2.69GHz, the GPU is NVIDIA 3050Ti, and the RAM is 16GB. The main software environment of the experiment is Windows 10 operating system, CUDA11.0, Anaconda's Python3.7 version, and the IDE used for programming and development is Pycharm2021.2.1.

[0112] In this embodiment, the evaluation indicators of the model include confusion matrix, accuracy, precision, recall rate and F1 value. The confusion matrix consists of four values, namely true positive (TP), false positive (FP), true negative (TN), and false negative (FN).

[0113] The hyperparameters of the model are learning rate and optimizer. Learning rate and optimizer refer to the amplitude and method of updating the parameters in the model network structure during each iterative training. This embodiment uses canopy images of five periods, including the seedling stage, tillering stage, wintering stage, greening stage, and jointing stage, as experimental data. There are 18,000 samples in total, of which 3,600 samples are in each stage, and the ratio of training and testing samples is 4:1. The training batches are set to 16 and 32, the optimizers are SGD and Adam, and the learning rates are 0.005, 0.001, 0.0005, and 0.0001, respectively. The experimental results are shown in the table:

[0114] Table Hyperparameter Comparison

[0115] Learning Rate Adam-16 (%) Adam-32 (%) SGD-16(%) SGD-32(%) 0.005 96.7 96.4 96.5 96.1 0.001 97.6 97.3 97.0 96.9 0.0005 96.8 96.5 97.1 96.9 0.0001 97.3 97.1 96.9 97.2

[0116] It can be seen from the table that when Adam is used as the optimizer, the learning rate is 0.001, and the training batch is 16, the model obtains the best accuracy. Therefore, all models of the present invention finally select the learning rate of 0.001, the Adam optimization algorithm, and 16 batches to train the model.

[0117] A typical convolutional neural network and the recognition model of this embodiment are compared and tested. The test is to use the accuracy of each epoch in the training process of the training set as a control. The model training of 100 epochs can well reflect the model convergence, so the accuracy of the first 100 epochs of the training set is used as a control. The comparison diagram is shown in Figure 12. The recognition model of this embodiment and the resnet50 model show a convergence speed that is better than other models. In terms of stability, the recognition model of the embodiment performs better than the resnet50 model. With the increase of the number of iterations, the performance of different models in various aspects tends to be close under the same number of iterations. Because the parameter scale of the VGG19 model is larger, the convergence speed during training is slightly slower than other models, and the performance is worse than other models. The results show that the recognition model of the embodiment shows better learning ability.

[0118] The recognition model MSN-W of this embodiment is then compared with typical convolutional neural networks, such as VGG19, Resnet50, MobileNetV3, and DenseNet, to test the accuracy of reproductive period recognition. The test results are shown in the table:

[0119] Table 3. Results of accuracy of wheat growth period

[0120] Model VGG19 Resnet50 MobileNetV3 DenseNet MSN-W Accuracy 95.4％ 96.3％ 95.6％ 97.0％ 97.6％

[0121] It can be seen that the MSN-W model has the highest accuracy, with an accuracy rate of 97.6% in identifying the growth period of wheat, which is better than the typical convolutional neural network model.

[0122] In order to have a more comprehensive understanding of the performance of the evaluation model, the following table shows the precision, recall and F1 value of the VGG19, Resnet50, MobileNetV3, DenseNet and MSN-W models on the test set.

[0123] Table 2 Model accuracy results on the test set

[0124] Reproductive period VGG19 Resnet50 MobileNetV3 DenseNet MSN-W Seedling stage 98.4％ 98.4％ 97.9％ 98.7％ 98.9％ Tillering stage 97.5％ 96.9％ 97.2％ 97.2％ 98.3％ Overwintering period 95.1％ 95.8％ 95.8％ 95.84％ 97.2％ Green period 90.2％ 95.1％ 91.6％ 96.53％ 95.6％ Jointing stage 95.6％ 95.2％ 95.6％ 97.08％ 97.9％

[0125] Table 2 Model recall results on the test set

[0126] Reproductive period VGG19 Resnet50 MobileNetV3 DenseNet MSN-W Seedling stage 98.8％ 98.1％ 98.7％ 99.0％ 99.0％ Tillering stage 96.4％ 97.0％ 96.5％ 97.9％ 97.3％ Overwintering period 94.4％ 96.2％ 96.3％ 96.7％ 97.9％ Green period 92.7％ 92.6％ 92.8％ 94.6％ 96.7％ Jointing stage 94.5％ 97.1％ 93.8％ 97.9％ 97.3％

[0127] Table 1 Model's f1 value results on the test set

[0128]

[0129]

[0130] From the above table data, we can see that from the evaluation index, the MSN-W model shows a high accuracy in identifying different growth periods of wheat. The model performs better than VGG19, Resnet50, MobileNetV3, DenseNet and other models by 1 to 2 percentage points in the seedling stage, tillering stage, wintering stage and jointing stage. Although the MSN-W model does not perform as well as DenseNet in the greening period, the recall rate and f1 value of the model in this period reach 96.7% and 0.963, which are 2.1% and 0.007 better than the DenseNet model respectively. The experimental results show the superiority of the MSN-W model.

[0131] In order to further verify the effectiveness of the MSN-W model construction method for wheat growth period recognition, resnet50 was used as the baseline model to conduct MSN-W model module ablation comparison experiments, and the model was evaluated on performance indicators such as accuracy and average recall rate of wheat growth period recognition. The phased experiment method was used for verification, and the ablation experiment was conducted using the same data set and training parameters. The accuracy evaluation results are shown in the table.

[0132] Table 2 Comparison results of model ablation experiment accuracy

[0133]

[0134] As can be seen from the above table, in terms of accuracy, the MSN-W model has an accuracy of 97.6% in identifying the growth period of wheat, which is 1.3% higher than the 96.3% accuracy of the Baseline model; it is 1.1% higher than the 96.5% accuracy of the Baseline+multi-scale model and 0.4% higher than the 97.2% accuracy of the Baseline+attention model; the Baseline+multi-scale model and the Baseline+attention model are 0.2% and 0.9% higher than the Baseline model, respectively.

[0135] Table 2 Ablation experiment recall results

[0136]

[0137] The recall evaluation results are shown in the table. In terms of recall, taking the seedling stage as an example, the recall rate of the MSN-W model in the seedling stage reached 99%, which was 0.9% higher than the 98.1% model of the Baseline model; it was 0.3% higher than the 98.7% of the Baseline+multi-scale model and 0.2% higher than the 98.8% of the Baseline+attention model; the Baseline+multi-scale model and the Baseline+attention model were 0.6% and 0.7% higher than the Baseline model, respectively. From the above analysis, it can be seen that both the multi-scale convolution module and the attention mechanism can improve the model performance, and the attention mechanism can improve the model performance better than the multi-scale convolution module. Multi-scale convolution can achieve attention to global and local features, and the attention mechanism can filter features, allowing the model to pay more attention to effective feature information. When the MSN-W model uses the multi-scale and attention mechanism methods on the model at the same time, the model performance is most significantly improved, proving the effectiveness of the method.

[0138] The MMN-W model of the present invention is a recognition and classification model for multimodal data. An image feature extraction network and an environment feature extraction network extract two types of features respectively, perform feature fusion, and use them for final classification.

[0139] This experiment is mainly to verify the effectiveness of the multimodal feature fusion method, compare the results of single-mode input and multi-mode input in terms of accuracy, precision, recall, and F1 value, and analyze the experimental results. The MMN-W model is a further improvement on the basis of MSN-W. In order to verify the recognition results of different networks based on single-mode or multi-modal feature input, the evaluation index is selected as Acc, as shown in the table, which shows the accuracy indicators of the MSN-W model and the multimodal MMN-W model respectively.

[0140] Table MMN-W model accuracy comparison experiment

[0141] Comparative experiment Accuracy MSN-W Model 97.6％ MMN-W Model 98.5％

[0142] From the experimental results in the above table, we can see that the accuracy of the MMN-W model is 98.5%. Compared with the MSN-W model in the test set, it is found that the accuracy of the MMN-W model is improved by 0.9 percentage points. The results verify the contribution of environmental features to the improvement of model accuracy, and show that the MMN-W model can comprehensively utilize various types of data to achieve better results.

[0143] In order to intuitively understand the classification accuracy of the MMN-W model and the single-modal MSN-W model in each category, the MMN-W model ( Fig.13 ) and MSN-W model confusion matrix diagram ( Fig.14), where the rows in the figure represent predicted values, the columns represent actual values, and the larger the value on the diagonal, the higher the model accuracy.

[0144] By comparing the confusion matrices of the MMN-W model and the MSN-W model in the same test set images, it can be found that the serialized environmental features help reduce the number of misclassifications across growth periods. The number of errors in the classification of wheat in each growth period by the MMN-W model is significantly lower than that of the unimodal model. Taking the wintering period as an example, since the wintering period is close to the tillering period and jointing period, the morphological and structural characteristics in the transition period are small and have high similarity, which makes it difficult to accurately identify and prone to misclassification. It can be seen that the MMN-W model divides the wintering period into the tillering period and the greening period 6 and 5 times, while the MSN-W model is 12 and 8 respectively, and there are corresponding problems in other periods. The experimental results show that the number of misclassifications of the model in each period is lower than that of the MSN-W model, indicating that the serialized environmental features have a good effect on improving the classification accuracy of the model. Therefore, the model using multimodal data features has stronger recognition ability than the single-factor feature model.

[0145] In order to have a more comprehensive understanding of the evaluation model, the following table shows the precision, recall, and F1 value of the MMN-W model and the MSN-W model on the test set.

[0146] Table 1. Various evaluations of the model on the test set

[0147]

[0148] From the above experimental results, it can be found that the MMN-W model introduces environmental data features on the basis of the original data features, which improves the model's precision, recall, and F1 value, and is significantly better than the indicators of the single modality. The main contribution of the MMN-W model proposed in the present invention is not only more accurate in identifying the various growth stages of wheat, but also solves the problem of misclassification of wheat across growth stages to a certain extent. The wheat growth stage recognition model with multimodal features is targeted at practical problems and has a significant improvement effect.

[0149] In summary, the use of multimodal features to identify different growth stages of wheat is better than using only single-modal features, and the method has better reliability and fault tolerance. The multimodal feature fusion model can extract features from different modalities and make full use of the complementarity between different modalities, which plays a certain role in further improving the performance of the model.

[0150] The previous branch of the MMN-W model is the MSN-W model, which is also valuable and can also be used to identify the growth period of wheat. The advantage is that the model is simple and the effect is good. This shows that the MMN-W model is more valuable.

[0151] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0152] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A wheat growth period identification method based on multimodal features, characterized in that: The steps are: S1, extracting wheat canopy image features using a multi-scale convolutional neural network model; The multi-scale convolutional neural network model includes a shallow feature extraction module, a deep feature extraction module, a multi-scale feature extraction module and a feature filtering module; the input of the multi-scale convolutional neural network model is a winter wheat canopy image; The shallow feature extraction module is used to extract shallow features of the input image to obtain a shallow feature map; The deep feature extraction module is used to extract deep features from the shallow feature map to obtain a deep feature map, and the number of feature channels of the deep feature extraction module is greater than the number of feature channels of the shallow feature extraction module; The multi-scale feature extraction module uses convolution kernels of different sizes to extract features from the deep feature map to obtain a multi-scale feature map; The feature filtering module is used to eliminate the interference features in the multi-scale feature map to obtain the wheat canopy image features. The multi-scale feature map is used as input. For the multi-scale feature map, it first passes through three convolution layers, and then passes the convolution-processed features into the CBAM module for feature weight distribution. The convolution features after the weight distribution are short-circuited with the multi-scale features at the time of input, and the output is the wheat canopy image features after activation; S2, using LSTM network model to extract environmental characteristics of wheat growth; The LSTM network model includes two LSTM network layers and one fully connected layer; S3, fusing wheat canopy image features with environmental features to obtain fused features; S4, uses deep neural network to process the fused features to identify the growth period of wheat.

2. The wheat growth period identification method based on multimodal features according to claim 1, characterized in that: After obtaining the wheat canopy image features in step S1, the wheat canopy image features can be directly input into a global average pooling layer for sampling to obtain a target image feature vector of the wheat canopy. The target image feature vector of the wheat canopy is input into a fully connected layer for processing and then processed by a classification function to identify the growth period of the wheat.

3. The wheat growth period identification method based on multimodal features according to claim 1 or 2, characterized in that: The shallow feature extraction module is provided with three convolutional layers and one maximum pooling layer. Each convolutional layer structure adopts Conv2d+BN+ReLU method to extract shallow features of the input image of the current layer, and the convolution kernel size of the first convolutional layer is the largest, and the convolution kernel sizes of the remaining two convolutional layers are the same.

4. The wheat growth period identification method based on multimodal features according to claim 3 is characterized in that: The convolution kernel of the first convolution layer is 7×7, and the convolution kernels of the remaining two convolution layers are 3×3.

5. The wheat growth period identification method based on multimodal features according to claim 3, characterized in that: The deep feature extraction module is provided with two convolutional layers, each convolutional layer adopts Conv2d+BN+ReLU method to extract deep features from the input of the current layer; and the convolution kernels of the two convolutional layers are the same size as the convolution kernels of the remaining two convolutional layers of the shallow feature extraction module.

6. The wheat growth period identification method based on multimodal features according to claim 4, characterized in that: The multi-scale feature extraction module is provided with two Inception-A structure layers connected in series; each Inception-A structure layer includes a first convolution branch, a second convolution branch, a third convolution branch and a pooling branch, the first convolution branch, the second convolution branch and the third convolution branch are connected in parallel with the pooling branch, and the convolution kernel of the first convolution branch is the smallest; the convolution kernel sizes of the second convolution branch and the third convolution branch are the same, and the third convolution branch is two convolution layers connected in series; the first convolution branch, the second convolution branch and the third convolution branch are spliced ​​with the pooling branch in the channel dimension as the output of the current Inception-A structure layer.

7. The wheat growth period identification method based on multimodal features according to claim 6, characterized in that: The convolution kernel of the first convolution branch is 1×1; the convolution kernels of the second and third convolution branches are 3×3; and the pooling branch is 3×3 maximum pooling.

8. The wheat growth period identification method based on multimodal features according to claim 6, characterized in that: The input of the LSTM network model is the environmental data feature vector of wheat growth. The elements of the environmental data feature vector include temperature, light, soil moisture and air humidity. The output of the LSTM network model is the serialization feature of the environmental data, and the length of the serialization feature of the environmental data is the same as the feature length of the wheat canopy image.

9. The wheat growth period identification method based on multimodal features according to claim 7, characterized in that: The training data for the LSTM network model is a time series environmental dataset. The time series environmental dataset is aligned with the image data acquisition time as the anchor point. The environmental data at the current acquisition time is fused with the label of the wheat growth period. Different wheat growth periods have different labels.

10. The wheat growth period identification method based on multimodal features according to claim 8, characterized in that: The wheat growth period includes the seedling stage, tillering stage, wintering stage, greening stage and jointing stage.

Images