Network model and device for plant identification, and electronic device

Abstract

The application relates to the technical field of agricultural production, in particular to a network model and device for plant identification and electronic equipment. The network model is used for extracting a plurality of first plant feature maps of different scales, performing feature fusion on plant features included in the plurality of first plant feature maps of different scales, obtaining a plurality of second plant feature maps of different scales, scanning the second plant feature maps by using a sliding window to obtain a plurality of preselected frame regions of different scales including plant feature information, processing the plurality of preselected frame regions of different scales by using a bilinear interpolation method to obtain a plurality of third plant feature maps of the same scale, performing feature recognition and classification on the plurality of third plant feature maps of the same scale to obtain a target feature map including plant features, and through comparison and testing, the target feature map of the plant extracted by using the network model is more accurate and the feature information is more abundant, which is helpful for the research on crop phenotypes.

Description

Technical Field

[0001] The present invention relates to the technical field of agricultural production, and in particular to a network model, device and electronic equipment for plant identification. Background Art

[0002] Increasing global food production requires increasing the effective genetic diversity of breeding germplasm resources by improving genotyping and enhancing phenotyping methods. Effective plant breeding programs rely on plant phenotyping to select for desirable genetic traits. However, traditional plant phenotyping techniques are often manual, time-consuming, and subject to significant human intervention. Currently, plant phenotyping has become a bottleneck in breeding programs.

[0003] In the existing technology, digital cameras are the most widely used sensors in crop phenotyping. They have three spectral channels: red, green, and blue, and can obtain images with high spatial resolution at a low cost. In addition, compared with multispectral and hyperspectral imaging, the impact of the external environment on digital sensor imaging is relatively small. Currently, digital imaging has been widely used in the study of crop phenotypic traits, such as seedling emergence rate, canopy coverage, aboveground biomass, leaf area, nitrogen content, plant height, etc. How to accurately obtain plant images from images obtained by digital cameras is an important part of plant phenotyping analysis. Currently, plant images are obtained by machine learning, image processing, and spectral analysis. However, experiments have found that existing methods have little potential for mining plant images. Therefore, plant images extracted using existing methods are not accurate enough. Summary of the Invention

[0004] The present invention provides a network model, a device and an electronic device for plant identification, which are used to solve the technical problem that plant images extracted in the prior art are not accurate enough.

[0005] In one aspect, the present invention provides a network model for plant recognition, comprising: a first backbone network, a feature pyramid network, a region generation network, a ROI alignment network, and a head network;

[0006] The first backbone network is used to extract features from the input plant image to obtain a plurality of first plant feature maps of different scales;

[0007] The feature pyramid network is used to perform feature fusion on the plant features included in the plurality of first plant feature maps of different scales to obtain a plurality of second plant feature maps of different scales;

[0008] The region generation network is used to scan the second plant feature map using a sliding window to obtain a plurality of pre-selected frame regions of different scales including plant feature information;

[0009] The ROI alignment network is used to process the pre-selected frame areas of the multiple different scales using a bilinear interpolation method to obtain multiple third plant feature maps of the same scale;

[0010] The head network is used to perform feature recognition and classification on the multiple third plant feature maps of the same scale to obtain a target feature map including plant features.

[0011] A network model for crop seedling identification provided by the present invention further includes: a second backbone network, a neck network, and a prediction network;

[0012] The second backbone network is used to extract leaf features in the target feature map to obtain a plurality of first leaf feature maps of different scales;

[0013] The neck network is used to perform feature fusion on the leaf features included in the leaf feature maps of different scales to obtain a plurality of second leaf feature maps of different scales;

[0014] The prediction network is used to perform statistical classification on the plurality of second leaf feature maps of different scales, obtain the number of leaves of the plant, and determine the leaf age of the plant according to the number of leaves.

[0015] According to a network model for crop seedling identification provided by the present invention, the first backbone network includes a plurality of feature extraction units of different dimensions, and the plurality of feature extraction units are connected in sequence;

[0016] The feature extraction unit includes a marker embedding module and a Shuffle Transformer Block module; the marker embedding module is used to adjust the dimension of the input plant image, and the Shuffle Transformer module is used to extract features from the dimension-adjusted plant image to obtain first plant feature maps of different scales.

[0017] According to a network model for crop seedling identification provided by the present invention, the ShuffleTransformer Block module includes: a first standardization module, a WMSA module, a first neighbor window connection module, a second standardization module, a first multi-layer perceptron module, a third standardization module, a Shuffle WMSA module, a second neighbor window connection module, a third standardization module, and a second multi-layer perceptron module, wherein the first standardization module, the WMSA module, the first neighbor window connection module, the second standardization module, the first multi-layer perceptron module, the third standardization module, the Shuffle WMSA module, the second neighbor window connection module, the third standardization module, and the second multi-layer perceptron module are connected in sequence;

[0018] The input end of the first standardization module is connected with the first near-neighbor window connection module in a residual manner; the output end of the WMSA module is connected with the input end of the second standardization module in a residual manner; the output end of the first near-neighbor window connection module is connected with the input end of the third standardization module in a residual manner; the output end of the first multi-layer perception module is connected with the input end of the second near-neighbor window connection module; the output end of the Shuffle WMSA module is connected with the input end of the third standardization module; and the output end of the second near-neighbor window connection module is connected with the output end of the second multi-layer perception module.

[0019] The first standardization module, the second standardization module and the third standardization module are used for normalizing input plant images; the WMSA module is used for dividing the normalized plant images into multiple windows of fixed size and without overlap, and calculating self-attention of each window; the Shuffle WMSA module is used for strengthening target spatial information; the first near-neighbor window connection module and the second near-neighbor window connection module are used for improving information flow of adjacent windows; and the first multi-layer perception module and the second multi-layer perception module are used for mapping multiple extracted features into plant features and background features respectively.

[0020] According to the network model for crop seedling recognition provided by the application, the second backbone network comprises a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a fifth convolution module, an attention module, an SPPF module, a first feature extraction module, a second feature extraction module, a third feature extraction module and a fourth feature extraction module.

[0021] The input end of the first convolution module is used for receiving the target feature map, the output end of the first convolution module is connected with the input end of the second convolution module, the output end of the second convolution module is connected with the input end of the first feature extraction module, the output end of the first feature extraction module is connected with the input end of the third convolution module, the output end of the third convolution module is connected with the input end of the second feature extraction module, the output end of the second feature extraction module is connected with the input end of the fourth convolution module, the output end of the fourth convolution module is connected with the input end of the third feature extraction module, the output end of the third feature extraction module is connected with the input end of the fifth convolution module, the output end of the fifth convolution module is connected with the input end of the fourth feature extraction module, the output end of the fourth feature extraction module is connected with the input end of the attention module, and the output end of the attention module is connected with the input end of the SPPF module.

[0022] The output ends of the first feature extraction module, the second feature extraction module, the third feature extraction module and the SPPF module are used for outputting four groups of first blade feature maps of different scales.

[0023] According to the network model for crop seedling recognition provided by the application, the neck network comprises a first fusion module, a second fusion module, a third fusion module, a fourth fusion module, a fifth fusion module, a sixth fusion module, a fifth feature extraction module, a sixth feature extraction module, a seventh feature extraction module, an eighth feature extraction module, a ninth feature extraction module, a tenth feature extraction module, a sixth convolution module, a seventh convolution module, an eighth convolution module, a ninth convolution module, a tenth convolution module, an eleventh convolution module, a first up-sampling module, a second up-sampling module and a third up-sampling module.

[0024] The input end of the first fusion module is connected with the output end of the first feature extraction module, the output end of the first fusion module is connected with the input end of the fifth feature extraction module, and the output end of the fifth feature extraction module is connected with the input end of the prediction network.

[0025] The input end of the second fusion module is connected with the output end of the second feature extraction module, the output end of the second fusion module is connected with the input end of the sixth feature extraction module, the output end of the sixth feature extraction module is connected with the input end of the seventh convolution module, the output end of the seventh convolution module is connected with the input end of the third fusion module, the output end of the third fusion module is connected with the input end of the seventh feature extraction module, and the output end of the seventh feature extraction module is connected with the input end of the prediction network.

[0026] The input end of the first up-sampling module is connected with the output end of the seventh convolution module, and the output end of the first up-sampling module is connected with the input end of the first fusion module; the input end of the sixth convolution module is connected with the output end of the fifth feature extraction module, and the output end of the sixth convolution module is connected with the input end of the third fusion module.

[0027] The input end of the fourth fusion module is connected with the output end of the third feature extraction module, the output end of the fourth fusion module is connected with the input end of the eighth feature extraction module, the output end of the eighth feature extraction module is connected with the input end of the ninth convolution module, the output end of the ninth convolution module is connected with the input end of the fifth fusion module, the output end of the fifth fusion module is connected with the input end of the ninth feature extraction module, and the output end of the ninth feature extraction module is connected with the input end of the prediction network; the input end of the second up-sampling module is connected with the output end of the ninth convolution module, and the output end of the second up-sampling module is connected with the input end of the second fusion module; the input end of the eighth convolution module is connected with the output end of the seventh feature extraction module, and the output end of the eighth convolution module is connected with the input end of the fifth fusion module.

[0028] The input end of the tenth convolution module is connected with the output end of the SPPF module, the output end of the tenth convolution module is connected with the input end of the sixth fusion module, the output end of the sixth fusion module is connected with the input end of the tenth feature extraction module, and the output end of the tenth feature extraction module is connected with the input end of the prediction network.

[0029] The input end of the third up-sampling module is connected with the output end of the tenth convolution module, and the output end of the third up-sampling module is connected with the input end of the fourth fusion module; the input end of the eleventh convolution module is connected with the output end of the ninth feature extraction module, and the output end of the eleventh convolution module is connected with the input end of the sixth fusion module.

[0030] In another aspect, the present application provides a device for plant identification, comprising:

[0031] A first processing module is configured to perform feature extraction on an input plant image to obtain a plurality of first plant feature maps of different scales;

[0032] A second processing module is configured to perform feature fusion on plant features included in the plurality of first plant feature maps of different scales to obtain a plurality of second plant feature maps of different scales;

[0033] A third processing module is configured to scan the second plant feature maps by using a sliding window to obtain a plurality of preselected frame regions of different scales including plant feature information;

[0034] A fourth processing module is configured to process the plurality of preselected frame regions of different scales by using a bilinear interpolation method to obtain a plurality of third plant feature maps of the same scale;

[0035] A fifth processing module is configured to perform feature identification and classification on the plurality of third plant feature maps of the same scale to obtain a target feature map including plant features.

[0036] According to the plant recognition device provided by the application, the device further comprises:

[0037] The sixth processing module is configured to extract the leaf features in the target feature map to obtain a plurality of first leaf feature maps of different scales.

[0038] The seventh processing module is configured to perform feature fusion on the leaf features included in the leaf feature maps of different scales to obtain a plurality of second leaf feature maps of different scales.

[0039] The prediction module is configured to perform statistical classification on the plurality of second leaf feature maps of different scales, to obtain the number of leaves of the plant, and to determine the leaf age of the plant according to the number of leaves.

[0040] In another aspect, the application provides a plant recognition method, comprising:

[0041] Feature extraction is performed on the input plant image to obtain a plurality of first plant feature maps of different scales.

[0042] Feature fusion is performed on the plant features included in the plurality of first plant feature maps of different scales to obtain a plurality of second plant feature maps of different scales.

[0043] The second plant feature map is scanned by using a sliding window to obtain a plurality of preselected frame regions of different scales including plant feature information.

[0044] The plurality of preselected frame regions of different scales are processed by using a bilinear interpolation method to obtain a plurality of third plant feature maps of the same scale.

[0045] Feature recognition and classification are performed on the plurality of third plant feature maps of the same scale to obtain a target feature map including plant features.

[0046] According to the plant recognition method provided by the application, the method further comprises:

[0047] The leaf features in the target feature map are extracted to obtain a plurality of first leaf feature maps of different scales.

[0048] Feature fusion is performed on the leaf features included in the leaf feature maps of different scales to obtain a plurality of second leaf feature maps of different scales.

[0049] Statistical classification is performed on the plurality of second leaf feature maps of different scales, to obtain the number of leaves of the plant, and to determine the leaf age of the plant according to the number of leaves.

[0050] On the other hand, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for plant identification as described in any one of the above items is implemented.

[0051] On the other hand, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for plant identification as described in any one of the above items is implemented.

[0052] On the other hand, the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for plant identification as described in any one of the above items is implemented.

[0053] The network model for plant identification provided by the present invention extracts multiple first plant feature maps of different scales, performs feature fusion on the plant features included in the multiple first plant feature maps of different scales to obtain multiple second plant feature maps of different scales, uses a sliding window to scan the second plant feature maps to obtain multiple pre-selected frame areas of different scales including plant feature information, uses an ROI alignment network to process the multiple pre-selected frame areas of different scales using a bilinear interpolation method to obtain multiple third plant feature maps of the same scale; performs feature recognition and classification on the multiple third plant feature maps of the same scale to obtain a target feature map including plant features. After comparative testing, the target feature map of the plant extracted using the network model of the present application is more accurate and has richer feature information, which is helpful for the study of crop phenotypes.

[0054] Furthermore, the present invention adopts a second backbone network to extract the leaf features of the target feature map in progress, and obtains multiple first leaf feature maps of different scales; adopts a neck network to perform feature fusion on the leaf features included in the leaf feature maps of different scales, and obtains multiple second leaf feature maps of different scales; finally, adopts a prediction network to perform statistical classification on the multiple second leaf feature maps of different scales, obtain the number of leaves of the plant, and determine the leaf age and confidence of the plant based on the number of leaves. After comparison with the manual collection method, the accuracy of the plant leaf age information collected by the network model of the present invention is above 80%. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 is one of the network model structure schematic diagrams provided by the present application;

[0057] Figure 2 is the second network model structure schematic diagram provided by the present application;

[0058] Figure 3 is the first main network structure schematic diagram provided by the present application;

[0059] Figure 4 is the Shuffle Transformer Block network structure schematic diagram provided by the present application;

[0060] Figure 5 is the third network model structure schematic diagram provided by the present application;

[0061] Figure 6 is the fourth network model structure schematic diagram provided by the present application;

[0062] Figure 7 is the original image of the plant provided by the present application;

[0063] Figure 8 is the target feature map of the plant provided by the present application;

[0064] Figure 9 is the leaf age annotation map of the plant provided by the present application;

[0065] Figure 10 is the plant recognition device structure schematic diagram provided by the present application;

[0066] Figure 11 is the plant recognition method flow chart provided by the present application;

[0067] Figure 12 is the structure schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0069] The technical scheme of the present application will be described in detail below in combination with Figures 1-12 the drawings.

[0070] Embodiment one:

[0071] The embodiment provides a network model for plant identification, as shown in Figure 1 and Figure 2 The network model can be understood as an improved Mask R-CNN network model, and the network model specifically comprises a first backbone network 10, a feature pyramid network 11, a region generation network 12, an ROI alignment network 13 and a head network 14.

[0072] The first backbone network (Backbone) 10 is used for feature extraction on an input plant image, and a plurality of first plant feature maps of different scales are obtained. The feature pyramid network (FPN) 11 is used for feature fusion on plant features included in the plurality of first plant feature maps of different scales, and a plurality of second plant feature maps of different scales are obtained. The region generation network (RPN) 12 is used for scanning the second plant feature maps by using a sliding window, and a plurality of preselected frame regions of different scales including plant feature information are obtained. The ROI alignment network (ROI Align) 13 is used for processing the plurality of preselected frame regions of different scales by using a bilinear interpolation method, and a plurality of third plant feature maps of the same scale are obtained. The head network (Head) 14 is used for feature recognition and classification on the plurality of third plant feature maps of the same scale, and a target feature map including plant features is obtained.

[0073] Specifically, the first backbone network 10 in the embodiment includes a Shuffle Transformer, and the Shuffle Transformer is used as a feature extractor to extract feature information of an input image (as shown in Figure 7 The FPN obtains high-resolution and strong semantic features by extracting different scale feature maps and fusing high resolution of a shallow layer and rich semantics of a deep layer. The RPN scans a feature image by using a sliding window to find a region where a target exists, and performs fine adjustment on a target position and a size by using non-maximum suppression to generate a final region proposal, and the final region proposal is transmitted to a next stage. The RoI Align uses a bilinear interpolation method to obtain a fixed-size feature map with more complete information according to a position coordinate of a preselected frame, so as to perform subsequent classification and bounding box regression operations. Finally, the Head includes full connection layers (FC layers) and a full convolutional network (FCN), the full connection layers are used to determine a target category of a bounding box, and the FCN classifies each pixel value of an input image to realize accurate segmentation of a target, so that a mask image, that is, the target feature map including plant features, is obtained, as shown in Figure 8 .

[0074] In the embodiment, the first backbone network 10 includes a plurality of feature extraction units of different dimensions, and the plurality of feature extraction units (that is, Shuffle Transformers) are connected in sequence. Figure 3,This embodiment includes four feature extraction units, such as Figure 3 In stage1, stage2, stage3 and stage4, the feature extraction unit includes a token embedding module (TokenEmbeding) and a Shuffle Transformer Block module; the token embedding module is used to adjust the dimension of the input plant image, and the Shuffle Transformer module is used to extract features from the dimension-adjusted plant image to obtain first plant feature maps of different scales. In this embodiment, after the plant image is input into the Shuffle Transformer, the dimension of the input image is adjusted by the token embedding module, and the Shuffle Transformer Block module implements feature learning based on the input stable feature image, and the two modules are combined to generate a hierarchical feature representation. The entire first backbone network 10 is composed of four Shuffle Transformers, and the feature dimension of the image is increased by splicing between each adjacent Shuffle Transformer. Therefore, the first plant feature map generated by the first backbone network 10 has four different feature dimensions.

[0075] Among them, such as Figure 4The Shuffle Transformer Block module of the embodiment includes a first normalization module (Norm) 101, a WMSA module 102, a first near-neighbor window connection module (NWC) 103, a second normalization module 104, a first multi-layer perception module (MLP) 105, a third normalization module 106, a Shuffle WMSA module 107, a second near-neighbor window connection module 108, a third normalization module 109, and a second multi-layer perception module 1010. The first normalization module 101, the WMSA module 102, the first near-neighbor window connection module 103, the second normalization module 104, the first multi-layer perception module 105, the third normalization module 106, the Shuffle WMSA module 107, the second near-neighbor window connection module 108, the third normalization module 109, and the second multi-layer perception module 1010 are sequentially connected, that is, the output end of the previous module is connected to the input end of the next module. At the same time, the input end of the first normalization module 101 is connected to the first near-neighbor window connection module 103 through a residual connection; the output end of the WMSA module 102 is connected to the input end of the second normalization module 104 through a residual connection; the output end of the first near-neighbor window connection module 103 is connected to the input end of the third normalization module 106 through a residual connection; the output end of the first multi-layer perception module 105 is connected to the input end of the second near-neighbor window connection module 108; the output end of the Shuffle WMSA module 107 is connected to the input end of the third normalization module 109; and the output end of the second near-neighbor window connection module 108 is connected to the output end of the second multi-layer perception module 1010. The first normalization module 101, the second normalization module 104, and the third normalization module 106 are used for normalizing the input plant image; the WMSA module 102 is used for dividing the normalized plant image into multiple windows of a fixed size and without overlap, and calculating the self-attention of each window; the Shuffle WMSA module 107 is used for strengthening the target spatial information; the first near-neighbor window connection module 103 and the second near-neighbor window connection module 108 are used for improving the information flow of adjacent windows; and the first multi-layer perception module 105 and the second multi-layer perception module 1010 are used for mapping the extracted multiple features to plant features and background features, respectively. To some extent, the deeper the network structure is, the better the performance is. However, with the increase of the network depth, the problem of gradient vanishing or gradient explosion may occur. Therefore, the Shuffle Transformer Block adopts a residual connection mode. First, after the input image, the Norm normalizes the image to speed up the model convergence. Second, the attention mechanism WMSA module 102 of the conventional window division strategy divides the normalized image into windows of a fixed size and without overlap, and calculates the self-attention in each window.After that, the residual connection is used to fuse the features and reduce the model calculation complexity. The pixel communication between the non-overlapping windows is difficult, which leads to a smaller receptive field and cannot perform dense prediction tasks. In this embodiment, the NWC module is used to improve the information flow of the neighboring window, so that the Shuffle Transformer Block can build rich cross-window connections and enhance the expression ability. After the residual connection, the third normalization module 109 is used for normalization again, and the second multi-layer perception module 1010 is used to map the input multiple features to corn seedling features and background features, respectively, and the first stage is completed. The second stage repeats the above operation, and the WMSA module 102 is replaced by the Shuffle WMSA module 107 in the second stage. The Shuffle WMSA module 107 is a spatial shuffle windowed attention mechanism, which is used to strengthen the target spatial information. Therefore, the Shuffle Transformer Block can build rich cross-window connections and enhance the expression ability.

[0076] Further, as Figure 5 The network model of the embodiment further includes a second backbone network (Backbone) 15, a neck network (Neck) 16, and a prediction network (Prediction) 17. The second backbone network (Backbone) 15, the neck network (Neck) 16, and the prediction network (Prediction) 17 constitute an improved YOLOV5.6 model.

[0077] The second backbone network 15 is used to extract the leaf features in the target feature map to obtain a plurality of first leaf feature maps of different scales. The neck network 16 is used to fuse the leaf features included in the leaf feature maps of different scales to obtain a plurality of second leaf feature maps of different scales. The prediction network 17 is used to statistically classify the plurality of second leaf feature maps of different scales to obtain the number of leaves of the plant, and determine the leaf age of the plant according to the number of leaves. In other embodiments, the prediction network 17 outputs the leaf age, the confidence, and the predicted category (new leaf or old leaf).

[0078] In this embodiment, the second backbone network (Backbone) 14 is composed of a convolution module (Conv), a C3 module (i.e., a feature extraction module), an attention mechanism module (CBAM), and a multi-scale fusion module (SPPF). The second backbone network 14 is used to extract the leaf features of the plant according to the bounding box region after inputting the image. Specifically, as Figure 6The second backbone network 14 of this embodiment includes a first convolution module 141, a second convolution module 142, a third convolution module 144, a fourth convolution module 146, a fifth convolution module 148, an attention module 1410, an SPPF module 1411, a first feature extraction module 143, a second feature extraction module 145, a third feature extraction module 147, and a fourth feature extraction module 149. The input end of the first convolution module 141 is used to receive the target feature map, the output end of the first convolution module 141 is connected to the input end of the second convolution module 142, the output end of the second convolution module 142 is connected to the input end of the first feature extraction module 143, the output end of the first feature extraction module 143 is connected to the input end of the third convolution module 144, the output end of the third convolution module is connected to the input end of the second feature extraction module 145, the output end of the second feature extraction module 145 is connected to the input end of the fourth convolution module 146, the output end of the fourth convolution module 146 is connected to the input end of the third feature extraction module 147, the output end of the third feature extraction module 147 is connected to the input end of the fifth convolution module 148, the output end of the fifth convolution module 148 is connected to the input end of the fourth feature extraction module 149, the output end of the fourth feature extraction module 149 is connected to the input end of the attention module 1410, and the output end of the attention module 1410 is connected to the input end of the SPPF module 1411. The output ends of the first feature extraction module 143 , the second feature extraction module 145 , the third feature extraction module 147 and the SPPF module 1411 are used to output four sets of first leaf feature maps of different scales to the neck network 15 .

[0079] The Neck network of this embodiment is composed of Conv, Upsample, Concat and C3 modules, which are used to better integrate the target features obtained by the first backbone network 14 and improve the detection performance of the network. After the Prediction network obtains the four outputs of the Neck network, it generates the bounding box coordinates, confidence and predicted category of the target. Specifically, Figure 6The neck network of the embodiment includes a first fusion module 150, a second fusion module 154, a third fusion module 157, a fourth fusion module 1502, a fifth fusion module 1505, a sixth fusion module 1510, a fifth feature extraction module 151, a sixth feature extraction module 155, a seventh feature extraction module 158, an eighth feature extraction module 1503, a ninth feature extraction module 1506, a tenth feature extraction module 1511, a sixth convolution module 153, a seventh convolution module 156, an eighth convolution module 1501, a ninth convolution module 1504, a tenth convolution module 1509, an eleventh convolution module 1508, a first upsampling module 152, a second upsampling module 159, and a third upsampling module 1507. An input end of the first fusion module 150 is connected with an output end of the first feature extraction module 143, and an output end of the first fusion module 150 is connected with an input end of the fifth feature extraction module 151. An output end of the fifth feature extraction module 151 is connected with an input end of the prediction network 16. An input end of the second fusion module 154 is connected with an output end of the second feature extraction module 145, and an output end of the second fusion module 154 is connected with an input end of the sixth feature extraction module 155. An output end of the sixth feature extraction module 155 is connected with an input end of the seventh convolution module 156, and an output end of the seventh convolution module 156 is connected with an input end of the third fusion module 157. An output end of the third fusion module 157 is connected with an input end of the seventh feature extraction module 158, and an output end of the seventh feature extraction module 158 is connected with an input end of the prediction network 16. An input end of the first upsampling module 152 is connected with an output end of the seventh convolution module 156, and an output end of the first upsampling module 152 is connected with an input end of the first fusion module 150. An input end of the sixth convolution module 153 is connected with an output end of the fifth feature extraction module 151, and an output end of the sixth convolution module 153 is connected with an input end of the third fusion module 157. An input end of the fourth fusion module 1502 is connected with an output end of the third feature extraction module 147, and an output end of the fourth fusion module 1502 is connected with an input end of the eighth feature extraction module 1503. An output end of the eighth feature extraction module 1503 is connected with an input end of the ninth convolution module 1504, and an output end of the ninth convolution module 1504 is connected with an input end of the fifth fusion module 1505. An output end of the fifth fusion module 1505 is connected with an input end of the ninth feature extraction module 1506, and an output end of the ninth feature extraction module 1506 is connected with an input end of the prediction network 16. An input end of the second upsampling module 159 is connected with an output end of the ninth convolution module 1504, and an output end of the second upsampling module 159 is connected with an input end of the second fusion module 154. An input end of the eighth convolution module 1501 is connected with an output end of the seventh feature extraction module 158, and an output end of the eighth convolution module 1501 is connected with an input end of the fifth fusion module 1505.An input end of the tenth convolution module 1509 is connected with an output end of the SPPF module 1411, an output end of the tenth convolution module 1509 is connected with an input end of the sixth fusion module 1510, an output end of the sixth fusion module 1510 is connected with an input end of the tenth feature extraction module 1511, and an output end of the tenth feature extraction module 1511 is connected with an input end of the prediction network 16. An input end of the third up-sampling module 1507 is connected with an output end of the tenth convolution module 1509, and an output end of the third up-sampling module 1507 is connected with an input end of the fourth fusion module 1502; an input end of the eleventh convolution module 1508 is connected with an output end of the ninth feature extraction module 1506, and an output end of the eleventh convolution module 1508 is connected with an input end of the sixth fusion module 1510.

[0080] The Conv module of the embodiment encapsulates convolution (Conv2d), batch normalization (Batch Normalization) and an activation function (SiLU), the Conv is used to extract target features and perform data dimension reduction; the C3 module contains three convolution layers and a plurality of bottleneck modules to perform residual processing on input feature information, increase gradient values of back propagation between layers, avoid gradient disappearance problems caused by deepening of the network, and thus extract finer-grained features; after input of a feature map, the CBAM module can sequentially generate attention feature map information in the channel and spatial dimensions, and perform adaptive feature correction with the input feature map to enhance target feature values; the SPPF module improves representation ability of the input feature map through fusion of local features and global features; the Upsample performs up-sampling on input feature information to expand the feature map; the Concat performs feature fusion on the input feature map to expand the dimension of the feature map; the Prediction adds a detection layer based on the original model, and outputs a feature of 160*160*21, and the larger feature map is more conducive to detection of small targets.

[0081] This embodiment takes corn seedlings in a certain area as an example to illustrate the effect of the network model of the application. In this embodiment, DJI Phantom 4 Pro V2.0 (Shenzhen, China) is used to obtain images of corn seedlings. Based on the corn breeding unmanned aerial vehicle digital image data obtained in Xinxiang, Henan in 2020 and 2021, the leaf age of corn seedlings is identified to verify the effectiveness of the method proposed in this embodiment. First, the high-resolution image obtained by the unmanned aerial vehicle is cropped to an image of 640x640 pixels, and the LabelMe software is used to label the complete corn seedling boundary and category information in the image to obtain a *.json format file. The json format file and the original image directly generate a corn seedling dataset, which is input into the improved Mask R-CNN network model provided in the above embodiment for model training and testing. Finally, the corn seedlings in the image are identified to obtain the target feature map of the single corn seedling characteristics. Then, the LabelImg software is used to label the leaf of the target feature map of the single corn seedling characteristics obtained after target recognition and detection, including the position and category information of the leaf, and process it into an xml format file required for leaf age recognition to obtain a leaf age dataset. The leaf age dataset is input into the improved YOLOV5.6 for model training and verification, and finally the corn leaf age information extraction is realized.

[0082] In this embodiment, a total of 1005 pictures are labeled, and the division ratio of the dataset is training set: validation set: test set = 7:2:1. In order to verify the effectiveness of the network model of this embodiment, the main indicators selected for evaluating the first stage improved Mask R-CNN model are: precision (Precision), recall (Recall) and class average pixel accuracy (Mean Pixel Accuracy), and the main indicators selected for evaluating the second stage improved YOLOV5.6 training model are: precision (Precision), recall (Recall), and average precision mAP (Mean Average Precision), the formulas are as follows:

[0083]

[0084]

[0085]

[0086]

[0087]

[0088] In the above formula, TP represents true positive, that is, the number of correctly predicted corn seedling pixels or the number of correctly predicted corn leaf types; FP represents false positive, that is, the number of pixels predicted as corn seedlings or the number of incorrect leaf type identifications; FN represents false negative, that is, the number of pixels of corn seedlings that are not segmented and the number of undetected leaves; N represents the number of categories of corn leaves; AP i represents the average precision value of the i th leaf type.

[0089] Finally, the average precision, average recall and average pixel accuracy of the Mask R-CNN model during verification are 89.57%, 74.78% and 94.43%, respectively, which basically achieves the goal of seedling segmentation. The improved YOLOV5.6 model is used for leaf age identification, and the detection results of corn leaf age include three cases: new leaves, fully expanded leaves and all leaves, as shown in Table 1. The precision of all leaves is more than 80%, and the precision of expanded leaves is more than 90%. Figure 9 The effect diagram of corn seedling and leaf age information identification is shown in Figure 9 where leaf represents old leaves (fully expanded leaves), and new represents new leaves. From Figure 9 It can be seen that the corn seedling identification is accurate, and the leaf age information extraction is also relatively accurate. It can be seen that the network model proposed in this embodiment has high precision in seedling identification and leaf age extraction based on unmanned aerial vehicle images.

[0090] Table 1: Corn leaf age detection results

[0091]

[0092] Example Two:

[0093] The embodiment provides a device for plant identification, such as Figure 10 The device comprises a first processing module 301, a second processing module 302, a third processing module 303, a fourth processing module 304 and a fifth processing module 305.

[0094] The first processing module 301 is configured to perform feature extraction on the input plant image to obtain a plurality of first plant feature maps of different scales. The second processing module 302 is configured to perform feature fusion on plant features included in the plurality of first plant feature maps of different scales to obtain a plurality of second plant feature maps of different scales. The third processing module 303 is configured to scan the second plant feature maps by using a sliding window to obtain a plurality of preselected frame regions of different scales including plant feature information. The fourth processing module 304 is configured to perform processing on the plurality of preselected frame regions of different scales by using a bilinear interpolation method to obtain a plurality of third plant feature maps of the same scale. The fifth processing module 305 is configured to perform feature recognition and classification on the plurality of third plant feature maps of the same scale to obtain a target feature map including plant features.

[0095] In an embodiment, the plant recognition device further includes a sixth processing module 306, a seventh processing module 307, and a prediction module 308. The sixth processing module 306 is configured to extract leaf features in the target feature map to obtain a plurality of first leaf feature maps of different scales. The seventh processing module 307 is configured to perform feature fusion on leaf features included in the plurality of first leaf feature maps of different scales to obtain a plurality of second leaf feature maps of different scales. The prediction module 308 is configured to perform statistical classification on the plurality of second leaf feature maps of different scales to obtain the number of leaves of the plant and determine the leaf age of the plant according to the number of leaves.

[0096] The functions of the modules of the embodiment and the functions of the network in the above-described embodiment one are corresponding.

[0097] Embodiment three:

[0098] The embodiment provides a method for plant recognition. Figure 11 The method includes:

[0099] Step 401: performing feature extraction on the input plant image to obtain a plurality of first plant feature maps of different scales.

[0100] Step 402: performing feature fusion on plant features included in the plurality of first plant feature maps of different scales to obtain a plurality of second plant feature maps of different scales.

[0101] Step 403: Scan the second plant feature map using a sliding window to obtain a plurality of pre-selected frame areas of different scales including plant feature information.

[0102] Step 404: a plurality of pre-selected frame regions of different scales are processed using a bilinear interpolation method to obtain a plurality of third plant feature maps of the same scale.

[0103] Step 405: performing feature recognition and classification on a plurality of third plant feature maps of the same scale to obtain a target feature map including plant features.

[0104] The above method of this embodiment can accurately obtain the target characteristic map of the plant, so as to accurately obtain the characteristic information of the plant.

[0105] Furthermore, in another embodiment, the method for plant identification further includes:

[0106] Step 406: extracting leaf features from the target feature map to obtain a plurality of first leaf feature maps of different scales.

[0107] Step 407: performing feature fusion on the leaf features included in the leaf feature maps of different scales to obtain a plurality of second leaf feature maps of different scales;

[0108] Step 408: Statistically classify the second leaf feature images of multiple different scales to obtain the number of leaves of the plant, and determine the leaf age of the plant based on the number of leaves.

[0109] The method of this embodiment can accurately obtain the type, leaf age and confidence information of the plant.

[0110] Example 4:

[0111] Figure 12 An example of a physical structure diagram of an electronic device is shown below. Figure 12 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the method for plant identification provided in the above embodiment.

[0112] In addition, the logic instructions in the memory 830 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0113] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the method for plant identification provided by the above-mentioned methods.

[0114] In another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method for plant identification provided by the above-mentioned methods.

[0115] The device embodiments described above are only schematic, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0116] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary universal hardware platforms, and of course, can also be realized by hardware. Based on such understanding, the above technical solutions essentially or the parts that contribute to the prior art can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0117] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A network model for plant identification, characterized in that, The method comprises the following steps: a first backbone network, a feature pyramid network, a region generation network, an ROI alignment network and a head network are included; The first backbone network is used for feature extraction of an input plant image to obtain a plurality of first plant feature maps of different scales; The feature pyramid network is used for feature fusion of plant features included in the plurality of first plant feature maps of different scales to obtain a plurality of second plant feature maps of different scales; The region generation network is used for scanning the second plant feature maps by using a sliding window to obtain a plurality of preselected box regions of different scales including plant feature information; The ROI alignment network is used for processing the plurality of preselected box regions of different scales by using a bilinear interpolation method to obtain a plurality of third plant feature maps of the same scale; The head network is used for feature recognition and classification of the plurality of third plant feature maps of the same scale to obtain a target feature map including plant features; The first backbone network comprises a plurality of feature extraction units of different dimensions, and the plurality of feature extraction units are connected in sequence; The feature extraction unit comprises a label embedding module and a Shuffle Transformer Block module; the label embedding module is used for adjusting the dimension of the input plant image, and the Shuffle Transformer Block module is used for feature extraction of the plant image with adjusted dimension to obtain first plant feature maps of different scales.

2. The network model for plant recognition according to claim 1, wherein, Further comprising: a second backbone network, a neck network and a prediction network; The second backbone network is used for extracting leaf features in the target feature map to obtain a plurality of first leaf feature maps of different scales; The neck network is used for feature fusion of leaf features included in the plurality of leaf feature maps of different scales to obtain a plurality of second leaf feature maps of different scales; The prediction network is used for statistical classification of the plurality of second leaf feature maps of different scales to obtain the number of leaves of the plant, and determine the leaf age of the plant according to the number of leaves.

3. The network model for plant recognition according to claim 1, wherein, The Shuffle Transformer Block module comprises a first standardization module, a WMSA module, a first near-neighbor window connection module, a second standardization module, a first multilayer perception module, a third standardization module, a Shuffle WMSA module, a second near-neighbor window connection module, a third standardization module and a second multilayer perception module, which are connected in sequence. The input end of the first standardization module is connected with the first near-neighbor window connection module in a residual manner; the output end of the WMSA module is connected with the input end of the second standardization module in a residual manner; the output end of the first near-neighbor window connection module is connected with the input end of the third standardization module in a residual manner; the output end of the first multi-layer perception module is connected with the input end of the second near-neighbor window connection module; the output end of the Shuffle WMSA module is connected with the input end of the third standardization module; and the output end of the second near-neighbor window connection module is connected with the output end of the second multi-layer perception module. The first standardization module, the second standardization module and the third standardization module are used for normalizing the input plant image; the WMSA module is used for dividing the normalized plant image into a plurality of windows of fixed size and without overlap, and calculating the self-attention of each window; the Shuffle WMSA module is used for strengthening the target spatial information; the first near-neighbor window connection module and the second near-neighbor window connection module are used for improving the information flow of adjacent windows; and the first multi-layer perception module and the second multi-layer perception module are used for respectively mapping the extracted plurality of features as plant features and background features.

4. The network model for plant recognition according to claim 2, wherein, The second backbone network comprises a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a fifth convolution module, an attention module, an SPPF module, a first feature extraction module, a second feature extraction module, a third feature extraction module and a fourth feature extraction module. The input end of the first convolution module is used for receiving the target feature map, the output end of the first convolution module is connected with the input end of the second convolution module, the output end of the second convolution module is connected with the input end of the first feature extraction module, the output end of the first feature extraction module is connected with the input end of the third convolution module, the output end of the third convolution module is connected with the input end of the second feature extraction module, the output end of the second feature extraction module is connected with the input end of the fourth convolution module, the output end of the fourth convolution module is connected with the input end of the third feature extraction module, the output end of the third feature extraction module is connected with the input end of the fifth convolution module, the output end of the fifth convolution module is connected with the input end of the fourth feature extraction module, the output end of the fourth feature extraction module is connected with the input end of the attention module, and the output end of the attention module is connected with the input end of the SPPF module. The output ends of the first feature extraction module, the second feature extraction module, the third feature extraction module and the SPPF module are used for outputting four groups of first leaf feature maps of different scales.

5. The network model for plant recognition according to claim 4, wherein, The neck network comprises a first fusion module, a second fusion module, a third fusion module, a fourth fusion module, a fifth fusion module, a sixth fusion module, a fifth feature extraction module, a sixth feature extraction module, a seventh feature extraction module, an eighth feature extraction module, a ninth feature extraction module, a tenth feature extraction module, a sixth convolution module, a seventh convolution module, an eighth convolution module, a ninth convolution module, a tenth convolution module, an eleventh convolution module, a first up-sampling module, a second up-sampling module and a third up-sampling module. The input end of the first fusion module is connected with the output end of the first feature extraction module, the output end of the first fusion module is connected with the input end of the fifth feature extraction module, and the output end of the fifth feature extraction module is connected with the input end of the prediction network. The input end of the second fusion module is connected with the output end of the second feature extraction module, the output end of the second fusion module is connected with the input end of the sixth feature extraction module, the output end of the sixth feature extraction module is connected with the input end of the seventh convolution module, the output end of the seventh convolution module is connected with the input end of the third fusion module, the output end of the third fusion module is connected with the input end of the seventh feature extraction module, and the output end of the seventh feature extraction module is connected with the input end of the prediction network. The input end of the first up-sampling module is connected with the output end of the seventh convolution module, and the output end of the first up-sampling module is connected with the input end of the first fusion module; the input end of the sixth convolution module is connected with the output end of the fifth feature extraction module, and the output end of the sixth convolution module is connected with the input end of the third fusion module. The input end of the fourth fusion module is connected with the output end of the third feature extraction module, the output end of the fourth fusion module is connected with the input end of the eighth feature extraction module, the output end of the eighth feature extraction module is connected with the input end of the ninth convolution module, the output end of the ninth convolution module is connected with the input end of the fifth fusion module, the output end of the fifth fusion module is connected with the input end of the ninth feature extraction module, the output end of the ninth feature extraction module is connected with the input end of the prediction network, the input end of the second up-sampling module is connected with the output end of the ninth convolution module, and the output end of the second up-sampling module is connected with the input end of the second fusion module; the input end of the eighth convolution module is connected with the output end of the seventh feature extraction module, and the output end of the eighth convolution module is connected with the input end of the fifth fusion module. The input end of the tenth convolution module is connected with the output end of the SPPF module, the output end of the tenth convolution module is connected with the input end of the sixth fusion module, the output end of the sixth fusion module is connected with the input end of the tenth feature extraction module, and the output end of the tenth feature extraction module is connected with the input end of the prediction network. An input end of the third up-sampling module is connected with an output end of the tenth convolutional module, and an output end of the third up-sampling module is connected with an input end of the fourth fusion module; an input end of the eleventh convolutional module is connected with an output end of the ninth feature extraction module, and an output end of the eleventh convolutional module is connected with an input end of the sixth fusion module.

6. An apparatus for plant identification, comprising: Comprise: A first processing module is configured to perform feature extraction on an input plant image to obtain a plurality of first plant feature maps of different scales; A second processing module is configured to perform feature fusion on plant features included in the plurality of first plant feature maps of different scales to obtain a plurality of second plant feature maps of different scales; A third processing module is configured to scan the second plant feature maps using a sliding window to obtain a plurality of preselected frame regions of different scales including plant feature information; A fourth processing module is configured to perform processing on the plurality of preselected frame regions of different scales using a bilinear interpolation method to obtain a plurality of third plant feature maps of the same scale; A fifth processing module is configured to perform feature recognition and classification on the plurality of third plant feature maps of the same scale to obtain a target feature map including plant features; The first processing module comprises a plurality of feature extraction units of different dimensions, and the plurality of feature extraction units are connected in sequence; The feature extraction unit comprises a token embedding module and a Shuffle Transformer Block module; the token embedding module is configured to adjust the dimension of the input plant image, and the Shuffle Transformer Block module is configured to perform feature extraction on the plant image of the adjusted dimension to obtain first plant feature maps of different scales.

7. The device for plant recognition according to claim 6, characterized in that, Further comprise: A sixth processing module is configured to extract leaf features in the target feature map to obtain a plurality of first leaf feature maps of different scales; A seventh processing module is configured to perform feature fusion on leaf features included in the plurality of leaf feature maps of different scales to obtain a plurality of second leaf feature maps of different scales; A prediction module is configured to perform statistical classification on the plurality of second leaf feature maps of different scales to obtain the number of leaves of a plant, and determine the leaf age of the plant according to the number of leaves.

8. A method for plant identification, characterized in that, Comprise: A first backbone network is configured to perform feature extraction on an input plant image to obtain a plurality of first plant feature maps of different scales, the first backbone network comprises a plurality of feature extraction units of different dimensions, and the plurality of feature extraction units are connected in sequence; the feature extraction unit comprises a token embedding module and a Shuffle Transformer Block module; the token embedding module is configured to adjust the dimension of the input plant image, and the Shuffle Transformer Block module is configured to perform feature extraction on the plant image of the adjusted dimension to obtain first plant feature maps of different scales; Perform feature fusion on plant features included in the plurality of first plant feature maps of different scales to obtain a plurality of second plant feature maps of different scales; The second plant feature map is scanned by using a sliding window to obtain a plurality of preselected frame regions of different scales including plant feature information; The plurality of preselected frame regions of different scales are processed by using a bilinear interpolation method to obtain a plurality of third plant feature maps of the same scale; Feature recognition and classification are performed on the plurality of third plant feature maps of the same scale to obtain a target feature map including plant features.

9. The method for plant recognition according to claim 8, characterized in that, Further comprising: Leaf features in the target feature map are extracted to obtain a plurality of first leaf feature maps of different scales; Feature fusion is performed on leaf features included in the plurality of leaf feature maps of different scales to obtain a plurality of second leaf feature maps of different scales; The number of leaves of the plant is obtained by statistical classification of the plurality of second leaf feature maps of different scales, and the leaf age of the plant is determined according to the number of leaves.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor executes the program to implement the method for plant identification according to any one of claims 8 or 9.

11. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method for plant identification according to any one of claims 8 or 9.

12. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method for plant identification according to any one of claims 8 or 9.

Images