Crop Growth and Development Monitoring Method, Device, Equipment and Medium

Abstract

The present invention provides a method, device, equipment and medium for monitoring the growth and development of crops, relating to the technical field of image recognition, including: obtaining an image sequence and meteorological data corresponding to a target crop; inputting the image sequence into a pre-trained crop leaf segmentation model to output a leaf segmentation result corresponding to each crop image data through the crop leaf segmentation model; adjusting the number of leaves based on the meteorological data to obtain the target number of leaves at different time points, so as to realize the dynamic counting of the leaves of the target crop; and tracking and matching the leaves of each crop image data to determine the leaf identifier associated with the leaves displayed in each crop image data, and combining the leaf mask and bounding box information to determine the target phenotypic parameters corresponding to each leaf identifier at different time points, so as to monitor the growth and development of the target crop. The present invention can significantly improve the accuracy of crop growth and development monitoring.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular, to a method, device, equipment and medium for monitoring the growth and development of crops. Background Art

[0002] With the rapid development of modern agricultural technology, the precise monitoring of the growth status of crops has become increasingly important. Traditional crop monitoring methods often rely on manual observation and recording, which is not only inefficient but also prone to errors, and cannot meet the requirements of modern agricultural production for precision and efficiency. Especially in greenhouse environments, crops grow relatively fast, and more efficient and accurate monitoring methods are needed to track their growth status. In recent years, deep learning technology has made remarkable progress in the field of computer vision, providing new possibilities for automated and precise crop monitoring. Through deep learning models, automatic detection and counting of crop leaves can be achieved, and further segmentation of plant organs can be carried out to calculate the leaf length, leaf width, and leaf area of individual leaves, thus greatly improving the efficiency and accuracy of monitoring. However, existing deep learning-based crop monitoring methods often rely only on image data, which to a certain extent limits the precision of monitoring. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a method, device, equipment and medium for monitoring the growth and development of crops, which can significantly improve the precision of monitoring the growth and development of crops.

[0004] In a first aspect, an embodiment of the present invention provides a method for monitoring the growth and development of crops, including:

[0005] Obtaining an image sequence corresponding to a target crop and meteorological data; wherein, the image sequence includes crop image data at different time points;

[0006] Inputting the image sequence into a pre-trained crop leaf segmentation model to output a leaf segmentation result corresponding to each crop image data through the crop leaf segmentation model; wherein, the leaf segmentation result is used to represent the number of leaves, leaf masks and bounding box information at different time points;

[0007] Adjusting the number of leaves based on the meteorological data to obtain the target number of leaves at different time points, so as to realize the dynamic counting of the leaves of the target crop; and, tracking and matching the leaves for each crop image data to determine the leaf identifier associated with the leaves shown in each crop image data, and combining the leaf mask and bounding box information to determine the target phenotypic parameters corresponding to each leaf identifier at different time points;

[0008] Using the target number of leaves at different time points and the target phenotypic parameters corresponding to each leaf identifier to monitor the growth and development of the target crop.

[0009] In one embodiment, the crop leaf segmentation model adopts an improved YOLOv8 model. The improved YOLOv8 model includes a backbone network, an improved neck network, and a head network. The improved neck network is obtained by replacing the concat layer in the traditional neck network with a BiFPN feature pyramid network; the leaf segmentation results corresponding to each crop image data are output through the crop leaf segmentation model, including:

[0010] Through the backbone network, feature extraction is performed on each crop image data in the image sequence to obtain multi-scale feature maps corresponding to each crop image data;

[0011] Through the improved neck network, instance segmentation is performed on the multi-scale feature maps to obtain instance segmentation feature maps corresponding to each crop image data;

[0012] Through the head network, leaf segmentation results corresponding to each crop image data are generated based on the instance segmentation feature maps.

[0013] In one embodiment, the improved neck network includes a BiFPN feature pyramid network and a multi-scale information aggregation layer; wherein,

[0014] The BiFPN feature pyramid network is used to connect the first feature map output by the previous layer corresponding to the BiFPN feature pyramid network layer and the second feature map in the multi-scale feature maps, and the scales of the first feature map and the second feature map are the same;

[0015] The multi-scale information aggregation layer is used to aggregate the context information of different scales and levels in the third feature map output by the previous layer corresponding to the multi-scale information aggregation layer.

[0016] In one embodiment, the improved neck network specifically includes a first upsampling layer, a first BiFPN feature pyramid network, a first C2f layer, a second upsampling layer, a second BiFPN feature pyramid network, a second C2f layer, a first standard convolution Conv layer, a third BiFPN feature pyramid network, a third C2f layer, a first multi-head multi-scale information aggregation layer, a second standard convolution Conv layer, a connection layer, a fourth C2f layer, and a second multi-scale information aggregation layer. The input ends of the first BiFPN feature pyramid network and the second BiFPN feature pyramid network are also connected to the backbone network, and the input end of the third BiFPN feature pyramid network is also connected to the output end of the first C2f layer.

[0017] In one embodiment, adjusting the leaf number based on meteorological data to obtain the target leaf number at different time points includes:

[0018] Determining the target effective accumulated temperature corresponding to the target crop based on the meteorological data;

[0019] According to the target effective accumulated temperature and the pre - constructed crop leaf accumulated temperature fitting curve, adjust the number of leaves to obtain the target number of leaves at different time points; wherein, the crop leaf accumulated temperature fitting curve is used to describe the mapping relationship between the effective accumulated temperature and the number of leaves.

[0020] In one implementation, the leaf mask and bounding box information are the leaf pixel mask and the leaf bounding box; perform leaf tracking and matching on each crop image data to determine the leaf identifier associated with the leaves shown in each crop image data, including:

[0021] Take the crop image data at the current time point as the query image data, and take the crop image data at all time points before the current time point as the target image data;

[0022] Determine the intersection - over - union ratio between the leaf pixel mask corresponding to the query image data and the leaf pixel mask corresponding to the target image data, so as to judge whether there is target image data matching the query image data based on the intersection - over - union ratio;

[0023] If so, take the leaf identifier associated with the matching target image data as the leaf identifier associated with the query image data;

[0024] If not, take the newly generated unique identifier as the leaf identifier associated with the query image data.

[0025] In one implementation, combine the leaf mask and bounding box information to determine the target phenotypic parameters corresponding to each leaf identifier at different time points, including:

[0026] According to the leaf pixel mask, determine the leaf area corresponding to each leaf identifier at different time points; and, according to the leaf bounding box, determine the leaf length and leaf width corresponding to each leaf identifier at different time points;

[0027] Wherein, the target phenotypic parameters include leaf area, leaf length and leaf width.

[0028] In a second aspect, an embodiment of the present invention further provides a crop growth and development monitoring device, including:

[0029] A data acquisition module, configured to acquire an image sequence and meteorological data corresponding to a target crop; wherein, the image sequence includes crop image data at different time points;

[0030] A leaf segmentation module, configured to input the image sequence into a pre - trained crop leaf segmentation model, so as to output the leaf segmentation result corresponding to each crop image data through the crop leaf segmentation model; wherein, the leaf segmentation result is used to characterize the number of leaves, leaf mask and bounding box information at different time points;

[0031] A quantity and phenotype determination module, configured to adjust the number of leaves based on meteorological data to obtain the target number of leaves at different time points, so as to achieve dynamic counting of the leaves of the target crop; and, perform leaf tracking and matching on each crop image data to determine the leaf identifier associated with the leaves shown in each crop image data, and combine the leaf mask and bounding box information to determine the target phenotype parameters corresponding to each leaf identifier at different time points;

[0032] A monitoring module, configured to monitor the growth and development of the target crop by using the target number of leaves at different time points and the target phenotype parameters corresponding to each leaf identifier.

[0033] In a third aspect, an embodiment of the present invention further provides an electronic device, including a processor and a memory, where the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of the first aspect.

[0034] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores computer executable instructions, and when the computer executable instructions are called and executed by a processor, the computer executable instructions cause the processor to implement the method according to any one of the first aspect.

[0035] The crop growth and development monitoring method, device, equipment and medium provided by the embodiments of the present invention first obtain an image sequence and meteorological data corresponding to the target crop, where the image sequence includes crop image data at different time points; then input the image sequence into a pre-trained crop leaf segmentation model to output the leaf segmentation result corresponding to each crop image data through the crop leaf segmentation model; wherein, the leaf segmentation result is used to characterize the number of leaves, leaf mask and bounding box information at different time points; then adjust the number of leaves based on the meteorological data to obtain the target number of leaves at different time points, so as to achieve dynamic counting of the leaves of the target crop; at the same time, perform leaf tracking and matching on each crop image data to determine the leaf identifier associated with the leaves shown in each crop image data, and combine the leaf mask and bounding box information to determine the target phenotype parameters corresponding to each leaf identifier at different time points; finally, monitor the growth and development of the target crop by using the target number of leaves at different time points and the target phenotype parameters corresponding to each leaf identifier. The above method combines image sequence and meteorological data through deep learning technology, realizes dynamic counting of the leaves of the target crop, and brings significant effects to the growth and development monitoring of the target crop; at the same time, by tracking and matching the leaves, and combining deep learning technology to dynamically monitor the target phenotype parameters corresponding to each leaf identifier at different time points, the accuracy of crop growth and development monitoring is comprehensively improved.

[0036] Other features and advantages of the present invention will be set forth in the following description, and in part will be obvious from the description, or may be learned by practice of the present invention. The objectives and other advantages of the present invention are realized and attained by the structure particularly pointed out in the specification, claims as well as the drawings.

[0037] To make the above objectives, features and advantages of the present invention more obvious and understandable, the following specific preferred embodiments are given, and in conjunction with the accompanying drawings, the detailed description is as follows. Brief Description of the Drawings

[0038] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0039] Figure 1 It is a schematic flow chart of a method for monitoring crop growth and development provided by an embodiment of the present invention;

[0040] Figure 2 It is an overall flow chart of a method for monitoring crop growth and development provided by an embodiment of the present invention;

[0041] Figure 3 It is an overall framework diagram of an improved YOLOv8 crop leaf segmentation model provided by an embodiment of the present invention;

[0042] Figure 4 It is a schematic diagram of the training results of an improved YOLOv8 crop leaf segmentation model provided by an embodiment of the present invention;

[0043] Figure 5 It is a schematic diagram of the dynamic counting effect of plant leaves provided by an embodiment of the present invention;

[0044] Figure 6 It is a working flow chart of a leaf tracking and matching algorithm for an improved LeTra model provided by an embodiment of the present invention;

[0045] Figure 7 It is a schematic diagram of the effect of a leaf tracking and matching algorithm for an improved LeTra model provided by an embodiment of the present invention;

[0046] Figure 8 It is a schematic diagram of the effect of extracting crop phenotypic parameters provided by an embodiment of the present invention;

[0047] Figure 9 It is a schematic structural diagram of a device for monitoring crop growth and development provided by an embodiment of the present invention;

[0048] Figure 10 The structural schematic diagram of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0049] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0050] Currently, when implementing crop leaf counting and calculating leaf phenotype data at the organ scale based on a deep learning model of static images, there is often a problem of leaf occlusion, making the leaf counting and phenotype data extraction inaccurate. At the same time, the impact of meteorological data on crop growth is ignored, and it is difficult to meet the accuracy requirements for continuous observation of the crop growth process. Meteorological conditions such as temperature, humidity, and light have an important impact on crop growth. Combining these data with image data can further improve the accuracy of crop leaf counting. Based on this, the embodiments of the present invention provide a crop growth and development monitoring method, device, equipment and medium, which can significantly improve the accuracy of crop growth and development monitoring.

[0051] To facilitate the understanding of this embodiment, first, a crop growth and development monitoring method disclosed in the embodiments of the present invention will be introduced in detail. Refer to Figure 1 The flowchart of a crop growth and development monitoring method shown in the figure. The method mainly includes the following steps S102 to step S108:

[0052] Step S102, obtaining an image sequence and meteorological data corresponding to a target crop.

[0053] Among them, the image sequence includes crop image data at different time points. The crop image data is the image of the target crop from the seedling stage to the fruiting stage captured by a camera installed on the greenhouse ceiling or the field support rod. The meteorological data is the daily average temperature data of the target crop from the seedling stage to the fruiting stage.

[0054] Step S104, inputting the image sequence into a pre-trained crop leaf segmentation model to output the leaf segmentation result corresponding to each crop image data through the crop leaf segmentation model.

[0055] Among them, the leaf segmentation results are used to characterize the number of leaves, leaf masks, and bounding box information at different time points. The leaf mask and bounding box information are the leaf pixel masks and leaf bounding boxes. The crop leaf segmentation model uses an improved YOLOv8 model. Specifically, in the existing YOLOv8 model, the Neck network is replaced with a BiFPN feature pyramid network, and the head network in the YOLOv8 model is improved using a multi-scale information aggregation (ContextAggregation) attention mechanism.

[0056] In one example, the crop image data in the image sequence can be annotated using the open-source application LabelMe. The annotated dataset is divided into a training set and a test set according to a ratio. The training set is used to train the crop leaf segmentation model, and the trained crop leaf segmentation model is used to perform leaf segmentation on the test set to obtain the leaf segmentation results corresponding to each crop image data in the test set.

[0057] Step S106: Adjust the number of leaves based on meteorological data to obtain the target number of leaves at different time points, so as to achieve dynamic counting of the leaves of the target crop; and track and match the leaves in each crop image data to determine the leaf identifier associated with the leaves shown in each crop image data, and combine the leaf mask and bounding box information to determine the target phenotypic parameters corresponding to each leaf identifier at different time points.

[0058] Among them, the leaf identifier is a unique code associated with each leaf, and the target phenotypic data is phenotypic information including leaf length, leaf width, leaf area, etc. of the crop plant under the camera.

[0059] In one example, the phenotypic data corresponding to each crop image data in the training set can be obtained. Combining the meteorological data and the phenotypic data, by calculating the daily effective accumulated temperature (abbreviated as effective accumulated temperature) of the target crop, and further analyzing the fitting relationship between the effective accumulated temperature and the daily number of crop leaves, the fitting relationship curve formula between the two can be obtained, denoted as the crop leaf accumulated temperature fitting curve; using the crop leaf accumulated temperature fitting curve and the temperature data corresponding to each crop image data in the test set, the aforementioned number of leaves is adjusted to obtain the target number of leaves at different time points.

[0060] In one example, the embodiment of the present invention optimizes the leaf tracking and matching algorithm in the traditional LeTra model. The embodiment of the present invention abandons the direct pairing method of consecutive two crop image data in the traditional LeTra model, and instead realizes the comprehensive comparison and pairing of newly emerging crop image data with all previously recorded crop image data. For leaves that fail to be directly successfully paired but show a high intersection over union, the algorithm will intelligently assign a new unique number to them to ensure the continuity and accuracy of leaf tracking, so as to determine the leaf identifier associated with the leaves shown in each crop image data. Combining the foregoing leaf mask and bounding box information, the target phenotypic parameters corresponding to each leaf identifier at different time points can be determined.

[0061] Step S108, using the number of target leaves at different time points and the target phenotypic parameters corresponding to each leaf identifier, monitor the growth and development of the target crop.

[0062] The crop growth and development monitoring method provided by the embodiment of the present invention realizes the dynamic counting of the leaves of the target crop through deep learning technology combined with image sequences and meteorological data, bringing significant effects to the growth and development monitoring of the target crop; at the same time, by tracking and matching the leaves, combined with deep learning technology to dynamically monitor the target phenotypic parameters corresponding to each leaf identifier at different time points, comprehensively improving the accuracy of crop growth and development monitoring.

[0063] The embodiment of the present invention proposes a method for dynamically counting crop leaves by combining deep learning, image time series data and meteorological data, and then automatically calculates leaf length, leaf width, leaf area, etc. based on the segmented images, aiming to achieve more efficient and accurate crop growth and development monitoring.

[0064] For ease of understanding, the embodiment of the present invention provides an overall flowchart of a crop growth and development monitoring method as shown in Figure 2 including: obtaining an image sequence and meteorological data corresponding to the target crop to form a crop data set; annotating the image sequence of the crop data set and dividing it into a training set and a test set according to a ratio; improving the YOLOv8 model by introducing a BiFPN feature pyramid network and a ContextAggregation attention mechanism; realizing the dynamic counting of crop leaves and the extraction of phenotypic parameters by combining the improved YOLOv8 model with the crop leaf accumulated temperature fitting curve.

[0065] In specific implementation, an embodiment of the present invention provides a specific implementation manner of the foregoing step S102, including: initial data collection, which includes acquiring an image sequence of the crop, phenotypic data, and meteorological data. The image sequence of the crop is a sequence composed of images of the crop plant (i.e., the target crop) from the seedling stage to the fruiting stage, which are obtained by a camera installed on the greenhouse ceiling or the field support pole. The phenotypic data is the phenotypic information of the crop plant under the camera, including leaf length, leaf width, leaf area, etc. The meteorological data is the daily average temperature data of the crop plant from the seedling stage to the fruiting stage.

[0066] On this basis, the images in the crop dataset are labeled through the open-source application LabelMe, and the labeled dataset is divided into a training set and a test set according to a ratio. Among them, the labeled content includes the precise contour bounding box of each leaf shown in the image.

[0067] Before explaining the foregoing step S104, an embodiment of the present invention provides a specific implementation manner of a crop leaf segmentation model. The crop leaf segmentation model uses an improved YOLOv8 model. The improved YOLOv8 model includes a backbone network, an improved neck network, and a head network. The improved neck network is obtained by replacing the concat layer in the traditional neck network with a BiFPN feature pyramid network; the improved neck network includes a BiFPN feature pyramid network (i.e., the Concat_BiFPN layer) and a multi-scale information aggregation layer.

[0068] Among them, the Concat_BiFPN layer is used to connect the first feature map output by the previous layer corresponding to the Concat_BiFPN layer and the second feature map in the multi-scale feature map, and the scales of the first feature map and the second feature map are the same. The Concat_BiFPN layer uses the BiFPN network structure to perform weighted fusion using a fast normalization method, and the calculation formula is as follows:

[0069]

[0070] It is stated that: I i is the input crop leaf feature, O is the output crop leaf feature, W i and W j are learnable weights, ε is a minimum learning rate used to constrain numerical oscillation, and when ε = 0.0001, stable output can be guaranteed.

[0071] Through the weighted cross-scale feature fusion mechanism, the network model can pay appropriate attention to the features of crop leaves of different sizes at different levels. For crop leaves of smaller sizes, BiFPN can retain and fuse the fine-grained features of the lower levels and combine them with the semantic features of higher levels, thereby improving the detection accuracy of crop leaves of smaller sizes. For crop leaves of larger sizes, BiFPN can, through an adaptive feature fusion mechanism, perform appropriate weighted fusion of the features from each level, making the network pay more attention to the level where the crop leaves of larger sizes are located, thereby improving the detection ability of crop leaves of larger sizes. Through bidirectional feature fusion and adaptive weight adjustment, BiFPN can well capture the features of crop leaves of different sizes, thereby improving the model detection accuracy.

[0072] Among them, the multi-scale information aggregation layer (i.e., the ContextAggregation attention mechanism) is used to aggregate the context information of different scales and levels in the third feature map output by the layer corresponding to the multi-scale information aggregation layer. The calculation formula of the ContextAggregation attention mechanism is as follows:

[0073]

[0074] As described, P i and Q i represent the input and output crop leaf feature maps of the i-th layer in the feature pyramid. Each crop leaf feature map contains N i pixels; j, m ∈ {1, N i}, representing the index of each pixel, and represent the feature vectors with index j on the input and output crop leaf feature maps of the i-th layer in the feature pyramid, is the scalar attention weight, w k and w v are linear transformation matrices used to project the feature maps. This module uses 1×1 convolution to perform the mapping.

[0075] Through the ContextAggregation attention mechanism, it is possible to learn to aggregate features from the entire crop leaf feature map and combine them into each pixel using adaptive weights, enhancing the feature extraction ability and expanding the receptive field. It can effectively fuse local and global features while reducing information confusion, improving the recognition effect of the Yolov8 network on crop leaf targets of smaller sizes and overlapping occlusions.

[0076] In one example, the improved neck network specifically includes a first upsampling layer, a first BiFPN feature pyramid network, a first C2f layer, a second upsampling layer, a second BiFPN feature pyramid network, a second C2f layer, a first standard convolutional Conv layer, a third BiFPN feature pyramid network, a third C2f layer, a first multi-head multi-scale information aggregation layer, a second standard convolutional Conv layer, a connection layer, a fourth C2f layer, and a second multi-scale information aggregation layer, which are connected in sequence. The input ends of the first BiFPN feature pyramid network and the second BiFPN feature pyramid network are also both connected to the backbone network, and the input end of the third BiFPN feature pyramid network is also connected to the output end of the first C2f layer.

[0077] Specifically, the embodiment of the present invention provides an overall framework diagram of an improved YOLOv8 crop leaf segmentation model as Figure 3 shown.

[0078] The network of the improved YOLOv8 model for extracting feature information from the images of the training set includes:

[0079] The first standard convolutional Conv module, with the number of channels = 64 and the convolutional kernel size = 3*3, performs a convolutional operation on the input image, the number of channels becomes 64, and the feature map is reduced to half of the original size.

[0080] The second standard convolutional Conv module, with the number of channels = 128 and the convolutional kernel size = 3*3, continues to perform a convolutional operation on the feature map, the number of channels becomes 128, and the feature map is reduced to half of the original size again.

[0081] The third C2f module, with the number of channels = 128, fuses feature maps of different scales.

[0082] The fourth standard convolutional module, with the number of channels = 256 and the convolutional kernel size = 3*3, performs a convolutional operation on the feature map again, the number of channels becomes 256, and the feature map is reduced to half of the original size again.

[0083] The fifth C2f module, with the number of channels = 256, further fuses the feature map.

[0084] The sixth standard convolutional module, with the number of channels = 512 and the convolutional kernel size = 3*3, continues to perform a convolutional operation on the feature map, the number of channels becomes 512, and the feature map is reduced to half of the original size again.

[0085] The seventh C2f module, with the number of channels = 512, further fuses the feature map.

[0086] The eighth standard convolutional module, with the number of channels = 512 and the convolutional kernel size = 3*3, performs a convolutional operation on the feature map again, the number of channels becomes 512, and the feature map is reduced to half of the original size again.

[0087] The ninth - layer C2f module, with the number of channels = 512, further fuses the feature maps.

[0088] The tenth - layer SPPF module, with the number of channels = 512, parameter = 5, performs spatial pyramid pooling operation, extracts features at different scales, so as to capture the detailed information of the target object at different scales.

[0089] The improved YOLOv8 model for instance segmentation network of the training set images includes:

[0090] The eleventh - layer Upsample layer, performs upsampling, doubling the resolution of the feature map.

[0091] The twelfth - layer Concat_BiFPN layer, connects the P4 feature map of the backbone network with other feature maps, and the other feature maps are the feature maps output by the eleventh - layer Upsample layer.

[0092] The thirteenth - layer C2f layer, with the number of channels being 512, processes the feature map.

[0093] The fourteenth - layer Upsample layer, performs upsampling, doubling the resolution of the feature map.

[0094] The fifteenth - layer Concat_BiFPN layer, connects the P3 feature map of the backbone network with other feature maps, and the other feature maps are the feature maps output by the fourteenth - layer Upsample layer.

[0095] The sixteenth - layer C2f layer, with the number of channels being 256, processes the feature map.

[0096] The seventeenth - layer standard convolution Conv module, with the number of channels being 256, performs convolution operation on the feature map, and the number of channels becomes 256.

[0097] The eighteenth - layer Concat_BiFPN layer, connects the P4 feature map of the head network with other feature maps.

[0098] The nineteenth - layer C2f layer, with the number of channels being 512, processes the feature map.

[0099] The twentieth - layer ContextAggregation layer, aggregates context information from different scales and levels in one layer to enhance the feature representation.

[0100] The twenty - first - layer standard convolution Conv module, with the number of channels = 512, performs convolution operation on the feature map, and the number of channels becomes 512.

[0101] The twenty-second Concat layer connects the P5 feature map of the head network with other feature maps.

[0102] The twenty-third C2f layer, with 512 channels, processes the feature map.

[0103] The twenty-fourth ContextAggregation layer enhances the feature representation by aggregating context information from different scales and levels in one layer.

[0104] The twenty-fifth Segment layer is used for instance segmentation to generate the final leaf segmentation result. The Segment layer is also the Figure 3 head network in. The output ends of the head network correspond to feature maps of different scales, as well as the bounding boxes and categories of the output targets.

[0105] Put the divided training set into the improved YOLOv8 model for training. Such as Figure 4 The schematic diagram of the training result of an improved YOLOv8 crop leaf segmentation model shown in, to obtain the final crop leaf segmentation model.

[0106] After the crop leaf segmentation model is trained, the test set can be put into the crop leaf segmentation model for leaf segmentation. In specific implementation, first, through the backbone network, feature extraction is performed on each crop image data in the image sequence to obtain multi-scale feature maps corresponding to each crop image data; then, through the improved neck network, instance segmentation is performed on the multi-scale feature maps to obtain instance segmentation feature maps corresponding to each crop image data; finally, through the head network, based on the instance segmentation feature maps, leaf segmentation results corresponding to each crop image data are generated. The specific data processing process can be referred to the foregoing Figure 3 and its explanation. The embodiments of the present invention will not be elaborated here.

[0107] For the foregoing step S106, the embodiments of the present invention provide a specific implementation manner for adjusting the number of leaves based on meteorological data to obtain the target number of leaves at different time points, including:

[0108] (1) Determine the target effective accumulated temperature corresponding to the target crop based on the meteorological data. The target effective accumulated temperature can be the daily effective accumulated temperature.

[0109] In one example, the calculation formula for the daily effective accumulated temperature of the crop is as follows:

[0110] K = ∑ i T i - C;

[0111] As described, K represents the daily effective accumulated temperature, Ti is the daily average temperature, and C is the base temperature.

[0112] (2) Adjust the number of leaves according to the target effective accumulated temperature and the pre-constructed crop leaf accumulated temperature fitting curve to obtain the target number of leaves at different time points; wherein, the crop leaf accumulated temperature fitting curve is used to describe the mapping relationship between the effective accumulated temperature and the number of leaves.

[0113] In one example, by combining meteorological data and phenotypic data of the crop, calculate the daily effective accumulated temperature of the crop, and further analyze the fitting relationship between the effective accumulated temperature and the daily number of crop leaves, so as to obtain the fitting relationship curve formula between the two. This curve formula is the crop leaf accumulated temperature fitting curve. By combining the crop leaf segmentation model with the crop leaf accumulated temperature fitting curve formula, realize the dynamic counting of crop leaves, and obtain the target number of leaves of the target crop at different time points.

[0114] See Figure 5 The schematic diagram of the dynamic counting effect of plant leaves shown in, where (a) is the counting effect when the number of leaves on the plant is 3; (b) is the counting effect when the number of leaves on the plant grows to 6; (c) is the counting effect when the number of leaves on the plant grows to 9; (d) is the counting effect when the number of leaves on the plant grows to 12; (e) is the counting effect when the number of leaves on the plant grows to 15.

[0115] For the foregoing step S106, the embodiment of the present invention also provides a specific implementation manner for tracking and matching leaves of each crop image data to determine the leaf identifier associated with the leaves shown in each crop image data, including: taking the crop image data at the current time point as the query image data, and taking the crop image data at all time points before the current time point as the target image data; determining the intersection over union between the leaf pixel mask corresponding to the query image data and the leaf pixel mask corresponding to the target image data, so as to judge whether there is target image data matching the query image data based on the intersection over union; if so, taking the leaf identifier associated with the matching target image data as the leaf identifier associated with the query image data; if not, taking the newly generated unique identifier as the leaf identifier associated with the query image data.

[0116] Specifically, the embodiment of the present invention optimizes the leaf tracking and matching algorithm in the LeTra model. This algorithm abandons the direct pairing method of two consecutive adjacent leaf images in the traditional LeTra model, and instead realizes the comprehensive comparison and pairing of the newly emerged leaf images with all the previous leaf image libraries. For the leaves that fail to be directly successfully paired but show a high intersection over union, the algorithm will intelligently assign a new unique number to them to ensure the continuity and accuracy of leaf tracking.

[0117] Exemplarily, such as Figure 6The working flowchart of an improved leaf tracking and matching algorithm for the LeTra model is shown. The working flowchart of the improved leaf tracking and matching algorithm for the LeTra model, where the area outlined by the dashed box is the improved part. (a) In the initial step, the algorithm retrieves the image mask at time step n (i.e., the current query) and then calculates the intersection over union (IoU) between this mask and the masks of all previously recorded leaf images (time steps from 0 to n - 1, as targets). (b) The decision loop is based on the generated IoU. If the query and the target match, the pair is assigned and the target is removed from the target pool. If no query is found, indicating that the leaf is covered in the image or the detection fails, the position is stored by saving the same mask. Finally, the remaining target masks are used as a new detection method. (c) A list of masks is shown in sequence, which is stored and set as the new query for the next time step.

[0118] Among them, the calculation formula of the intersection over union is as follows:

[0119]

[0120] IoU (Intersection over Union) is an index to measure the overlapping degree of two images (A and B). It is defined as the ratio of the area of the intersection region of image A and image B to the area of their union region, rather than the ratio of their symmetric difference to the union. Generally, the higher the IoU value, the more overlapping parts there are between the two images.

[0121] See Figure 7 The schematic diagram of the effect of an improved leaf tracking and matching algorithm for the LeTra model is shown. It can be seen that the improved LeTra model provided by the embodiment of the present invention can track and match leaves more accurately.

[0122] For the foregoing step S106, the embodiment of the present invention also provides a specific implementation manner for determining the target phenotypic parameters corresponding to each leaf identifier at different time points by combining the leaf mask and the bounding box information, including: determining the leaf area corresponding to each leaf identifier at different time points according to the leaf pixel mask; and determining the leaf length and leaf width corresponding to each leaf identifier at different time points according to the leaf bounding box.

[0123] In practical applications, first, the leaf pixel mask output by the crop leaf segmentation model is used to accurately calculate the leaf area; subsequently, based on the leaf target bounding box information provided by the model at the same time, the leaf length and leaf width of the crop leaf are further analyzed and estimated, so as to achieve comprehensive and accurate acquisition of phenotypic parameters. See Figure 8Schematic diagram of the extraction effect of crop phenotypic parameters. Among them, (a) shows the distribution of the leaf area (in pixels) of each leaf. (b) shows the change in the leaf area of a single leaf during the growth time series of the plant. (c) shows the distribution of the leaf length and width (in pixels) of each leaf. (d) shows the change in the leaf length and width of a single leaf during the growth time series of the plant.

[0124] Finally, in the embodiments of the present invention, through the crop leaf segmentation model combined with the crop leaf accumulated temperature fitting curve formula, the dynamic counting of crop leaves is realized. Based on the crop leaf segmentation model, the phenotypic parameters of crop leaves are obtained, the real-time monitoring of the crop growth process is realized, timely intervention measures are ensured to prevent and control pests and diseases, and it helps to improve the estimation accuracy of crop yield and quality.

[0125] In summary, the crop growth and development monitoring method proposed in the embodiments of the present invention realizes the dynamic counting of crop leaves through deep learning technology combined with image time series data and meteorological data, bringing significant effects to the monitoring of crop growth and development. This method uses computer vision technology to efficiently and non-destructively obtain the phenotypic parameters of crop leaves, accurately detects and segments leaf images through an improved YOLOv8 model, and combines meteorological data to accurately judge the number of leaves, providing the ability to monitor the growth process of crop plants at any time and place. This method not only improves the understanding of the growth status of crop plants, but also can detect and intervene in pests and diseases at an early stage, effectively guaranteeing the yield and quality of crops. In addition, the technologies and achievements studied by this method have wide applicability in the field of crops, providing a safer and more efficient solution for modern agricultural production.

[0126] On the basis of the foregoing embodiments, the embodiments of the present invention provide a crop growth and development monitoring device. Refer to Figure 9 Schematic diagram of the structure of a crop growth and development monitoring device as shown. The device mainly includes the following parts:

[0127] The data acquisition module 902 is used to acquire the image sequence and meteorological data corresponding to the target crop; among them, the image sequence includes crop image data at different time points;

[0128] The leaf segmentation module 904 is used to input the image sequence into a pre-trained crop leaf segmentation model, so as to output the leaf segmentation result corresponding to each crop image data through the crop leaf segmentation model; among them, the leaf segmentation result is used to represent the number of leaves, leaf masks and bounding box information at different time points;

[0129] The quantity and phenotype determination module 906 is used to adjust the leaf quantity based on meteorological data to obtain the target leaf quantity at different time points, so as to realize the dynamic counting of the leaves of the target crop; and, track and match the leaves in each crop image data to determine the leaf identifiers associated with the leaves shown in each crop image data, and combine the leaf mask and bounding box information to determine the target phenotype parameters corresponding to each leaf identifier at different time points;

[0130] The monitoring module 908 is used to monitor the growth and development of the target crop by using the target leaf quantity at different time points and the target phenotype parameters corresponding to each leaf identifier.

[0131] The crop growth and development monitoring device provided by the embodiment of the present invention realizes the dynamic counting of the leaves of the target crop through the deep learning technology combined with the image sequence and meteorological data, bringing remarkable effects to the growth and development monitoring of the target crop; at the same time, by tracking and matching the leaves, and combining the deep learning technology to dynamically monitor the target phenotype parameters corresponding to each leaf identifier at different time points, the accuracy of crop growth and development monitoring is comprehensively improved.

[0132] In one implementation manner, the crop leaf segmentation model adopts an improved YOLOv8 model. The improved YOLOv8 model includes a backbone network, an improved neck network, and a head network. The improved neck network is obtained by replacing the concat layer in the traditional neck network with a BiFPN feature pyramid network; the leaf segmentation module 904 is specifically used for:

[0133] Through the backbone network, feature extraction is performed on each crop image data in the image sequence to obtain a multi-scale feature map corresponding to each crop image data;

[0134] Through the improved neck network, instance segmentation is performed on the multi-scale feature map to obtain an instance segmentation feature map corresponding to each crop image data;

[0135] Through the head network, a leaf segmentation result corresponding to each crop image data is generated based on the instance segmentation feature map.

[0136] In one implementation manner, the improved neck network includes a BiFPN feature pyramid network and a multi-scale information aggregation layer; wherein,

[0137] The BiFPN feature pyramid network is used to connect the first feature map output by the previous layer corresponding to the BiFPN feature pyramid network layer and the second feature map in the multi-scale feature map, and the scales of the first feature map and the second feature map are the same;

[0138] The multi-scale information aggregation layer is used to aggregate the context information of different scales and levels in the third feature map output by the previous layer corresponding to the multi-scale information aggregation layer.

[0139] In one embodiment, the improved neck network specifically includes a first upsampling layer, a first BiFPN feature pyramid network, a first C2f layer, a second upsampling layer, a second BiFPN feature pyramid network, a second C2f layer, a first standard convolution Conv layer, a third BiFPN feature pyramid network, a third C2f layer, a first multi-head multi-scale information aggregation layer, a second standard convolution Conv layer, a connection layer, a fourth C2f layer, and a second multi-scale information aggregation layer, which are connected in sequence. The input ends of the first BiFPN feature pyramid network and the second BiFPN feature pyramid network are also connected to the backbone network, and the input end of the third BiFPN feature pyramid network is also connected to the output end of the first C2f layer.

[0140] In one embodiment, the quantity and phenotype determination module 906 is specifically configured to:

[0141] Determine the target effective accumulated temperature corresponding to the target crop based on meteorological data;

[0142] Adjust the number of leaves according to the target effective accumulated temperature and the pre-constructed crop leaf accumulated temperature fitting curve to obtain the target number of leaves at different time points; wherein, the crop leaf accumulated temperature fitting curve is used to describe the mapping relationship between the effective accumulated temperature and the number of leaves.

[0143] In one embodiment, the leaf mask and bounding box information are the leaf pixel mask and the leaf bounding box; the quantity and phenotype determination module 906 is specifically configured to:

[0144] Take the crop image data at the current time point as the query image data, and take the crop image data at all time points before the current time point as the target image data;

[0145] Determine the intersection over union between the leaf pixel mask corresponding to the query image data and the leaf pixel mask corresponding to the target image data, so as to judge whether there is target image data matching the query image data based on the intersection over union;

[0146] If so, use the leaf identifier associated with the matching target image data as the leaf identifier associated with the query image data;

[0147] If not, use the newly generated unique identifier as the leaf identifier associated with the query image data.

[0148] In one embodiment, the quantity and phenotype determination module 906 is specifically configured to:

[0149] Determine the leaf area corresponding to each leaf identifier at different time points according to the leaf pixel mask; and determine the leaf length and leaf width corresponding to each leaf identifier at different time points according to the leaf bounding box;

[0150] Among them, the target phenotypic parameters include leaf area, leaf length, and leaf width.

[0151] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those in the foregoing method embodiment. For the sake of brief description, for the parts not mentioned in the device embodiment, reference may be made to the corresponding content in the foregoing method embodiment.

[0152] The embodiment of the present invention provides an electronic device. Specifically, the electronic device includes a processor and a storage device; a computer program is stored on the storage device, and when the computer program is run by the processor, it executes the method according to any one of the foregoing embodiments.

[0153] Figure 10 FIG. 11 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. The electronic device 100 includes: a processor 10, a memory 11, a bus 12, and a communication interface 13. The processor 10, the communication interface 13, and the memory 11 are connected through the bus 12; the processor 10 is used to execute an executable module stored in the memory 11, such as a computer program.

[0154] Among them, the memory 11 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. Through at least one communication interface 13 (which may be wired or wireless), a communication connection is established between this system network element and at least one other network element, and the Internet, wide area network, local area network, metropolitan area network, etc. can be used.

[0155] The bus 12 may be an ISA bus, a PCI bus, an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 10 only a bidirectional arrow is used in FIG. 11, but it does not mean that there is only one bus or one type of bus.

[0156] Among them, the memory 11 is used to store a program. After receiving an execution instruction, the processor 10 executes the program. The method executed by the device defined by the flow process disclosed in any one of the foregoing embodiments of the present invention can be applied to the processor 10 or implemented by the processor 10.

[0157] The processor 10 may be an integrated circuit chip with the ability to process signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 10 or the instructions in the form of software. The above-mentioned processor 10 may be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP for short), an application specific integrated circuit (ASIC for short), a field-programmable gate array (FPGA for short), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute each method, step and logic block diagram disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present invention can be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 11, and the processor 10 reads the information in the memory 11 and combines its hardware to complete the steps of the above method.

[0158] The computer program product of the readable storage medium provided by the embodiments of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the method described in the foregoing method embodiments. For the specific implementation, reference can be made to the foregoing method embodiments and will not be elaborated herein.

[0159] When the above-mentioned function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or a part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0160] Finally, it should be noted that the above-mentioned embodiments are only specific implementation manners of the present invention, used to illustrate the technical solutions of the present invention, rather than limiting it. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: any person skilled in the art within the technical scope disclosed by the present invention can still modify the technical solutions recorded in the foregoing embodiments or easily think of changes, or perform equivalent replacements on some of the technical features; and these modifications, changes, or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A method for monitoring the growth and development of crops, characterized in that, Including: Obtain an image sequence and meteorological data corresponding to a target crop; wherein, the image sequence includes crop image data at different time points; Input the image sequence into a pre-trained crop leaf segmentation model to output a leaf segmentation result corresponding to each piece of the crop image data through the crop leaf segmentation model; wherein, the leaf segmentation result is used to characterize the number of leaves, leaf masks, and bounding box information at different time points; Adjust the number of leaves based on the meteorological data to obtain the target number of leaves at different time points, so as to achieve dynamic counting of the leaves of the target crop; and, perform leaf tracking and matching on each piece of the crop image data to determine the leaf identifier associated with the leaves shown in each piece of the crop image data, and combine the leaf mask and bounding box information to determine the target phenotypic parameters corresponding to each leaf identifier at different time points; Utilize the target number of leaves at different time points and the target phenotypic parameters corresponding to each leaf identifier to monitor the growth and development of the target crop; Adjusting the number of leaves based on the meteorological data to obtain the target number of leaves at different time points includes: determining the target effective accumulated temperature corresponding to the target crop based on the meteorological data; adjusting the number of leaves according to the target effective accumulated temperature and a pre-constructed crop leaf accumulated temperature fitting curve to obtain the target number of leaves at different time points; wherein, the crop leaf accumulated temperature fitting curve is used to describe the mapping relationship between the effective accumulated temperature and the number of leaves; The leaf mask and bounding box information are leaf pixel masks and leaf bounding boxes; performing leaf tracking and matching on each piece of the crop image data to determine the leaf identifier associated with the leaves shown in each piece of the crop image data includes: using the crop image data at the current time point as query image data, and using the crop image data at all time points before the current time point as target image data; determining the intersection over union between the leaf pixel mask corresponding to the query image data and the leaf pixel mask corresponding to the target image data, so as to judge whether there is target image data matching the query image data based on the intersection over union; if so, using the leaf identifier associated with the matching target image data as the leaf identifier associated with the query image data; if not, using a newly generated unique identifier as the leaf identifier associated with the query image data.

2. The crop growth and development monitoring method according to claim 1, characterized in that The crop leaf segmentation model adopts an improved YOLOv8 model, and the improved YOLOv8 model includes a backbone network, an improved neck network, and a head network, and the improved neck network is obtained by replacing the concat layer in the traditional neck network with a BiFPN feature pyramid network; Outputting the leaf segmentation result corresponding to each piece of the crop image data through the crop leaf segmentation model includes: Performing feature extraction on each piece of the crop image data in the image sequence through the backbone network to obtain a multi-scale feature map corresponding to each piece of the crop image data; Through the improved neck network, instance segmentation is performed on the multi-scale feature maps to obtain instance segmentation feature maps corresponding to each piece of the crop image data; Through the head network, a leaf segmentation result corresponding to each piece of the crop image data is generated based on the instance segmentation feature maps.

3. The crop growth and development monitoring method according to claim 2, wherein The improved neck network includes a BiFPN feature pyramid network and a multi-scale information aggregation layer; wherein, The BiFPN feature pyramid network is configured to connect a first feature map output by the previous layer corresponding to the BiFPN feature pyramid network layer and a second feature map in the multi-scale feature maps, and the first feature map and the second feature map have the same scale; The multi-scale information aggregation layer is configured to aggregate context information of different scales and levels in a third feature map output by the previous layer corresponding to the multi-scale information aggregation layer.

4. The crop growth and development monitoring method according to claim 3, characterized in that The improved neck network specifically includes a first upsampling layer, a first BiFPN feature pyramid network, a first C2f layer, a second upsampling layer, a second BiFPN feature pyramid network, a second C2f layer, a first standard convolution Conv layer, a third BiFPN feature pyramid network, a third C2f layer, a first multi-head multi-scale information aggregation layer, a second standard convolution Conv layer, a connection layer, a fourth C2f layer, and a second multi-scale information aggregation layer. The input ends of the first BiFPN feature pyramid network and the second BiFPN feature pyramid network are also respectively connected to the backbone network, and the input end of the third BiFPN feature pyramid network is also connected to the output end of the first C2f layer.

5. The crop growth and development monitoring method according to claim 1, characterized in that Combining the leaf mask and the bounding box information to determine the target phenotypic parameters corresponding to each leaf identifier at different time points, including: Determining the leaf area corresponding to each leaf identifier at different time points according to the leaf pixel mask; and determining the leaf length and leaf width corresponding to each leaf identifier at different time points according to the leaf bounding box; Wherein, the target phenotypic parameters include the leaf area, the leaf length, and the leaf width.

6. A crop growth and development monitoring device, characterized in that, Including: A data acquisition module, configured to acquire an image sequence and meteorological data corresponding to a target crop; wherein, the image sequence includes crop image data at different time points; A leaf segmentation module, configured to input the image sequence into a pre-trained crop leaf segmentation model, so as to output a leaf segmentation result corresponding to each piece of the crop image data through the crop leaf segmentation model; wherein, the leaf segmentation result is used to represent the number of leaves, the leaf mask, and the bounding box information at different time points; A quantity and phenotype determination module, configured to adjust the number of leaves based on the meteorological data to obtain the target number of leaves at different time points, so as to realize dynamic counting of the leaves of the target crop; and perform leaf tracking and matching on each piece of the crop image data to determine the leaf identifier associated with the leaves shown in each piece of the crop image data, and combine the leaf mask and the bounding box information to determine the target phenotypic parameters corresponding to each leaf identifier at different time points; A monitoring module, configured to monitor the growth and development of the target crop by using the number of target leaves at different time points and the target phenotypic parameters corresponding to each leaf identifier. The quantity and phenotype determination module is specifically configured to: determine the target effective accumulated temperature corresponding to the target crop based on the meteorological data; adjust the number of leaves according to the target effective accumulated temperature and a pre-constructed crop leaf accumulated temperature fitting curve to obtain the number of target leaves at different time points; wherein, the crop leaf accumulated temperature fitting curve is used to describe the mapping relationship between the effective accumulated temperature and the number of leaves. The leaf mask and bounding box information are a leaf pixel mask and a leaf bounding box; the quantity and phenotype determination module is specifically configured to: use the crop image data at the current time point as query image data, and use the crop image data at all time points before the current time point as target image data; determine the intersection over union between the leaf pixel mask corresponding to the query image data and the leaf pixel mask corresponding to the target image data, so as to judge whether there is target image data matching the query image data based on the intersection over union; if so, use the leaf identifier associated with the matching target image data as the leaf identifier associated with the query image data; if not, use the newly generated unique identifier as the leaf identifier associated with the query image data.

7. An electronic device, characterized in that, It includes a processor and a memory, the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer executable instructions, and when the computer executable instructions are called and executed by a processor, the computer executable instructions cause the processor to implement the method according to any one of claims 1 to 5.

Images