Crop growth and development monitoring method, device, equipment and medium

Abstract

The invention provides a crop growth and development monitoring method and device, equipment and a medium, and relates to the technical field of image recognition, and the method comprises the steps: obtaining an image sequence and meteorological data corresponding to a target crop; inputting the image sequence into a pre-trained crop leaf segmentation model so as to output a leaf segmentation result corresponding to each piece of crop image data through the crop leaf segmentation model; adjusting the number of leaves based on the meteorological data to obtain the number of target leaves at different time points so as to dynamically count the leaves of the target crop; and performing leaf tracking and matching on each piece of crop image data to determine a leaf identifier associated with a leaf displayed by each piece of crop image data, and determining a target phenotype parameter corresponding to each leaf identifier at different time points in combination with the leaf mask and the bounding box information so as to perform growth and development monitoring on a target crop. The accuracy of crop growth and development monitoring can be remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a crop growth and development monitoring method, device, equipment and medium. Background Art

[0002] With the rapid development of modern agricultural technology, accurate monitoring of crop growth status has become increasingly important. Traditional crop monitoring methods often rely on manual observation and recording, which is not only inefficient but also prone to errors and cannot meet the requirements of modern agricultural production for accuracy and efficiency. Especially in greenhouse environments, crops grow faster and require more efficient and accurate monitoring methods to track their growth status. In recent years, deep learning technology has made significant progress in the field of computer vision, providing new possibilities for automated and accurate crop monitoring. Through deep learning models, crop leaves can be automatically detected and counted, and plant organs can be further segmented to calculate the length, width, and area of ​​a single leaf, thereby greatly improving the efficiency and accuracy of monitoring. However, existing crop monitoring methods based on deep learning often rely only on image data, which limits the accuracy of monitoring to a certain extent. Summary of the invention

[0003] In view of this, an object of the present invention is to provide a crop growth and development monitoring method, device, equipment and medium, which can significantly improve the accuracy of crop growth and development monitoring.

[0004] In a first aspect, an embodiment of the present invention provides a crop growth and development monitoring method, comprising:

[0005] Obtaining image sequences and meteorological data corresponding to target crops; wherein the image sequences include crop image data at different time points;

[0006] The image sequence is input into a pre-trained crop leaf segmentation model, so as to output a leaf segmentation result corresponding to each crop image data through the crop leaf segmentation model; wherein the leaf segmentation result is used to characterize the number of leaves and leaf mask and bounding box information at different time points;

[0007] The number of leaves is adjusted based on meteorological data to obtain the target number of leaves at different time points, so as to realize dynamic counting of leaves of target crops; and the leaf tracking and matching is performed on each crop image data to determine the leaf identifier associated with the leaf displayed in each crop image data, and the target phenotypic parameters corresponding to each leaf identifier at different time points are determined by combining the leaf mask and bounding box information;

[0008] The growth and development of target crops are monitored using the target leaf numbers at different time points and the target phenotypic parameters corresponding to each leaf mark.

[0009] In one embodiment, the crop leaf segmentation model adopts an improved YOLOv8 model, the improved YOLOv8 model includes a backbone network, an improved neck network and a head network, the improved neck network is obtained by replacing the concat layer in the traditional neck network with a BiFPN feature pyramid network; the crop leaf segmentation model outputs a leaf segmentation result corresponding to each crop image data, including:

[0010] Through the backbone network, feature extraction is performed on each crop image data in the image sequence to obtain a multi-scale feature map corresponding to each crop image data;

[0011] By improving the neck network, instance segmentation is performed on the multi-scale feature map to obtain the instance segmentation feature map corresponding to each crop image data;

[0012] Through the head network, the leaf segmentation result corresponding to each crop image data is generated based on the instance segmentation feature map.

[0013] In one embodiment, the improved neck network includes a BiFPN feature pyramid network and a multi-scale information aggregation layer; wherein,

[0014] The BiFPN feature pyramid network is used to connect the first feature map output by the previous layer corresponding to the BiFPN feature pyramid network layer with the second feature map in the multi-scale feature map, and the scale of the first feature map is consistent with that of the second feature map;

[0015] The multi-scale information aggregation layer is used to aggregate context information of different scales and levels in the third feature map output by the previous layer corresponding to the multi-scale information aggregation layer.

[0016] In one embodiment, the improved neck network specifically includes a first upsampling layer, a first BiFPN feature pyramid network, a first C2f layer, a second upsampling layer, a second BiFPN feature pyramid network, a second C2f layer, a first standard convolution Conv layer, a third BiFPN feature pyramid network, a third C2f layer, a first multi-head multi-scale information aggregation layer, a second standard convolution Conv layer, a connection layer, a fourth C2f layer and a second multi-scale information aggregation layer, which are connected in sequence. The input ends of the first BiFPN feature pyramid network and the second BiFPN feature pyramid network are also connected to the backbone network, and the input end of the third BiFPN feature pyramid network is also connected to the output end of the first C2f layer.

[0017] In one embodiment, adjusting the number of leaves based on meteorological data to obtain a target number of leaves at different time points includes:

[0018] Determine the target effective accumulated temperature corresponding to the target crop based on meteorological data;

[0019] According to the target effective accumulated temperature and the pre-constructed crop leaf accumulated temperature fitting curve, the number of leaves is adjusted to obtain the target number of leaves at different time points; wherein the crop leaf accumulated temperature fitting curve is used to describe the mapping relationship between the effective accumulated temperature and the number of leaves.

[0020] In one embodiment, the leaf mask and bounding box information is a leaf pixel mask and a leaf bounding box; performing leaf tracking and matching on each crop image data to determine a leaf identifier associated with a leaf displayed by each crop image data includes:

[0021] The crop image data at the current time point is used as the query image data, and the crop image data at all time points before the current time point is used as the target image data;

[0022] Determine an intersection-and-union ratio between a leaf pixel mask corresponding to the query image data and a leaf pixel mask corresponding to the target image data, so as to determine whether there is target image data matching the query image data based on the intersection-and-union ratio;

[0023] If yes, use the leaf identifier associated with the matched target image data as the leaf identifier associated with the query image data;

[0024] If not, the newly generated unique identifier is used as the leaf identifier associated with the query image data.

[0025] In one embodiment, the target phenotypic parameters corresponding to each leaf identifier at different time points are determined in combination with the leaf mask and the bounding box information, including:

[0026] Determine the leaf area corresponding to each leaf identifier at different time points according to the leaf pixel mask; and determine the leaf length and leaf width corresponding to each leaf identifier at different time points according to the leaf bounding box;

[0027] Among them, the target phenotypic parameters include leaf area, leaf length and leaf width.

[0028] In a second aspect, an embodiment of the present invention further provides a crop growth and development monitoring device, comprising:

[0029] A data acquisition module is used to acquire image sequences and meteorological data corresponding to target crops; wherein the image sequences include crop image data at different time points;

[0030] A leaf segmentation module is used to input the image sequence into a pre-trained crop leaf segmentation model, so as to output the leaf segmentation result corresponding to each crop image data through the crop leaf segmentation model; wherein the leaf segmentation result is used to characterize the number of leaves and leaf mask and bounding box information at different time points;

[0031] The number and phenotype determination module is used to adjust the number of leaves based on meteorological data to obtain the target number of leaves at different time points, so as to realize the dynamic counting of leaves of the target crop; and to track and match the leaves of each crop image data to determine the leaf identifier associated with the leaves displayed in each crop image data, and determine the target phenotypic parameters corresponding to each leaf identifier at different time points in combination with the leaf mask and bounding box information;

[0032] The monitoring module is used to monitor the growth and development of the target crop by using the target leaf quantity at different time points and the target phenotypic parameters corresponding to each leaf mark.

[0033] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement any one of the methods provided in the first aspect.

[0034] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement any one of the methods provided in the first aspect.

[0035] The crop growth and development monitoring method, device, equipment and medium provided by the embodiments of the present invention first obtain an image sequence and meteorological data corresponding to a target crop, wherein the image sequence includes crop image data at different time points; then the image sequence is input into a pre-trained crop leaf segmentation model, so as to output a leaf segmentation result corresponding to each crop image data through the crop leaf segmentation model; wherein the leaf segmentation result is used to characterize the number of leaves and leaf mask and bounding box information at different time points; then the number of leaves is adjusted based on the meteorological data to obtain the target number of leaves at different time points, so as to realize dynamic counting of leaves of the target crop; at the same time, leaf tracking and matching are performed on each crop image data to determine the leaf identifier associated with the leaf displayed by each crop image data, and the target phenotypic parameters corresponding to each leaf identifier at different time points are determined in combination with the leaf mask and bounding box information; finally, the target number of leaves at different time points and the target phenotypic parameters corresponding to each leaf identifier are used to monitor the growth and development of the target crop. The above method combines image sequences and meteorological data through deep learning technology to achieve dynamic counting of leaves of target crops, which brings significant results for the growth and development monitoring of target crops. At the same time, by tracking and matching leaves, combined with deep learning technology, the target phenotypic parameters corresponding to each leaf identification at different time points are dynamically monitored, which comprehensively improves the accuracy of crop growth and development monitoring.

[0036] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0037] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A schematic diagram of a process for monitoring crop growth and development provided by an embodiment of the present invention;

[0040] Figure 2 An overall flow chart of a crop growth and development monitoring method provided by an embodiment of the present invention;

[0041] Figure 3 An overall framework diagram of an improved YOLOv8 crop leaf segmentation model provided in an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of training results of an improved YOLOv8 crop leaf segmentation model provided in an embodiment of the present invention;

[0043] Figure 5 A schematic diagram of a dynamic counting effect of plant leaves provided by an embodiment of the present invention;

[0044] Figure 6 A workflow diagram of a blade tracking and matching algorithm of an improved LeTra model provided by an embodiment of the present invention;

[0045] Figure 7 A schematic diagram of the effect of a blade tracking and matching algorithm of an improved LeTra model provided by an embodiment of the present invention;

[0046] Figure 8 A schematic diagram of crop phenotypic parameter extraction effect provided by an embodiment of the present invention;

[0047] Fig. 9 A schematic diagram of the structure of a crop growth and development monitoring device provided by an embodiment of the present invention;

[0048] Fig.10 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] At present, the calculation of crop leaf counts and organ-scale leaf phenotypic data based on deep learning models of static images often results in leaf occlusion problems, which makes leaf counts and phenotypic data extraction inaccurate. At the same time, the impact of meteorological data on crop growth is ignored, making it difficult to meet the demand for the accuracy of continuous observation of the crop growth process. Meteorological conditions such as temperature, humidity, and light have an important impact on crop growth. Combining these data with image data can further improve the accuracy of crop leaf counts. Based on this, the present invention provides a crop growth and development monitoring method, device, equipment, and medium, which can significantly improve the accuracy of crop growth and development monitoring.

[0051] To facilitate understanding of this embodiment, a crop growth and development monitoring method disclosed in an embodiment of the present invention is first described in detail. Figure 1 The schematic diagram of a method for monitoring crop growth and development is shown in FIG. 1 , and the method mainly includes the following steps S102 to S108:

[0052] Step S102, obtaining image sequences and meteorological data corresponding to the target crop.

[0053] Among them, the image sequence contains crop image data at different time points. The crop image data are images of the target crops from the seedling stage to the fruiting stage taken by a camera installed on the greenhouse ceiling or field support poles. The meteorological data are the daily average temperature data of the target crops from the seedling stage to the fruiting stage.

[0054] Step S104, inputting the image sequence into a pre-trained crop leaf segmentation model, so as to output a leaf segmentation result corresponding to each crop image data through the crop leaf segmentation model.

[0055] Among them, the leaf segmentation results are used to characterize the number of leaves and the leaf mask and bounding box information at different time points. The leaf mask and bounding box information are the leaf pixel mask and the leaf bounding box. The crop leaf segmentation model adopts the improved YOLOv8 model. Specifically, the neck network in the existing YOLOv8 model is replaced with the BiFPN feature pyramid network, and the multi-scale information aggregation (Context Aggregation) attention mechanism is used to improve the head network in the YOLOv8 model.

[0056] In one example, the crop image data in the image sequence can be annotated through the open source application LabelMe, and the annotated data set can be divided into a training set and a test set in proportion. The training set is used to train the crop leaf segmentation model, and the trained crop leaf segmentation model is used to perform leaf segmentation on the test set to obtain the leaf segmentation result corresponding to each crop image data in the test set.

[0057] Step S106, adjusting the number of leaves based on meteorological data to obtain the target number of leaves at different time points, so as to realize dynamic counting of leaves of the target crop; and tracking and matching leaves for each crop image data to determine the leaf identifier associated with the leaves displayed in each crop image data, and combining the leaf mask and bounding box information to determine the target phenotypic parameters corresponding to each leaf identifier at different time points.

[0058] The leaf identifier is a unique code associated with each leaf, and the target phenotypic data is the phenotypic information of the crop plant under the camera, including leaf length, leaf width, leaf area, etc.

[0059] In one example, the phenotypic data corresponding to each crop image data in the training set can be obtained, and the meteorological data and the phenotypic data can be combined to calculate the daily effective cumulative temperature of the target crop (referred to as the effective accumulated temperature), and further analyze the fitting relationship between the effective accumulated temperature and the daily number of crop leaves, so as to obtain the fitting relationship curve formula between the two, which is recorded as the crop leaf accumulated temperature fitting curve; using the crop leaf accumulated temperature fitting curve and the temperature data corresponding to each crop image data in the test set, the aforementioned number of leaves is adjusted to obtain the target number of leaves at different time points.

[0060] In one example, an embodiment of the present invention optimizes the leaf tracking and matching algorithm in the traditional LeTra model. The embodiment of the present invention abandons the direct pairing method of two consecutive crop image data in the traditional LeTra model, and instead realizes a comprehensive comparison and pairing of the newly appeared crop image data with all the crop image data recorded in the previous period. For leaves that fail to be directly successfully paired but show a high intersection-over-union ratio, the algorithm will intelligently assign them a new unique number to ensure the continuity and accuracy of leaf tracking, thereby determining the leaf identifier associated with the leaf displayed by each crop image data. Combined with the aforementioned leaf mask and bounding box information, the target phenotypic parameters corresponding to each leaf identifier at different time points can be determined.

[0061] Step S108, monitoring the growth and development of the target crop using the target leaf numbers at different time points and the target phenotypic parameters corresponding to each leaf identifier.

[0062] The crop growth and development monitoring method provided by the embodiment of the present invention realizes dynamic counting of leaves of target crops by combining deep learning technology with image sequences and meteorological data, bringing significant effects on the growth and development monitoring of target crops; at the same time, by tracking and matching leaves and combining deep learning technology to dynamically monitor the target phenotypic parameters corresponding to each leaf identification at different time points, the accuracy of crop growth and development monitoring is comprehensively improved.

[0063] The embodiment of the present invention proposes a method for dynamically counting crop leaves that combines deep learning, image time series data and meteorological data, and then automatically calculates leaf length, leaf width, leaf area, etc. based on segmented images, aiming to achieve more efficient and accurate crop growth and development monitoring.

[0064] For ease of understanding, the embodiments of the present invention provide Figure 2 The overall flow chart of a crop growth and development monitoring method shown in the figure includes: obtaining image sequences and meteorological data corresponding to target crops to form a crop data set; annotating the image sequences of the crop data set and dividing them into training sets and test sets according to proportion; improving the YOLOv8 model, introducing the BiFPN feature pyramid network and the ContextAggregation attention mechanism; and realizing the dynamic counting of crop leaves and the extraction of phenotypic parameters by combining the improved YOLOv8 model with the crop leaf accumulated temperature fitting curve.

[0065] In specific implementation, an embodiment of the present invention provides a specific implementation of the aforementioned step S102, including: initial data acquisition, the initial data acquisition includes acquiring an image sequence, phenotypic data, and meteorological data of the crop. The image sequence of the crop is a sequence of images of the crop plant (i.e., the target crop) from the seedling stage to the fruiting stage, viewed directly by a camera installed on the greenhouse ceiling or field support poles. The phenotypic data is phenotypic information of the crop plant under the camera, including leaf length, leaf width, leaf area, etc. The meteorological data is the daily average temperature data of the crop plant from the seedling stage to the fruiting stage.

[0066] On this basis, the crop dataset images were annotated using the open source application LabelMe, and the annotated dataset was divided into training and test sets in proportion. The annotated content includes the precise outline bounding box of each leaf shown in the image.

[0067] Before explaining the aforementioned step S104, an embodiment of the present invention provides a specific implementation of a crop leaf segmentation model. The crop leaf segmentation model adopts an improved YOLOv8 model. The improved YOLOv8 model includes a backbone network, an improved neck network and a head network. The improved neck network is obtained by replacing the concat layer in the traditional neck network with a BiFPN feature pyramid network; the improved neck network includes a BiFPN feature pyramid network (that is, the Concat_BiFPN layer) and a multi-scale information aggregation layer.

[0068] Among them, the Concat_BiFPN layer is used to connect the first feature map output by the previous layer corresponding to the Concat_BiFPN layer with the second feature map in the multi-scale feature map, and the scale of the first feature map is consistent with that of the second feature map. The Concat_BiFPN layer uses the BiFPN network structure and the fast normalization method for weighted fusion. The calculation formula is as follows:

[0069]

[0070] Said: I i is the input crop leaf feature, O is the output crop leaf feature, W i and W j is the learnable weight, ε is the minimum learning rate used to constrain numerical oscillations, and stable output can be guaranteed when ε = 0.0001.

[0071] Through the weighted cross-scale feature fusion mechanism, the network model can pay appropriate attention to the features of crop leaves of different sizes at different levels. For smaller crop leaves, BiFPN can retain and fuse the fine-grained features of the bottom level and combine it with the semantic features of higher levels, thereby improving the detection accuracy of smaller crop leaves. For larger crop leaves, BiFPN can appropriately weight the features from each level through an adaptive feature fusion mechanism, so that the network pays more attention to the level where larger crop leaves are located, thereby improving the detection ability of larger crop leaves. Through bidirectional feature fusion and adaptive weight adjustment, BiFPN can well capture the features of crop leaves of different sizes, thereby improving the detection accuracy of the model.

[0072] Among them, the multi-scale information aggregation layer (also known as the ContextAggregation attention mechanism) is used to aggregate context information of different scales and levels in the third feature map output by the previous layer corresponding to the multi-scale information aggregation layer. The calculation formula of the ContextAggregation attention mechanism is as follows:

[0073]

[0074] As mentioned above, i and Q i Represents the input and output crop leaf feature maps of the i-th layer in the feature pyramid. Each crop leaf feature map contains N i pixels; j, m∈{1, N i}, represents the index of each pixel, and Represents the feature vector with index j on the input and output crop leaf feature map of the i-th layer in the feature pyramid, is the scalar attention weight, w k and w v is the linear transformation matrix used to project the feature map. This module uses 1×1 convolution to perform the mapping.

[0075] The ContextAggregation attention mechanism can learn to aggregate features from the entire crop leaf feature map and combine them into each pixel using adaptive weights, which enhances the feature extraction capability and expands the receptive field. It can effectively fuse local and global features while reducing information confusion, thus improving the recognition effect of the Yolov8 network on smaller and overlapping crop leaf targets.

[0076] In one example, the improved neck network specifically includes a first upsampling layer, a first BiFPN feature pyramid network, a first C2f layer, a second upsampling layer, a second BiFPN feature pyramid network, a second C2f layer, a first standard convolution Conv layer, a third BiFPN feature pyramid network, a third C2f layer, a first multi-head multi-scale information aggregation layer, a second standard convolution Conv layer, a connection layer, a fourth C2f layer and a second multi-scale information aggregation layer, which are connected in sequence. The input ends of the first BiFPN feature pyramid network and the second BiFPN feature pyramid network are also connected to the backbone network, and the input end of the third BiFPN feature pyramid network is also connected to the output end of the first C2f layer.

[0077] Specifically, the embodiments of the present invention provide Figure 3 The overall framework diagram of an improved YOLOv8 crop leaf segmentation model is shown.

[0078] The network used to improve the YOLOv8 model for extracting feature information from images in the training set includes:

[0079] The first layer of standard convolution Conv module has 64 channels and 3*3 kernel size. The convolution operation is performed on the input image, the number of channels becomes 64, and the feature map is reduced to half of the original size.

[0080] The second layer of standard convolution Conv module has the number of channels = 128 and the convolution kernel size = 3*3. The convolution operation is continued on the feature map. The number of channels becomes 128 and the feature map is reduced to half of the original size again.

[0081] The third layer C2f module has 128 channels and fuses feature maps of different scales.

[0082] The fourth layer of standard convolution module has the number of channels = 256 and the convolution kernel size = 3*3. The feature map is convolved again, the number of channels becomes 256, and the feature map is reduced to half of its original size.

[0083] The fifth layer C2f module, with the number of channels = 256, further fuses the feature maps.

[0084] The sixth layer of standard convolution module, the number of channels = 512, the convolution kernel size = 3*3, continues to perform convolution operation on the feature map, the number of channels becomes 512, and the feature map is reduced to half of the original size again.

[0085] The seventh layer C2f module, with the number of channels = 512, further fuses the feature maps.

[0086] The eighth layer of standard convolution module, the number of channels = 512, the convolution kernel size = 3*3, the feature map is convolved again, the number of channels becomes 512, and the feature map is reduced to half of the original size again.

[0087] The ninth layer C2f module, with the number of channels = 512, further fuses the feature maps.

[0088] The tenth layer SPPF module, with the number of channels = 512 and the parameter = 5, performs a spatial pyramid pooling operation to extract features at different scales, thereby capturing the detailed information of the target object at different scales.

[0089] The improved YOLOv8 model is used to perform instance segmentation on the images of the training set. The network includes:

[0090] The eleventh layer, Upsample, performs upsampling and doubles the resolution of the feature map.

[0091] The twelfth layer, Concat_BiFPN, connects the P4 feature map of the backbone network with other feature maps, which are also the feature maps output by the eleventh layer, Upsample layer.

[0092] The thirteenth layer, C2f, has 512 channels and processes the feature map.

[0093] The fourteenth Upsample layer performs upsampling and doubles the resolution of the feature map.

[0094] The fifteenth Concat_BiFPN layer connects the P3 feature map of the backbone network with other feature maps, which are also the feature maps output by the fourteenth Upsample layer.

[0095] The sixteenth layer, C2f, has 256 channels and processes the feature map.

[0096] The seventeenth layer of the standard convolution Conv module has 256 channels. The convolution operation is performed on the feature map, and the number of channels becomes 256.

[0097] The eighteenth Concat_BiFPN layer connects the P4 feature map of the head network with other feature maps.

[0098] The nineteenth layer, C2f, has 512 channels and processes the feature map.

[0099] The twentieth ContextAggregation layer enhances feature representation by aggregating context information from different scales and levels in one layer.

[0100] The 21st layer standard convolution Conv module has a channel number of 512. The convolution operation is performed on the feature map, and the channel number becomes 512.

[0101] The 22nd Concat layer connects the P5 feature map of the head network with other feature maps.

[0102] The twenty-third layer C2f has 512 channels and processes the feature map.

[0103] The twenty-fourth ContextAggregation layer enhances feature representation by aggregating context information from different scales and levels in one layer.

[0104] The 25th Segment layer is used for instance segmentation and generates the final leaf segmentation result. Figure 3 The output of the head network in the figure corresponds to feature maps of different scales, as well as the bounding box and category of the output target.

[0105] Put the divided training set into the improved YOLOv8 model for training, such as Figure 4 A schematic diagram of the training results of an improved YOLOv8 crop leaf segmentation model is shown, and the final crop leaf segmentation model is obtained.

[0106] After the crop leaf segmentation model training is completed, the test set can be put into the crop leaf segmentation model for leaf segmentation. In the specific implementation, firstly, the backbone network is used to extract the features of each crop image data in the image sequence to obtain a multi-scale feature map corresponding to each crop image data; then, the multi-scale feature map is instance segmented through the improved neck network to obtain an instance segmentation feature map corresponding to each crop image data; finally, the head network is used to generate the leaf segmentation result corresponding to each crop image data based on the instance segmentation feature map. For details of the data processing process, please refer to the above. Figure 3 The embodiments of the present invention will not be further described in detail.

[0107] For the aforementioned step S106, the embodiment of the present invention provides a specific implementation method of adjusting the number of blades based on meteorological data to obtain the target number of blades at different time points, including:

[0108] (1) Determine the target effective accumulated temperature corresponding to the target crop based on meteorological data. The target effective accumulated temperature can be the daily effective accumulated temperature.

[0109] In one example, the daily effective accumulated temperature of the crop is calculated as follows:

[0110] K=∑ i T i -C;

[0111] As mentioned above, K represents the daily effective cumulative temperature, Ti is the daily average temperature, and C is the reference temperature.

[0112] (2) According to the target effective accumulated temperature and the pre-constructed crop leaf accumulated temperature fitting curve, the number of leaves is adjusted to obtain the target number of leaves at different time points; wherein the crop leaf accumulated temperature fitting curve is used to describe the mapping relationship between the effective accumulated temperature and the number of leaves.

[0113] In one example, by combining meteorological data and crop phenotypic data, the daily effective cumulative temperature of the crop is calculated, and the fitting relationship between the effective accumulated temperature and the daily number of crop leaves is further analyzed to obtain the fitting relationship curve formula between the two. The curve formula is the crop leaf accumulated temperature fitting curve. The crop leaf segmentation model is combined with the crop leaf accumulated temperature fitting curve formula to realize the dynamic counting of crop leaves and obtain the target number of leaves of the target crop at different time points.

[0114] See also Figure 5 The figure shows a schematic diagram of the dynamic counting effect of leaves on a plant, wherein (a) the counting effect when the number of leaves on the plant is 3; (b) the counting effect when the number of leaves on the plant increases to 6; (c) the counting effect when the number of leaves on the plant increases to 9; (d) the counting effect when the number of leaves on the plant increases to 12; (e) the counting effect when the number of leaves on the plant increases to 15.

[0115] For the aforementioned step S106, an embodiment of the present invention further provides a specific implementation method for tracking and matching leaves for each crop image data to determine the leaf identifier associated with the leaves displayed by each crop image data, including: taking the crop image data at the current time point as the query image data, and taking the crop image data at all time points before the current time point as the target image data; determining the intersection-and-union ratio between the leaf pixel mask corresponding to the query image data and the leaf pixel mask corresponding to the target image data, so as to determine whether there is target image data matching the query image data based on the intersection-and-union ratio; if so, taking the leaf identifier associated with the matched target image data as the leaf identifier associated with the query image data; if not, taking the newly generated unique identifier as the leaf identifier associated with the query image data.

[0116] Specifically, the embodiment of the present invention optimizes the leaf tracking and matching algorithm in the LeTra model. The algorithm abandons the direct pairing of two consecutive adjacent leaf images in the traditional LeTra model, and instead realizes a comprehensive comparison and pairing of the newly appeared leaf image with all the leaf image libraries recorded in the previous period. For leaves that fail to be directly successfully paired but show a high intersection-to-union ratio, the algorithm will intelligently assign them a new unique number to ensure the continuity and accuracy of leaf tracking.

[0117] For example, Figure 6The workflow diagram of the leaf tracking and matching algorithm of the improved LeTra model is shown, and the workflow diagram of the leaf tracking and matching algorithm of the improved LeTra model is shown, in which the area outlined by the dashed line is the improved part. (a) In the initial step, the algorithm retrieves the image mask of time step n (i.e., the current query), and then calculates the intersection over union (IoU) between the mask and the masks of all leaf images recorded in the previous period (time steps from 0 to n-1, as targets). (b) The decision loop is based on the generated IoU. If the query and target match, the pair is assigned and the target is removed from the target pool. If the query is not found, indicating that the leaf is covered in the image or the detection fails, the position is stored by saving the same mask. Finally, the remaining target mask is used as a new detection method. (c) A list of masks is shown in order, which is stored and set as the new query for the next time step.

[0118] The calculation formula of intersection-over-union ratio is as follows:

[0119]

[0120] IoU (Intersection over Union) is a measure of the degree of overlap between two images (A and B). It is defined as the ratio of the area of ​​the intersection of image A and image B to the area of ​​their union, rather than the ratio of their symmetric difference to their union. Generally, the higher the IoU value, the more overlap there is between the two images.

[0121] See also Figure 7 The schematic diagram of the effect of the blade tracking and matching algorithm of the improved LeTra model is shown. It can be seen that the improved LeTra model provided by the embodiment of the present invention can track and match the blades more accurately.

[0122] For the aforementioned step S106, an embodiment of the present invention further provides a specific implementation method for determining the target phenotypic parameters corresponding to each leaf identifier at different time points by combining the leaf mask and the bounding box information, including: determining the leaf area corresponding to each leaf identifier at different time points according to the leaf pixel mask; and determining the leaf length and leaf width corresponding to each leaf identifier at different time points according to the leaf bounding box.

[0123] In practical applications, the leaf pixel mask output by the crop leaf segmentation model is first used to accurately calculate the leaf area; then, based on the leaf target bounding box information provided by the model, the leaf length and width of the crop leaf are further analyzed and estimated, thereby achieving comprehensive and accurate acquisition of phenotypic parameters. Figure 8A schematic diagram of crop phenotypic parameter extraction effect is shown, where (a) shows the leaf area (in pixels) distribution of each leaf. (b) shows the change of leaf area of ​​a single leaf during the plant growth time series. (c) shows the leaf length and leaf width (in pixels) distribution of each leaf. (d) shows the change of leaf length and leaf width of a single leaf during the plant growth time series.

[0124] Finally, the embodiment of the present invention realizes dynamic counting of crop leaves through the crop leaf segmentation model combined with the crop leaf accumulated temperature fitting curve formula, obtains the phenotypic parameters of crop leaves according to the crop leaf segmentation model, realizes real-time monitoring of the crop growth process, ensures timely intervention measures to prevent and control pests and diseases, and helps to improve the estimation accuracy of crop yield and quality.

[0125] In summary, the crop growth and development monitoring method proposed in the embodiment of the present invention realizes the dynamic counting of crop leaves by combining image time series data and meteorological data through deep learning technology, which brings remarkable results to the growth and development monitoring of crops. The method uses computer vision technology to efficiently and non-destructively obtain the phenotypic parameters of crop leaves, accurately detects and segments leaf images through the improved YOLOv8 model, and accurately determines the number of leaves in combination with meteorological data, providing the ability to monitor the growth process of crop plants anytime and anywhere. This method not only improves the level of understanding of the growth status of crop plants, but also can detect and intervene in pests and diseases at an early stage, effectively ensuring the yield and quality of crops. In addition, the technology and results studied by this method have wide applicability in the field of crops, providing a safer and more efficient solution for modern agricultural production.

[0126] Based on the above embodiments, the present invention provides a crop growth monitoring device. Fig. 9 The structure diagram of a crop growth and development monitoring device shown in FIG. 1 mainly includes the following parts:

[0127] The data acquisition module 902 is used to acquire image sequences and meteorological data corresponding to the target crop; wherein the image sequence includes crop image data at different time points;

[0128] The leaf segmentation module 904 is used to input the image sequence into a pre-trained crop leaf segmentation model, so as to output the leaf segmentation result corresponding to each crop image data through the crop leaf segmentation model; wherein the leaf segmentation result is used to characterize the number of leaves and leaf mask and bounding box information at different time points;

[0129] The number and phenotype determination module 906 is used to adjust the number of leaves based on meteorological data to obtain the target number of leaves at different time points, so as to realize dynamic counting of leaves of the target crop; and to track and match leaves for each crop image data to determine the leaf identifier associated with the leaf displayed by each crop image data, and to determine the target phenotypic parameters corresponding to each leaf identifier at different time points in combination with the leaf mask and bounding box information;

[0130] The monitoring module 908 is used to monitor the growth and development of the target crop by using the target leaf numbers at different time points and the target phenotypic parameters corresponding to each leaf identifier.

[0131] The crop growth and development monitoring device provided in the embodiment of the present invention realizes dynamic counting of leaves of target crops by combining deep learning technology with image sequences and meteorological data, bringing remarkable effects to the growth and development monitoring of target crops; at the same time, by tracking and matching leaves and combining deep learning technology to dynamically monitor the target phenotypic parameters corresponding to each leaf identification at different time points, the accuracy of crop growth and development monitoring is comprehensively improved.

[0132] In one embodiment, the crop leaf segmentation model adopts an improved YOLOv8 model, which includes a backbone network, an improved neck network and a head network. The improved neck network is obtained by replacing the concat layer in the traditional neck network with a BiFPN feature pyramid network; the leaf segmentation module 904 is specifically used for:

[0133] Through the backbone network, feature extraction is performed on each crop image data in the image sequence to obtain a multi-scale feature map corresponding to each crop image data;

[0134] By improving the neck network, instance segmentation is performed on the multi-scale feature map to obtain the instance segmentation feature map corresponding to each crop image data;

[0135] Through the head network, the leaf segmentation result corresponding to each crop image data is generated based on the instance segmentation feature map.

[0136] In one embodiment, the improved neck network includes a BiFPN feature pyramid network and a multi-scale information aggregation layer; wherein,

[0137] The BiFPN feature pyramid network is used to connect the first feature map output by the previous layer corresponding to the BiFPN feature pyramid network layer with the second feature map in the multi-scale feature map, and the scale of the first feature map is consistent with that of the second feature map;

[0138] The multi-scale information aggregation layer is used to aggregate context information of different scales and levels in the third feature map output by the previous layer corresponding to the multi-scale information aggregation layer.

[0139] In one embodiment, the improved neck network specifically includes a first upsampling layer, a first BiFPN feature pyramid network, a first C2f layer, a second upsampling layer, a second BiFPN feature pyramid network, a second C2f layer, a first standard convolution Conv layer, a third BiFPN feature pyramid network, a third C2f layer, a first multi-head multi-scale information aggregation layer, a second standard convolution Conv layer, a connection layer, a fourth C2f layer and a second multi-scale information aggregation layer, which are connected in sequence. The input ends of the first BiFPN feature pyramid network and the second BiFPN feature pyramid network are also connected to the backbone network, and the input end of the third BiFPN feature pyramid network is also connected to the output end of the first C2f layer.

[0140] In one embodiment, the quantity and phenotype determination module 906 is specifically used to:

[0141] Determine the target effective accumulated temperature corresponding to the target crop based on meteorological data;

[0142] According to the target effective accumulated temperature and the pre-constructed crop leaf accumulated temperature fitting curve, the number of leaves is adjusted to obtain the target number of leaves at different time points; wherein the crop leaf accumulated temperature fitting curve is used to describe the mapping relationship between the effective accumulated temperature and the number of leaves.

[0143] In one embodiment, the leaf mask and bounding box information is a leaf pixel mask and a leaf bounding box; the quantity and phenotype determination module 906 is specifically used to:

[0144] The crop image data at the current time point is used as the query image data, and the crop image data at all time points before the current time point is used as the target image data;

[0145] Determine an intersection-and-union ratio between a leaf pixel mask corresponding to the query image data and a leaf pixel mask corresponding to the target image data, so as to determine whether there is target image data matching the query image data based on the intersection-and-union ratio;

[0146] If yes, use the leaf identifier associated with the matched target image data as the leaf identifier associated with the query image data;

[0147] If not, the newly generated unique identifier is used as the leaf identifier associated with the query image data.

[0148] In one embodiment, the quantity and phenotype determination module 906 is specifically used to:

[0149] Determine the leaf area corresponding to each leaf identifier at different time points according to the leaf pixel mask; and determine the leaf length and leaf width corresponding to each leaf identifier at different time points according to the leaf bounding box;

[0150] Among them, the target phenotypic parameters include leaf area, leaf length and leaf width.

[0151] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0152] An embodiment of the present invention provides an electronic device. Specifically, the electronic device includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes the method described in any one of the above-mentioned implementation methods.

[0153] Fig.10 A structural diagram of an electronic device provided in an embodiment of the present invention, the electronic device 100 includes: a processor 10, a memory 11, a bus 12 and a communication interface 13, wherein the processor 10, the communication interface 13 and the memory 11 are connected via the bus 12; the processor 10 is used to execute an executable module stored in the memory 11, such as a computer program.

[0154] The memory 11 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 13 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0155] The bus 12 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0156] Among them, the memory 11 is used to store programs, and the processor 10 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 10 or implemented by the processor 10.

[0157] The processor 10 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 10. The above processor 10 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 11, and the processor 10 reads the information in the memory 11 and completes the steps of the above method in combination with its hardware.

[0158] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be referred to the previous method embodiments, which will not be repeated here.

[0159] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0160] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for monitoring crop growth and development, characterized in that: include: Acquire an image sequence and meteorological data corresponding to a target crop; wherein the image sequence includes crop image data at different time points; Inputting the image sequence into a pre-trained crop leaf segmentation model, so as to output a leaf segmentation result corresponding to each crop image data through the crop leaf segmentation model; wherein the leaf segmentation result is used to characterize the number of leaves and leaf mask and bounding box information at different time points; The number of leaves is adjusted based on the meteorological data to obtain the target number of leaves at different time points, so as to realize the dynamic counting of leaves of the target crop; and leaf tracking and matching are performed on each crop image data to determine the leaf identifier associated with each leaf displayed by the crop image data, and the target phenotypic parameter corresponding to each leaf identifier at different time points is determined in combination with the leaf mask and bounding box information; The growth and development of the target crop is monitored by using the target leaf quantity at different time points and the target phenotypic parameter corresponding to each leaf identifier.

2. The crop growth monitoring method according to claim 1, characterized in that: The crop leaf segmentation model adopts an improved YOLOv8 model, which includes a backbone network, an improved neck network and a head network. The improved neck network is obtained by replacing the concat layer in the traditional neck network with a BiFPN feature pyramid network. Outputting the leaf segmentation result corresponding to each crop image data through the crop leaf segmentation model includes: Through the backbone network, feature extraction is performed on each crop image data in the image sequence to obtain a multi-scale feature map corresponding to each crop image data; By using the improved neck network, instance segmentation is performed on the multi-scale feature map to obtain an instance segmentation feature map corresponding to each crop image data; Through the head network, a leaf segmentation result corresponding to each crop image data is generated based on the instance segmentation feature map.

3. The crop growth monitoring method according to claim 2, characterized in that: The improved neck network includes a BiFPN feature pyramid network and a multi-scale information aggregation layer; wherein, The BiFPN feature pyramid network is used to connect a first feature map output by a previous layer corresponding to the BiFPN feature pyramid network layer and a second feature map in the multi-scale feature map, wherein the scale of the first feature map is consistent with that of the second feature map; The multi-scale information aggregation layer is used to aggregate context information of different scales and levels in the third feature map output by a previous layer corresponding to the multi-scale information aggregation layer.

4. The crop growth monitoring method according to claim 3, characterized in that: The improved neck network specifically includes a first upsampling layer, a first BiFPN feature pyramid network, a first C2f layer, a second upsampling layer, a second BiFPN feature pyramid network, a second C2f layer, a first standard convolution Conv layer, a third BiFPN feature pyramid network, a third C2f layer, a first multi-head multi-scale information aggregation layer, a second standard convolution Conv layer, a connection layer, a fourth C2f layer and a second multi-scale information aggregation layer, which are connected in sequence. The input ends of the first BiFPN feature pyramid network and the second BiFPN feature pyramid network are also connected to the backbone network, and the input end of the third BiFPN feature pyramid network is also connected to the output end of the first C2f layer.

5. The crop growth monitoring method according to claim 1, characterized in that: Adjusting the number of blades based on the meteorological data to obtain target numbers of blades at different time points includes: Determining a target effective accumulated temperature corresponding to the target crop based on the meteorological data; According to the target effective accumulated temperature and the pre-constructed crop leaf accumulated temperature fitting curve, the number of leaves is adjusted to obtain the target number of leaves at different time points; wherein the crop leaf accumulated temperature fitting curve is used to describe the mapping relationship between the effective accumulated temperature and the number of leaves.

6. The crop growth monitoring method according to claim 1, characterized in that: The leaf mask and bounding box information are a leaf pixel mask and a leaf bounding box; Tracking and matching leaves on each of the crop image data to determine a leaf identifier associated with a leaf displayed by each of the crop image data includes: Using the crop image data at the current time point as query image data, and using the crop image data at all time points before the current time point as target image data; Determine an intersection-and-union ratio between the leaf pixel mask corresponding to the query image data and the leaf pixel mask corresponding to the target image data, so as to determine whether there is the target image data matching the query image data based on the intersection-and-union ratio; If yes, using the leaf identifier associated with the matched target image data as the leaf identifier associated with the query image data; If not, the newly generated unique identifier is used as the leaf identifier associated with the query image data.

7. The crop growth monitoring method according to claim 6, characterized in that: Determining the target phenotypic parameters corresponding to each leaf identifier at different time points in combination with the leaf mask and bounding box information includes: Determine the leaf area corresponding to each leaf identifier at different time points according to the leaf pixel mask; and determine the leaf length and leaf width corresponding to each leaf identifier at different time points according to the leaf boundary box; Wherein, the target phenotypic parameters include the leaf area, the leaf length and the leaf width.

8. A crop growth monitoring device, characterized in that: include: A data acquisition module, used to acquire image sequences and meteorological data corresponding to target crops; wherein the image sequences include crop image data at different time points; A leaf segmentation module, used for inputting the image sequence into a pre-trained crop leaf segmentation model, so as to output a leaf segmentation result corresponding to each crop image data through the crop leaf segmentation model; wherein the leaf segmentation result is used for representing the number of leaves and leaf mask and bounding box information at different time points; A quantity and phenotype determination module is used to adjust the number of leaves based on the meteorological data to obtain the target number of leaves at different time points, so as to realize dynamic counting of leaves of the target crop; and to track and match leaves for each crop image data to determine the leaf identifier associated with each leaf displayed by the crop image data, and to determine the target phenotypic parameters corresponding to each leaf identifier at different time points in combination with the leaf mask and bounding box information; The monitoring module is used to monitor the growth and development of the target crop by using the target leaf quantity at different time points and the target phenotypic parameter corresponding to each leaf identifier.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 7.

Images