A rice growth state monitoring method based on image and environment information fusion

Abstract

The application discloses a rice growth state monitoring method based on image and environment information fusion, comprising: acquiring image information and environment information in the growth process of rice; determining a physiological zero point of environment characteristic accumulation; constructing a multi-dimensional environment information vector based on the physiological zero point and environment information after the physiological zero point; performing visual feature extraction on the image information by using a visual backbone network to obtain image features; splitting the multi-dimensional environment information vector into three environment sub-feature groups; mapping the accumulated light and temperature resource feature group and the recent light and temperature driving feature group to cross-attention interaction with the image features to obtain environment corrected image global features; and modulating the environment corrected image global features based on an environment fluctuation stress feature group to obtain modulated fusion features, inputting the modulated fusion features into a regression prediction network, and outputting a rice growth state prediction value. Based on this, the application improves the accuracy, robustness and cross-field generalization ability of rice growth state monitoring.

Description

Technical Field

[0001] This invention relates to the field of agricultural remote sensing and crop phenotypic monitoring technology, specifically to a method for monitoring the growth status of rice based on the fusion of image and environmental information. Background Technology

[0002] Existing methods for monitoring rice growth stages and vigor can be broadly categorized into two types: models based on meteorological and physiological priors and models driven by observational data. Typical methods based on meteorological and physiological priors use accumulated temperature (growing degree-days, GDD) or light and temperature resources as core driving factors, using empirical formulas, developmental rate models, or crop models to extrapolate key stages such as tillering, jointing, heading, and grain filling. While these methods can depict the macroscopic laws of the growth process, they typically require parameter calibration for variety, planting date, and field management (fertilization, irrigation, density). When there are significant differences in soil physicochemical properties, soil fertility, and moisture conditions between fields, or when there are climate anomalies, model errors can easily accumulate, leading to delayed or premature stage judgments, making it difficult to directly and stably reuse the model at a regional scale.

[0003] With the development of field sensing and visual computing, image-based direct regression and classification have gradually become important approaches for plant growth monitoring. Existing solutions include: using imagery from surveillance cameras, fixed ground cameras, or drones to construct regression models based on handcrafted features such as color indices (e.g., greenness index), texture features, and canopy coverage; and using deep learning networks such as convolutional neural networks (CNNs) and visual transformers to directly map images to indicators such as growth stage, leaf area index (LAI / vegetation area index), plant height, or biomass. Image-based direct regression can reflect the true plant growth in the field to some extent, but its effectiveness is highly dependent on the consistency of the shooting conditions and the scene.

[0004] Besides accumulated temperature information, it is difficult to obtain stable, generalizable and time-consistent multi-indicator monitoring results in complex field environments by using only image direct regression. Summary of the Invention

[0005] To overcome the shortcomings of existing rice growth period and growth monitoring methods, such as difficulty in directly and stably reusing them at the regional scale, high dependence on the consistency of shooting conditions and scenes, and difficulty in obtaining stable, generalizable, and time-consistent multi-indicator monitoring results in complex field environments, this invention provides a rice growth status monitoring method based on the fusion of image and environmental information. By deeply integrating environmental and image information, this method improves the accuracy, robustness, and cross-field generalization ability of rice growth status monitoring.

[0006] According to one aspect of this invention, a method for monitoring the growth status of rice based on image and environmental information fusion is provided, comprising: acquiring image information and environmental information during the rice growth process to obtain a monitoring sequence; determining the date when the rice is in the tillering stage in the monitoring sequence and using it as the physiological zero point for accumulating environmental features; constructing a multidimensional environmental information vector based on the physiological zero point and the environmental information after the physiological zero point; preprocessing the image information and using a visual backbone network to extract visual features from the preprocessed image information to obtain image features; performing standardization processing on the multidimensional environmental information vector and according to... The environmental information is divided into three groups based on its physiological meaning: cumulative light and temperature resource features, recent light and temperature driven features, and environmental fluctuation stress features. The cumulative light and temperature resource features and recent light and temperature driven features are mapped to the same feature dimension as the image features, allowing them to interact across attention levels to obtain global features of the environment-corrected image. The environmental fluctuation stress features are then used to perform feature-level linear modulation, residual feature-level linear modulation, or gated modulation on the global features of the environment-corrected image to obtain modulated fused features. These modulated fused features are then input into a regression prediction network to output predicted values ​​for rice growth status.

[0007] Further, mapping the cumulative light temperature resource feature group and the recent light temperature driven feature group to the same feature dimension as the image features, enabling them to interact across attention with the image features to obtain global features of the environment-corrected image, includes: mapping the cumulative light temperature resource feature group and the recent light temperature driven feature group to the same feature dimension as the image spatial features or global image features through a fully connected layer to obtain environment embedding features that can interact across attention with the image spatial features or global image features; calculating the attention response of the image spatial features or global image features to environmental information based on the environment embedding features to obtain environment-corrected image features after cross-attention fusion; and globally converging the environment-corrected image features to obtain global features of the environment-corrected image.

[0008] Furthermore, the global features of the environmental correction image are subjected to feature-level linear modulation using the environmental fluctuation stress feature set, and the corresponding formula is:

[0009] ,

[0010] in, Indicates scaling and modulation parameters; Indicates the offset modulation parameter; This represents the fused features after feature-level linear modulation; This indicates global features of the environment-corrected image; This indicates element-wise multiplication.

[0011] Furthermore, the residual feature level linear modulation is performed on the global features of the environmental correction image using the environmental fluctuation stress feature set, and the corresponding formula is:

[0012] ,

[0013] In the formula, This represents the fusion characteristics after linear modulation of the residual feature level; This represents the residual modulation intensity coefficient.

[0014] Furthermore, the visual backbone network uses a convolutional neural network or a visual Transformer network to extract features from the image information to obtain image features; the image features include image spatial features and global image features, wherein the image spatial features are used to characterize the rice canopy structure, color, texture, leaf density and panicle morphology, and the global image features are used to characterize the overall growth status of the rice field.

[0015] Furthermore, the multidimensional environmental information vector includes cumulative effective accumulated temperature, cumulative shortwave radiation, effective accumulated temperature increment in the first recent period, effective accumulated temperature increment in the second recent period, shortwave radiation in the first recent period, shortwave radiation in the second recent period, standard deviation of effective accumulated temperature in the second recent period, standard deviation of shortwave radiation in the second recent period, standard deviation of temperature in the second recent period, and average daily variation in the first recent period.

[0016] Furthermore, the multidimensional environmental information vector is divided into three environmental sub-feature groups according to the physiological meaning of the environmental information. The cumulative light and temperature resource feature group includes the cumulative effective accumulated temperature and cumulative shortwave radiation, which are used to characterize the cumulative light and temperature resources of rice since the beginning of tillering. The recent light and temperature driving feature group includes the effective accumulated temperature increment in the first recent period, the effective accumulated temperature increment in the second recent period, the shortwave radiation in the first recent period, and the shortwave radiation in the second recent period, which are used to characterize the recent light and temperature growth rate and the short-term growth driving intensity. The environmental fluctuation stress feature group includes the standard deviation of the effective accumulated temperature in the second recent period, the standard deviation of the shortwave radiation in the second recent period, the standard deviation of the temperature in the second recent period, and the average daily variation in the first recent period, which are used to characterize the recent environmental fluctuations, stress intensity, and diurnal temperature variation characteristics.

[0017] Furthermore, the regression prediction network is trained based on the Huber loss function, as shown in the following formula:

[0018]

[0019] In the formula, Let Huber's loss function be used. For real labels, The predicted values ​​of the regression prediction network, This is a preset threshold.

[0020] According to one aspect of the present invention, a rice growth status monitoring system based on image and environmental information fusion is provided, for implementing the aforementioned rice growth status monitoring method based on image and environmental information fusion, comprising: an image and environmental information acquisition module, used to acquire image information and environmental information during the rice growth process to obtain a monitoring sequence; a physiological zero-point determination module, used to determine the date when the rice is in the tillering stage in the monitoring sequence, and use it as the physiological zero point for environmental feature accumulation; a multi-dimensional environmental information vector construction module, used to construct a multi-dimensional environmental information vector based on the physiological zero point and environmental information after the physiological zero point; a visual feature extraction module, used to preprocess the image information and use a visual backbone network to extract visual features from the preprocessed image information to obtain image features; and an environmental information acquisition module. The environmental feature standardization and grouping module is used to perform standardization processing on the multidimensional environmental information vector and split it into cumulative light and temperature resource feature group, recent light and temperature driven feature group, and environmental fluctuation stress feature group according to the physiological meaning of the environmental information. The cross-attention fusion module is used to map the cumulative light and temperature resource feature group and the recent light and temperature driven feature group to the same feature dimension as the image features, so that they can perform cross-attention interaction with the image features to obtain the global features of the environmentally corrected image. The modulation module is used to use the environmental fluctuation stress feature group to perform feature-level linear modulation, residual feature-level linear modulation, or gated modulation on the global features of the environmentally corrected image to obtain the modulated fused features. The regression prediction module is used to input the modulated fused features into the regression prediction network and output the predicted value of rice growth status.

[0021] According to one aspect of the present invention, a non-transitory computer-readable storage medium is provided, characterized in that the non-transitory computer-readable storage medium stores computer instructions that cause the computer to execute the rice growth status monitoring method based on image and environmental information fusion.

[0022] The above technical solution involves collecting field images and environmental data of rice during its growth process using agricultural monitoring poles; using the date when rice is in the tillering stage as the physiological zero point; constructing a multidimensional environmental information vector based on the physiological zero point and diurnal temperature and shortwave radiation data after the physiological zero point; preprocessing the image information collected by the monitoring poles and extracting image features using a visual backbone network; standardizing the multidimensional environmental information vector and dividing it into a cumulative light and temperature resource feature group, a recent light and temperature driven feature group, and an environmental fluctuation stress feature group according to physiological semantics; using the cumulative light and temperature resource feature group and the recent light and temperature driven feature group to perform cross-attention fusion with the image features, and using the environmental fluctuation stress feature group to perform FiLM modulation, residual FiLM modulation, or gated modulation on the fused features to obtain fused features; finally, inputting the fused features into a regression prediction network to output predicted values ​​of rice growth status such as growth stage, vegetation area index, leaf weight, stem weight, panicle weight, or aboveground biomass.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0024] (1) The present invention can monitor the growth status of rice by using only the image information and environmental information (such as accumulated temperature and radiation) provided by the monitoring rod, thus avoiding dependence on field management information and a large amount of manual parameter adjustment.

[0025] (2) This invention divides environmental information into three groups of environmental features according to physiological semantics, which improves the interpretability of environmental information participating in model fusion.

[0026] (3) The present invention achieves deep fusion of image information and environmental information through cross-attention mechanism.

[0027] (4) This invention utilizes environmental fluctuation stress feature groups to perform FiLM, residual FiLM modulation or gated modulation on image-environment fusion features, thereby improving the model's adaptability to abnormal environments and complex field conditions.

[0028] (5) This invention integrates image features and environmental information (effective accumulated temperature, shortwave radiation, etc.), and achieves interaction through environment modulation, thereby improving the prediction accuracy and robustness of the model and providing a more comprehensive description of rice growth.

[0029] (6) The present invention can realize real-time monitoring of rice growth status without relying on destructive sampling, which reduces the dependence of traditional methods on sample collection and reduces interference with crop growth.

[0030] (7) This invention significantly improves the generalization ability across fields by fusing multi-source features and modulating environmental information, and can adapt to rice growth monitoring under different fields, different cultivation methods and different climatic conditions. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a flowchart illustrating a rice monitoring method based on the fusion of field monitoring pole images and environmental information, provided as an embodiment of the present invention.

[0033] Figure 2 This is a schematic diagram illustrating the improvement in the accuracy of fertility prediction provided by an embodiment of the present invention.

[0034] Figure 3 This is a schematic diagram illustrating the improvement in the accuracy of vegetation area index prediction provided in an embodiment of the present invention.

[0035] Figure 4 This is a schematic diagram illustrating the improvement in leaf quality prediction accuracy provided in an embodiment of the present invention.

[0036] Figure 5 This is a schematic diagram illustrating the improvement in stem quality prediction accuracy provided in an embodiment of the present invention.

[0037] Figure 6 This is a schematic diagram illustrating the improvement in the accuracy of ear quality prediction provided in an embodiment of the present invention.

[0038] Figure 7 This is a schematic diagram illustrating the improvement in biomass prediction accuracy provided in an embodiment of the present invention.

[0039] Figure 8 This is a schematic diagram illustrating the continuous improvement in the accuracy of fertility prediction provided in an embodiment of the present invention.

[0040] Figure 9 This is a schematic diagram illustrating the continuous improvement in the accuracy of vegetation area index prediction provided in an embodiment of the present invention.

[0041] Figure 10 This is a schematic diagram illustrating the continuous improvement of leaf quality prediction accuracy provided in an embodiment of the present invention.

[0042] Figure 11 This is a schematic diagram illustrating the continuous improvement in the accuracy of stem quality prediction provided in an embodiment of the present invention.

[0043] Figure 12 This is a schematic diagram illustrating the continuous improvement in the accuracy of ear quality prediction provided in an embodiment of the present invention.

[0044] Figure 13 This is a schematic diagram illustrating the continuous improvement in biomass prediction accuracy provided in an embodiment of the present invention. Detailed Implementation

[0045] It should be noted that:

[0046] The terms “comprising” and “having”, and any variations thereof, in the specification, claims, and accompanying drawings of this invention are intended to cover a non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0047] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices. The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be decomposed, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form new technical solutions. Such combinations are not bound by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0049] like Figure 1 As shown, this invention provides a rice monitoring method based on image and environmental information fusion, which can integrate meteorological priors and visual observations, and perform refined monitoring of regional-scale rice with robustness to scale differences and image quality fluctuations. Specifically, it includes the following steps:

[0050] Step S1: Collect image and environmental information during the rice growth process to obtain a monitoring sequence.

[0051] In step S1, agricultural monitoring poles are used to synchronously collect image and environmental information on the rice growth process in the field on an hourly (i.e., daily) scale to obtain a monitoring sequence (equivalent to an image-environment information dataset). The environmental information includes at least hourly air temperature and shortwave radiation.

[0052] In this embodiment, a plain was selected as the experimental site, and 170 agricultural monitoring poles were deployed. These monitoring poles are equipped with high-definition cameras and meteorological sensors, enabling them to collect daily-scale image information and daily-scale meteorological information of the rice paddies in real time.

[0053] Step S2: Determine the date when the rice is in the tillering stage in the monitoring sequence, and use it as the physiological zero point for the accumulation of environmental characteristics.

[0054] Step S2 aims to determine the physiological zero point. Based on field records, manual surveys, or model estimations, the date when rice is in the tillering stage is designated as the physiological zero point, denoted as d0. Starting from this physiological zero point, environmental variables are accumulated and statistically analyzed. For example, hourly air temperature and shortwave radiation data for d0 and afterwards are cleaned and aggregated to obtain the daily effective accumulated temperature GDD(d) and daily radiation Rad(d). Furthermore, corresponding cumulative environmental variables before d0 are set to 0 or excluded from accumulation to avoid interference from the ineffective growth stage in the early planting period on the model's judgment.

[0055] Step S3: Construct a multi-dimensional environmental information vector based on the physiological zero point and the environmental information after the physiological zero point.

[0056] In step S3, the multidimensional environmental information vector includes cumulative effective accumulated temperature, cumulative shortwave radiation, effective accumulated temperature increment in the first recent period, effective accumulated temperature increment in the second recent period, shortwave radiation in the first recent period, shortwave radiation in the second recent period, standard deviation of effective accumulated temperature in the second recent period, standard deviation of shortwave radiation in the second recent period, standard deviation of temperature in the second recent period, and average daily variation in the first recent period.

[0057] In this embodiment, the first recent period is the past 3 days, and the second recent period is the past 7 days. Based on the physiological zero point and diurnal meteorological data after the physiological zero point, the cumulative effective accumulated temperature, cumulative shortwave radiation, effective accumulated temperature increment in the past 3 days, effective accumulated temperature increment in the past 7 days, shortwave radiation in the past 3 days, shortwave radiation in the past 7 days, standard deviation of effective accumulated temperature in the past 7 days, standard deviation of shortwave radiation in the past 7 days, standard deviation of temperature in the past 7 days, and average diurnal temperature variation in the past 3 days are calculated to construct a 10-dimensional environmental information vector, represented as follows:

[0058]

[0059] (1)

[0060] in, This represents a 10-dimensional environmental information vector from the physiological zero point d0 to the image acquisition date D. Indicates the cumulative effective accumulated temperature. Indicates cumulative shortwave radiation. This indicates the effective accumulated temperature increase over the past 3 days. This indicates the effective accumulated temperature increase over the past 7 days. This indicates the amount of shortwave radiation over the past 3 days. This indicates the amount of shortwave radiation over the past 7 days. This represents the standard deviation of the effective accumulated temperature over the past 7 days. This represents the standard deviation of shortwave radiation over the past 7 days. This represents the standard deviation of temperature over the past 7 days. This indicates the average daily range over the past 3 days.

[0061]

[0062]

[0063] in, This represents the effective accumulated temperature on day d. This indicates the temperature at hour h on day d; This represents the base temperature for accumulated temperature calculation; This indicates the operation of retrieving the maximum value.

[0064]

[0065] in, This represents the cumulative shortwave radiation on day d. This represents the shortwave radiation at hour h on day d.

[0066]

[0067]

[0068] in, GDD represents the cumulative effective accumulated temperature from physiological zero point d0 to the image acquisition date D; d This represents the effective accumulated temperature on day d. Represents the cumulative shortwave radiation from physiological zero point d0 to the image acquisition date D; RAD d This represents the daily shortwave radiation on day d.

[0069]

[0070]

[0071]

[0072]

[0073] in, This represents the effective accumulated temperature increment within the last 3 days prior to the image acquisition date D; This represents the effective accumulated temperature increment within the seven days prior to the image acquisition date D; This indicates the shortwave radiation within the last 3 days prior to the image acquisition date D; This indicates the shortwave radiation within the last 7 days prior to the image acquisition date D; This indicates that the image acquisition date (D) is approximately 3 days prior to the acquisition date. This indicates that the image was acquired approximately 7 days prior to date D.

[0074]

[0075]

[0076]

[0077]

[0078]

[0079] In the formula, This represents the standard deviation of the effective accumulated temperature over the seven days preceding the image acquisition date D. This represents the average effective accumulated temperature over the seven days preceding the image acquisition date D; This represents the standard deviation of shortwave radiation over the seven days preceding the image acquisition date D. This represents the average shortwave radiation over the seven days preceding the image acquisition date D; This represents the standard deviation of temperature over the seven days preceding the image acquisition date D. This represents the daily average temperature on day d. This indicates the average temperature over the past 7 days; This indicates the daily range on day d; , This indicates the highest and lowest temperatures on day d. This represents the average daily variation over the three days preceding the image acquisition date (DDD).

[0080] Step S4: Preprocess the image information and use a visual backbone network to extract visual features from the preprocessed image information to obtain image features.

[0081] Step S4 aims to perform image preprocessing and visual feature extraction. Image preprocessing includes, but is not limited to, size normalization, standardization, and data augmentation. The visual backbone network is a convolutional neural network or a visual Transformer network. The visual backbone network is used to extract visual features from the image information collected by the monitoring pole, resulting in image features. Image features include spatial features and global image features. Spatial features are used to characterize the rice canopy structure, color, texture, leaf density, and panicle morphology, while global image features are used to characterize the overall growth status of the rice field.

[0082] In this embodiment, the image information is scaled to 256. After 256 pixels, the center is cropped to 224. The 224-pixel portion is normalized according to the mean and standard deviation of the training set images. A convolutional neural network convnextv2_tiny is selected as the backbone network, outputting image spatial features and global image features; the dimension of the image spatial feature pairs is 7. 7 768.

[0083] Step S5: Perform standardization processing on the multidimensional environmental information vector, and split it into cumulative light and temperature resource feature group, recent light and temperature driven feature group, and environmental fluctuation stress feature group according to the physiological meaning of the environmental information.

[0084] Step S5 aims to standardize and group environmental features. Specifically, the multidimensional environmental information vector is divided into three environmental sub-feature groups according to the physiological meaning of the environmental information: the cumulative light and temperature resource feature group includes the cumulative effective accumulated temperature and cumulative shortwave radiation, used to characterize the cumulative light and temperature resources of rice since the beginning of tillering; the recent light and temperature driving feature group includes the effective accumulated temperature increment in the first recent period, the effective accumulated temperature increment in the second recent period, the shortwave radiation in the first recent period, and the shortwave radiation in the second recent period, used to characterize the recent light and temperature growth rate and short-term growth driving intensity; the environmental fluctuation stress feature group includes the standard deviation of the effective accumulated temperature in the second recent period, the standard deviation of the shortwave radiation in the second recent period, the standard deviation of the temperature in the second recent period, and the average daily variation in the first recent period, used to characterize the recent environmental fluctuations, stress intensity, and diurnal temperature variation characteristics.

[0085] In this embodiment, the 10-dimensional environmental information is divided into three environmental sub-feature groups according to the physiological meaning of the environmental information. The cumulative light and temperature resource feature group token1 includes the cumulative effective accumulated temperature. and cumulative shortwave radiation The recent photothermal driven feature group token2 includes the effective accumulated temperature increment within the past 3 days. Effective accumulated temperature increase within the past 7 days Shortwave radiation levels in the past 3 days Shortwave radiation levels in the past 7 days ; Environmental fluctuation stress characteristic group token3, including the standard deviation of effective accumulated temperature over the past 7 days. Standard deviation of shortwave radiation over the past 7 days Temperature standard deviation over the past 7 days The average daily range over the past 3 days .

[0086] Step S6: Map the cumulative light temperature resource feature group and the recent light temperature driven feature to the same feature dimension as the image feature, so that they can interact across attention with the image feature to obtain the global feature of the environment-corrected image.

[0087] In step S6, the cumulative light-temperature resource feature group token1 and the recent light-temperature driven feature group token2 are mapped to the same feature dimension as the image spatial features or global image features through a fully connected layer, resulting in environment embedding features that can interact with the image spatial features or global image features across attention. Based on the environment embedding features, the attention response of the image spatial features or global image features to environmental information is calculated, resulting in environment-corrected image features after cross-attention fusion. The environment-corrected image features are then globally aggregated to obtain global environment-corrected image features. It should be noted that the steps for cross-attention interaction between environment embedding features and image spatial features, and between environment embedding features and global image features, are the same. The following description uses cross-attention interaction between environment embedding features and image spatial features as an example:

[0088] Step S61: Map the cumulative light-temperature resource feature group token1 and the recent light-temperature driven feature group token2 through a fully connected layer to the same feature dimension as the image spatial features, to obtain the environment embedding feature Z that can interact across attention with the image spatial features. i .

[0089]

[0090] In the formula, Z i This represents the environment embedding feature after mapping the i-th environment sub-feature group; Represents an environmental feature mapping network; T i W represents the i-th environmental sub-feature group; i and b i σ represents the weight matrix and bias of the fully connected layer, respectively; σ represents the nonlinear activation function.

[0091] Step S62: Based on the image spatial features and environment embedding features, calculate the attention response of the image spatial features to the environment information to obtain the environment-corrected image features after cross-attention fusion.

[0092] Step S62 introduces a cross-attention fusion mechanism, including: Let the image spatial features output by the visual backbone network be F1, which contains image features at M spatial locations, each with a feature dimension of C. The image spatial features are used as the query matrix Query, and the environment embedding features Z are... 12 Using the key matrix (Key) and value matrix (Value), we calculate the attentional response of image spatial features to environmental information. The formula is as follows:

[0093]

[0094] In the formula, Q represents the query matrix generated from image spatial features; K and V represent the key matrix and value matrix generated from environment embedding features, respectively; W Q W K W V Represent the learnable linear mapping matrix; F1 represents the image spatial features; Z represents the image spatial features. i This represents the contextual embedding features involved in cross-attention.

[0095]

[0096] In the formula, A represents the attention weight matrix between image spatial features and environment embedding features; d represents the attention feature dimension; Used for scaling of dot product attention. This represents the transpose of the key matrix.

[0097]

[0098]

[0099] In the formula, Fatt represents the environmentally perceived image features obtained by weighting environmental information, which reflects the degree of response of different image regions to cumulative light and temperature resources and recent light and temperature driving conditions; Fxattn represents the environmentally corrected image features after cross-attention fusion; W O LN represents the output mapping matrix; F1 represents the layer normalization; F1 is the residual term used to preserve the spatial phenotypic information of the original image.

[0100] Step S63: Globally converge the environmental correction image features to obtain the global features of the environmental correction image.

[0101] In step S63, the environment-corrected image features after cross-attention fusion are globally converged to obtain the global features of the environment-corrected image. These global features simultaneously include image phenotypic information, cumulative light temperature information, and recent light temperature-driven information. The calculation formula is as follows:

[0102]

[0103] In the formula, This represents the global features of the environment-corrected image after cross-attention fusion; This indicates average pooling, max pooling, or global pooling operations.

[0104] In this embodiment, the cumulative light-temperature resource feature group token1 and the recent light-temperature driven feature group token2 are converted into d-dimensional environment embedding features via an environment feature mapping network, and cross-attention is performed with image features to obtain environment-corrected global image features. In a preferred embodiment, the cross-attention module adopts multi-head attention with 8 attention heads, an attention temperature of 1.0, and enables bidirectional collaborative attention from image to environment and from environment to image.

[0105] Step S7: Use the environmental fluctuation stress feature group to perform feature-level linear modulation, residual feature-level linear modulation, or gated modulation on the global features of the environmental correction image to obtain the modulated fused features.

[0106] In step S7, the environmental fluctuation stress feature group token3 is mapped to feature-level linear modulation conditions (i.e., FiLM modulation conditions), residual feature-level linear modulation conditions (i.e., residual FiLM modulation conditions), or gated modulation conditions. The global features of the environmental correction image are scaled, offset, or gated to obtain the modulated fused features.

[0107] ① The environmental fluctuation stress feature group token3 is mapped to FiLM modulation conditions, which are used to scale and offset the global features of the environmental correction image according to recent environmental fluctuations.

[0108]

[0109] In the formula, This represents the environmental fluctuation embedding feature after the token3 mapping of the environmental fluctuation stress feature group; token3 represents the characteristic group of environmental fluctuation stress; This represents a group of characteristics related to environmental fluctuation stress. and These represent the weight matrix and bias of the mapping layer for the environmental fluctuation stress feature group token3, respectively. This represents a non-linear activation function.

[0110]

[0111] In the formula, This represents the scaling and modulation parameters generated by the environmental fluctuation stress feature group token3; This represents the offset modulation parameters generated by the environmental fluctuation stress feature group token3; This indicates the fusion characteristics after FiLM modulation; This represents the global features of the environment-corrected image obtained in step S6; This represents element-wise multiplication; Used for scaling and modulation while preserving the original image features; Used for offset modulation based on environmental fluctuation information.

[0112] ② Map the environmental fluctuation stress feature group token3 to residual FiLM modulation conditions to reduce the damage to image phenotypic information when environmental fluctuation features are too strong:

[0113]

[0114] In the formula, This indicates the fusion characteristics after residual FiLM modulation; Represents the residual modulation intensity coefficient; when When the size is small, the model mainly preserves image features; when When the stress is large, the model enhances the moderating effect of the environmental fluctuation stress feature group on the prediction results.

[0115] ③ Map the environmental fluctuation stress feature group token3 to a gating modulation condition, and perform gating modulation on the global features of the environmental correction image, including: mapping the environmental fluctuation stress feature group token3 to generate a gating vector g using a multilayer perceptron, and adjusting the fused features according to the following formula:

[0116]

[0117] in, This indicates the fusion characteristics after gating modulation. This is the modulation intensity coefficient.

[0118] In addition, to ensure stability in the early stages of training, the last layer of the modulation network can be initialized with zero, so that the initial state of the model is close to that of the image-only model, and then the environmental modulation effect is gradually learned during the training process.

[0119] Step S8: Input the modulated fusion features into the regression prediction network and output the predicted value of rice growth status.

[0120] Step S8 aims to perform regression prediction by inputting the fused features after FiLM modulation, residual FiLM modulation, or gated modulation into a regression prediction network (such as convolutional neural networks convnextv2, resnet34, etc.) to obtain the predicted value of rice growth status.

[0121] The regression prediction network consists of two fully connected layers, outputting normalized predicted values, which are then denormalized to obtain the growth index at the physical quantity scale. The regression prediction network is trained based on the Huber loss function, the formula of which is as follows:

[0122]

[0123] In the formula, Let Huber's loss function be used. This is a real label; These are the predicted values ​​from the regression prediction network; This is a preset threshold used to control the growth rate of the Huber loss function when there is a large error.

[0124] This invention can be used to predict indicators such as rice growth period, vegetation area index, leaf weight, stem weight, panicle weight, aboveground biomass, and leaf nitrogen content. The system can input monitoring rod images collected daily or hourly along with environmental information into a trained model to obtain continuous field-level monitoring results, which can then be further used for fertilizer and water management, pest and weed risk assessment, yield formation analysis, and regional growth comparison.

[0125] Furthermore, this embodiment compares traditional image regression and environmental information fusion methods. For example... Figures 2 to 13 As shown, Figures 2 to 7 This demonstrates the improved accuracy of rice growth status prediction after integrating environmental information (such as accumulated temperature and radiation). Figures 8 to 13 This diagram illustrates the continuous improvement of the invention in cross-field applications, showcasing the application of the method under different field conditions and its ability to continuously monitor rice growth. The invention demonstrates more reasonable temporal trends and higher prediction accuracy in predicting rice growth stages, vegetation area index, stem, leaf, and panicle quality, and aboveground biomass.

[0126] Based on the same technical concept as the aforementioned embodiments, this invention also provides a rice growth status monitoring system based on image and environmental information fusion, comprising: an image and environmental information acquisition module for acquiring image information and environmental information during the rice growth process to obtain a monitoring sequence; a physiological zero-point determination module for determining the date when the rice is in the tillering stage in the monitoring sequence and using it as the physiological zero point for environmental feature accumulation; a multi-dimensional environmental information vector construction module for constructing a multi-dimensional environmental information vector based on the physiological zero point and environmental information after the physiological zero point; a visual feature extraction module for preprocessing the image information and using a visual backbone network to extract visual features from the preprocessed image information to obtain image features; and an environmental feature standardization and grouping module. The system is used to perform standardization processing on the multidimensional environmental information vector and split it into a cumulative light and temperature resource feature group, a recent light and temperature driven feature group, and an environmental fluctuation stress feature group according to the physiological meaning of the environmental information; a cross-attention fusion module is used to map the cumulative light and temperature resource feature group and the recent light and temperature driven feature group to the same feature dimension as the image features, so that they can interact with the image features across attention to obtain the global features of the environmentally corrected image; a modulation module is used to use the environmental fluctuation stress feature group to perform feature-level linear modulation, residual feature-level linear modulation, or gated modulation on the global features of the environmentally corrected image to obtain the modulated fused features; a regression prediction module is used to input the modulated fused features into a regression prediction network and output the predicted value of rice growth status.

[0127] Based on the same technical concept as the foregoing embodiments, the present invention also provides a non-transitory computer-readable storage medium that stores computer instructions that cause the computer to execute the rice growth status monitoring method based on image and environmental information fusion.

[0128] In summary, this invention discloses a method for monitoring rice growth status based on image and environmental information fusion, belonging to the field of agricultural remote sensing and crop phenotypic monitoring technology. The method includes: collecting field image information and environmental data during the rice growth process using agricultural monitoring poles; using the date corresponding to the rice tillering stage as the physiological zero point; constructing a multidimensional environmental information vector based on the physiological zero point and diurnal temperature and shortwave radiation data after the physiological zero point; preprocessing the monitoring pole images and extracting image features using a visual backbone network; standardizing the multidimensional environmental information vector and dividing it according to physiological semantics into a cumulative light and temperature resource feature group, a recent light and temperature driven feature group, and an environmental fluctuation stress feature group; performing cross-attention fusion using the cumulative light and temperature resource feature group and the recent light and temperature driven feature group with image features, and using the environmental fluctuation stress feature group to perform FiLM modulation, residual FiLM modulation, or gated modulation on the fused features to obtain fused features; finally, inputting the fused features into a regression prediction network to output predicted values ​​of rice growth status such as growth stage, vegetation area index, leaf weight, stem weight, panicle weight, or aboveground biomass. This invention enables deep fusion of environmental and image information, improving the accuracy, robustness, and cross-field generalization ability of rice growth status monitoring.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A method for monitoring the growth status of rice based on image and environmental information fusion, characterized in that, include: To obtain image and environmental information during the rice growth process, a monitoring sequence is obtained. The date when rice is in the tillering stage in the monitoring sequence is determined and used as the physiological zero point for the accumulation of environmental characteristics; Based on the physiological zero point and the environmental information after the physiological zero point, a multi-dimensional environmental information vector is constructed. The image information is preprocessed, and a visual backbone network is used to extract visual features from the preprocessed image information to obtain image features. The multidimensional environmental information vector is standardized and then divided into a cumulative light and temperature resource feature group, a recent light and temperature driven feature group, and an environmental fluctuation stress feature group according to the physiological meaning of the environmental information. The cumulative light temperature resource feature group and the recent light temperature driven feature group are mapped to the same feature dimension as the image features, so that they can interact across attention with the image features to obtain the global features of the environment-corrected image. The environmental fluctuation stress feature group is used to perform feature-level linear modulation, residual feature-level linear modulation, or gated modulation on the global features of the environmental correction image to obtain the modulated fused features; The modulated fusion features are input into the regression prediction network, and the predicted value of rice growth status is output. The multidimensional environmental information vector includes cumulative effective accumulated temperature, cumulative shortwave radiation, effective accumulated temperature increment in the first recent period, effective accumulated temperature increment in the second recent period, shortwave radiation in the first recent period, shortwave radiation in the second recent period, standard deviation of effective accumulated temperature in the second recent period, standard deviation of shortwave radiation in the second recent period, standard deviation of temperature in the second recent period, and average daily variation in the first recent period. The multidimensional environmental information vector is divided into three environmental sub-feature groups according to the physiological meaning of the environmental information. The cumulative light and temperature resource feature group includes the cumulative effective accumulated temperature and cumulative shortwave radiation, which is used to characterize the cumulative light and temperature resources of rice since the beginning of tillering. The recent light and temperature driving feature group includes the effective accumulated temperature increment in the first recent period, the effective accumulated temperature increment in the second recent period, the shortwave radiation in the first recent period, and the shortwave radiation in the second recent period, which is used to characterize the recent light and temperature growth rate and the intensity of short-term growth driving. The environmental fluctuation stress feature group includes the standard deviation of the effective accumulated temperature in the second recent period, the standard deviation of the shortwave radiation in the second recent period, the standard deviation of the temperature in the second recent period, and the average daily variation in the first recent period, which is used to characterize the recent environmental fluctuations, stress intensity, and diurnal temperature variation characteristics.

2. The method for monitoring rice growth status based on image and environmental information fusion as described in claim 1, characterized in that, The cumulative light-temperature resource feature group and the recent light-temperature driven feature group are mapped to the same feature dimension as the image features, and then cross-attention interaction is performed between them to obtain the global features of the environment-corrected image, including: The cumulative light temperature resource feature group and the recent light temperature driven feature group are mapped to the same feature dimension as the image features through a fully connected layer to obtain an environment embedding feature that can interact across attention with image spatial features or global image features. Based on the environment embedding features, the attention response of the image spatial features or global image features to the environment information is calculated to obtain the environment-corrected image features after cross-attention fusion. The environmentally corrected image features are globally aggregated to obtain the global features of the environmentally corrected image.

3. The method for monitoring rice growth status based on image and environmental information fusion as described in claim 1, characterized in that, The global features of the environmental correction image are linearly modulated at the feature level using the environmental fluctuation stress feature set, and the corresponding formula is: ， in, Indicates scaling and modulation parameters; Indicates the offset modulation parameter; This represents the fused features after feature-level linear modulation; This indicates global features of the environment-corrected image; This indicates element-wise multiplication.

4. The method for monitoring rice growth status based on image and environmental information fusion as described in claim 3, characterized in that, The residual feature level linear modulation is performed on the global features of the environmental correction image using the environmental fluctuation stress feature set, and the corresponding formula is: ， In the formula, This represents the fusion characteristics after linear modulation of the residual feature level; This represents the residual modulation intensity coefficient.

5. The method for monitoring rice growth status based on image and environmental information fusion as described in claim 1, characterized in that, The visual backbone network uses a convolutional neural network or a visual Transformer network to extract features from image information to obtain image features. The image features include image spatial features and global image features. The image spatial features are used to characterize the rice canopy structure, color, texture, leaf density, and panicle morphology, while the global image features are used to characterize the overall growth status of the rice field.

6. The method for monitoring rice growth status based on image and environmental information fusion as described in claim 1, characterized in that, The regression prediction network is trained based on the Huber loss function, as shown in the following formula: In the formula, Let Huber's loss function be used. For real labels, The predicted values ​​of the regression prediction network, This is a preset threshold.

7. A rice growth status monitoring system based on image and environmental information fusion, characterized in that, A method for monitoring rice growth status based on image and environmental information fusion as described in any one of claims 1 to 6, comprising: The image and environmental information acquisition module is used to acquire image and environmental information during the rice growth process to obtain a monitoring sequence; The physiological zero-point determination module is used to determine the date when rice is in the tillering stage in the monitoring sequence and to use it as the physiological zero point for the accumulation of environmental characteristics; The multidimensional environmental information vector construction module is used to construct multidimensional environmental information vectors based on environmental information at and after the physiological zero point. The visual feature extraction module is used to preprocess the image information and use a visual backbone network to extract visual features from the preprocessed image information to obtain image features. The environmental feature standardization and grouping module is used to perform standardization processing on the multidimensional environmental information vector and split it into cumulative light and temperature resource feature group, recent light and temperature driven feature group and environmental fluctuation stress feature group according to the physiological meaning of the environmental information; The cross-attention fusion module is used to map the cumulative light temperature resource feature group and the recent light temperature driven feature group to the same feature dimension as the image features, so that they can interact with the image features across attention to obtain the global features of the environment-corrected image. The modulation module is used to perform feature-level linear modulation, residual feature-level linear modulation, or gated modulation on the global features of the environmental correction image using the environmental fluctuation stress feature group, so as to obtain the modulated fused features. The regression prediction module is used to input the modulated fused features into the regression prediction network and output the predicted value of rice growth status.

8. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that cause the computer to execute the rice growth status monitoring method based on image and environmental information fusion as described in any one of claims 1 to 6.

Images