Field crop phenotype and environment real-time monitoring system based on multi-sensor fusion

Abstract

The invention relates to the technical field of agricultural information, and discloses a field crop phenotype and environment real-time monitoring system based on multi-sensor fusion, which comprises a data acquisition module, a data processing module and a result generation module. The method comprises the following steps: acquiring field real-time multi-sensor data; the data is processed to construct a structured semantic graph, and the construction process comprises the steps that primitive features are extracted from the original data to serve as graph nodes, and the internal incidence relation between the features is analyzed to define the edges of the graph; and finally, based on the structured semantic graph, adopting a graph neural network model, iteratively updating a node state through a multilayer information propagation mechanism, and finally identifying a crop phenotype and an environment state. According to the method, the discrete data is converted into the structured semantic graph rich in context association, and deep analysis is performed by using the graph neural network, so that the deep association among the data can be fully mined, and the accuracy and comprehensiveness of field monitoring are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of agricultural information technology, specifically to a real-time monitoring system for crop phenotypes and the environment in the field based on multi-sensor fusion. Background Technology

[0002] In modern precision agriculture, to achieve refined management of crop growth and environmental conditions, it is typically necessary to deploy diverse sensors in the field for data collection. These sensors, such as visible light cameras, multispectral imagers, and lidar mounted on drones or ground robots, can capture crop morphological and physiological information at both macroscopic and microscopic scales. Meanwhile, soil and weather sensors permanently deployed in the field provide continuous environmental parameters. By comprehensively analyzing this multi-source, heterogeneous data, precise decision support can be provided for agricultural activities such as irrigation, fertilization, and pest and disease control.

[0003] However, effectively integrating these heterogeneous data from different sources, in different formats, and at different spatiotemporal scales remains a core challenge for current technology. Existing data processing methods typically extract features from each individual sensor dataset, then simply concatenate and combine these features before feeding them into traditional machine learning models for analysis. While this approach achieves data aggregation to some extent, it fundamentally ignores the deep-seated intrinsic relationships between different data features. For example, the spectral information of crop leaves, the three-dimensional structural information of the plant, and the moisture information of the soil around the roots are closely linked physically and physiologically. Simple feature concatenation cannot effectively express this structured contextual relationship, resulting in a shallow level of information fusion and difficulty in accurately identifying complex crop stress states caused by the combined effects of multiple factors.

[0004] To enhance analytical capabilities, some solutions have turned to deep learning models, such as using convolutional neural networks to process image data and recurrent neural networks to process time-series data, then fusing the outputs of different models in a later stage. While these methods improve feature extraction capabilities for single data modalities, data fusion often occurs at the end of feature extraction, leaving different data streams largely independent in the processing flow. This "late-stage fusion" strategy also fails to fully explore and utilize the inherent correlations present in the data at the initial collection stage, such as the interactions between different physical quantities at specific spatial locations. Therefore, existing technologies lack an effective way to uniformly model the inherent structural relationships within the data when representing and utilizing the complex system of crops and the environment in the field. This limits the accuracy and reliability of monitoring systems in identifying early, non-significant crop problems. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a real-time monitoring system for crop phenotypes and the environment in the field based on multi-sensor fusion. It aims to solve the technical problem that existing technologies fail to effectively model and utilize the deep-seated structural relationships between data when fusing multi-source heterogeneous sensor data in the field, resulting in insufficient accuracy and reliability of crop status monitoring.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a real-time monitoring system for field crop phenotypes and the environment based on multi-sensor fusion. The system includes a data acquisition module, a data processing module, and a result generation module. The data acquisition module acquires real-time multi-sensor data from the field. The data processing module, connected to the data acquisition module, preprocesses the multi-sensor data to construct a structured semantic graph, and uses a graph neural network model to identify crop phenotypes and environmental states based on the structured semantic graph. The result generation module, connected to the data processing module, generates a monitoring report based on the results of the crop phenotype and environmental state identification.

[0007] One of the core innovations of this invention lies in the unique processing flow executed by the data processing module. Instead of directly inputting raw sensor data into the recognition model, it first uses a preprocessing step to transform the discrete, heterogeneous sensor data into a structured semantic graph that explicitly expresses the inherent relationships between the data. This graph-structured data provides richer and more effective input information for subsequent accurate recognition.

[0008] Specifically, when constructing the structured semantic graph, the data processing module first extracts primitive features from the multi-sensor data. These primitive features are the basic information units that constitute complex field scenes, such as crop canopy areas and leaf outlines extracted from image data, or temperature and humidity abrupt changes extracted from sensor data. Subsequently, the inherent relationships between these primitive features are analyzed, such as spatial proximity or overlap, temporal sequence, or physical causal relationships, and the structured semantic graph is constructed based on these relationships.

[0009] In one implementation, the structured semantic graph includes a set of nodes and a set of edges. Each node corresponds to a primitive feature, while each edge corresponds to an intrinsic relationship between two primitive features. To enrich the semantic information of the graph, the data processing module also labels the nodes and edges in the graph with initial attribute vectors. The initial attribute vector of a node can characterize the detailed attributes of its corresponding primitive feature, and the initial attribute vector of an edge can characterize the type or strength of its corresponding relationship.

[0010] After constructing the structured semantic graph, another core innovation of this invention lies in employing a graph neural network model particularly suitable for processing graph-structured data for subsequent recognition tasks. Specifically, the data processing module iteratively updates the hidden state vectors of each node in the graph through multi-layer information propagation. In each layer of propagation, a node aggregates information from its neighboring nodes to update its own state, thereby enabling the final state vector of each node to contain its local and even global structural information within the graph.

[0011] In one specific implementation, the iterative update process includes two steps: information aggregation and state update. The information aggregation step aims to gather information from the neighborhood of each node, and its process can be described by the following formula: ; in, Indicates the current propagation layer number. For the current node, For nodes The set of neighboring nodes, For neighboring nodes, For neighboring nodes In the The hidden state vector of the layer, For connecting nodes and The edge at the 1st The state vector of the layer, For aggregation functions that are invariant to substitutions, such as summation or mean calculation, This is a message function used to merge information from nodes and edges.

[0012] Following the information aggregation step, a state update step is performed. This step combines the information from the previous layer of the node with the aggregated neighborhood information to generate a new state for the current layer. This process can be described by the following formula: ; in, For nodes In the The hidden state vector of the layer, The aggregated information vector is calculated using the above formula. For updating functions, such as a nonlinear transformation function.

[0013] After multiple layers of information propagation, the data processing module uses a readout function to integrate the final state vectors of all nodes in the graph to generate a graph-level representation vector that can characterize the global information of the entire graph. Finally, based on the graph-level representation vector, a classifier calculates the predicted probability of the target category for different crop phenotypes or environmental states, thereby completing the identification task.

[0014] Optionally, the multi-sensor data includes at least one selected from: visible light image data, multispectral image data, hyperspectral image data, lidar point cloud data, temperature sensor data, and humidity sensor data. Optionally, the crop phenotype includes at least one selected from: crop plant height, leaf area index, canopy coverage, pest and disease stress status, and nutrient stress status.

[0015] This invention employs a technical solution that first constructs a structured semantic graph and then utilizes a graph neural network for recognition. This approach fully explores and utilizes the deep-seated relationships between multi-source heterogeneous sensor data, transforming isolated data points into a whole with rich contextual information for analysis. This significantly improves the accuracy and comprehensiveness of real-time monitoring of crop phenotypes and environmental conditions in the field.

[0016] This invention provides a real-time monitoring system for crop phenotypes and the environment in the field based on multi-sensor fusion. It has the following beneficial effects: 1. This invention achieves a deeper level of fusion of heterogeneous information by transforming raw multi-sensor data into a structured semantic graph. Unlike the simple feature stitching or post-decision fusion in existing technologies, this scheme explicitly models the spatial, temporal, and cross-modal physical relationships between data by using extracted primitive features as nodes of the graph and defining the inherent relationships between features as edges. This structured data representation allows the system to view and analyze all input information holistically within a unified framework, greatly improving the breadth and depth of data fusion.

[0017] 2. This invention employs a graph neural network model to analyze the constructed structured semantic graph, significantly improving the accuracy and sensitivity of monitoring. Through its unique multi-layered information propagation mechanism, the graph neural network enables the final state vector of each node to aggregate rich contextual information within its multi-hop neighborhood. This means that when judging the state of a local feature, the model can simultaneously consider its surrounding environmental factors and structural information, thereby effectively identifying early crop stress states caused by the combined effects of multiple factors that are not significant in a single data dimension.

[0018] 3. The technical solution of this invention enhances the robustness of the monitoring system. In complex field environments, some sensor data are inevitably subject to noise interference or incomplete data. Because this invention organizes the data into an interconnected graph, when the quality of the feature information corresponding to a node in the graph is poor, the graph neural network model can still perform inference and compensation based on the information of its neighboring nodes and edges. This ability to use contextual information for error correction and completion makes the entire system insensitive to local data quality problems, thereby ensuring the stability and reliability of the final output results.

[0019] 4. The method proposed in this invention possesses excellent flexibility and scalability. When it is necessary to introduce new sensors or monitor new agricultural parameters, this solution does not require disruptive modifications to the core algorithm framework. New data sources only need to be defined as new types of nodes in the graph, and their relationships with other nodes can be established based on their physical meaning. This modular design based on graph theory enables the system to easily integrate future sensing technologies, adapt to diverse monitoring needs, and demonstrates strong technological compatibility and vitality.

[0020] 5. This invention provides a more interpretive analytical approach to understanding the complex interactions between crops and the environment. By analyzing the finally trained graph neural network model, the contribution of different nodes to specific identification results can be explored. This not only allows for the drawing of monitoring conclusions but also reveals, to some extent, which combinations of key factors led to these conclusions, providing deeper insights beyond simple classification results for subsequent agricultural decision-making and scientific research. Attached Figure Description

[0021] Figure 1 This is a system structure block diagram of the present invention; Figure 2 This is a flowchart of the monitoring method of the present invention; Figure 3 This is a schematic diagram of the level instrument structure of the present invention.

[0022] Among them, 1. Photosynthetically active radiation sensor; 2. Temperature and humidity sensor A; 3. Wind speed sensor; 4. Multi-mode camera; 5. Touch screen; 6. Microcomputer; 7. Near-surface temperature and humidity sensor; 8. PAR sensor; 9. Soil probe interface; 10. Level. Detailed Implementation

[0023] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Please see the appendix Figure 1-2 This invention provides a real-time monitoring system for crop phenotypes and environment in the field based on multi-sensor fusion. The system includes a data acquisition module, a data processing module, and a result generation module.

[0025] The data acquisition module collects real-time multi-sensor data from the field and outputs it to the data processing module. The data processing module processes and analyzes the received data, generates identification results of crop phenotypes and environmental states, and sends these results to the result generation module. Based on the identification results, the result generation module generates and outputs a final monitoring report.

[0026] Specifically, the data acquisition module is responsible for systematically collecting raw multi-sensor data reflecting crop growth status and environmental factors from the field monitoring environment. This module integrates sensor hardware with different characteristics deployed on different platforms to build a comprehensive and three-dimensional field information sensing network.

[0027] The multi-sensor data includes, but is not limited to, data collected by aerial platforms or ground equipment, as well as data collected by fixed sensors deployed in the field.

[0028] Aerial platforms, such as drones, can be equipped with visible light cameras to obtain high-resolution crop canopy texture and color information, i.e., visible light image data; multispectral or hyperspectral imagers to obtain crop reflectivity information under different electromagnetic bands, i.e., multispectral or hyperspectral image data, for in-depth analysis of the crop's physiological health status; and lidar to obtain crop three-dimensional point cloud data for precise extraction of structural parameters such as plant height and canopy width.

[0029] Fixed sensors are deployed at specific monitoring points in the field to continuously or periodically collect environmental factor data, such as soil temperature and humidity data collected by soil sensors distributed at different depths, and air temperature and humidity data collected by environmental sensors deployed above and below the canopy.

[0030] Through the collaborative work of the data acquisition modules, this system can acquire heterogeneous datasets covering crop internal and external physiological phenotypes, population structure, and the surrounding microenvironment. These raw data streams provide a rich and necessary information foundation for subsequent data processing modules to perform in-depth fusion analysis and accurate identification.

[0031] In one embodiment of the present invention, the data processing module receives raw multi-sensor data from the data acquisition module. Its core task is to transform and unify these data streams, which differ in form, dimension, and semantics, into a structured data format specifically designed for subsequent intelligent analysis, namely a structured semantic graph. This process is a key prerequisite for achieving high-precision fusion analysis and recognition.

[0032] A schematic diagram illustrating the structured semantic graph construction process according to an embodiment of the present invention. This construction process mainly includes primitive feature extraction, intrinsic relationship analysis, and graph construction and attribute annotation.

[0033] First, the data processing module processes the received multi-sensor data. Perform primitive feature extraction. This step aims to identify and separate information units with clear physical or phenotypic meaning from continuous or unstructured raw data. These units constitute the basic nodes of the subsequent graph structure. For example, for visible light or hyperspectral image data, a semantic segmentation network model can be used to accurately extract crop canopy regions, individual leaf outlines, or lesion areas caused by pests and diseases as primitive features. For lidar point cloud data, point cloud clustering and fitting algorithms are used to extract the three-dimensional skeleton points of the plant, canopy density distribution parameters, or plant height information as primitive features. For time-series data such as temperature and humidity from fixed sensors, sliding window analysis is used to extract statistics such as mean, variance, peak value, or rate of change within a specific time window as primitive features. The output of this step is a set containing all identified primitive features. .

[0034] In obtaining the set of primitive features The data processing module then analyzes the inherent relationships between these features to define edges in the graph structure. These relationships reveal contextual information between isolated features. For example, by analyzing the geospatial coordinates or pixel positions of each primitive feature in an image, spatial proximity or overlap can be determined. By comparing the timestamps of the data acquisition for each feature, synchronous or temporal relationships can be determined. Furthermore, cross-modal relationships can be defined; for instance, soil moisture characteristics of a specific plot can be correlated with the spectral characteristics of the crop canopy on that plot to characterize the potential physical connections between the environment and crop physiology.

[0035] Finally, based on the extracted primitive feature set and the analyzed intrinsic relationships, the data processing module formally constructs a structured semantic graph. This diagram can be formally represented as... , where the set of nodes Each node in Uniquely corresponds to a primitive feature Edge set Each edge in Represents a node and There are predefined intrinsic relationships between the corresponding primitive features.

[0036] To ensure the graph structure can be effectively processed by subsequent graph neural network models, the data processing module also needs to label the attributes of each node and each edge in the graph, that is, assign them initial attribute vectors. For each node... Its initial node attribute vector It is constructed as a real number vector that encodes its corresponding primitive feature. The quantification attribute. For example, if For the leaf area, then It can include the average reflectance of the region across multiple spectral bands, its area size, and parameters describing its shape. For each edge... Its initial edge attribute vector It is also constructed as a real-number vector to encode the type and strength of the association it represents. For example, if the edge represents a spatial proximity relationship, then... This can include the actual distance between two features or the intersection-union ratio of their bounding boxes. Through this step, the raw multi-sensor data is completely transformed into a structured semantic graph rich in numerical attributes and structural information, which can be directly used for machine learning.

[0037] In an embodiment of the present invention, the data processing module successfully constructs a structured semantic graph. Next, the identification stage of crop phenotype and environmental state begins. The core of this stage is to use a graph neural network model to perform deep graph learning and reasoning on the structured semantic graph.

[0038] Before information transmission begins, the model first maps the graph. Initial attribute vector of middle node Initial attribute vectors of edges Mapping this to the model's hidden state space yields the hidden state vectors of the nodes at layer 0. and edge hidden state vector This mapping can be achieved through a learnable linear transformation.

[0039] This graph neural network model executes... Information is propagated through layers to iteratively update the hidden state vector of each node in the graph. This represents the preset total number of propagation layers. In each layer, each node aggregates information from its neighboring nodes and related edges, and combines this with its state from the previous layer to calculate and generate its new state for the current layer. This process allows the node's final state to capture and merge its surrounding data. Rich structural and semantic information within the jump neighborhood.

[0040] Specifically, in the first layer( In information dissemination, for any node in the graph Its state update first performs an information aggregation step. This step aims to obtain information from the nodes. All neighboring nodes Information is collected and an aggregated information vector is generated. The process is described by the following formula:

[0041] in: Represents a node In the The aggregated information vector received by the layer; In the diagram In and nodes The set of directly connected neighbor nodes; Representing neighboring nodes In the The hidden state vector of the layer; Indicates the connection node and The edge at the 1st The state vector of a layer; in some implementations, this state can also be the initial state of the edge. The message function is used to fuse the states of neighboring nodes and the states of connecting edges to generate a message to be aggregated. This function can be implemented as a neural network layer, such as a multilayer perceptron. This represents an aggregation function that is permutation-invariant and is used to aggregate messages from all neighbors. This function can be implemented as a summation (Sum), mean (Mean), or maximum (Max) operation.

[0042] The aggregated information vector is calculated. Next, the data processing module executes the state update step. This step combines the node... its own previous state and the neighborhood information just aggregated To calculate nodes In the The final new state of the layer The update process is described by the following formula:

[0043] in: Represents a node In the The new hidden state vector after the layer update; Represents a node In the The hidden state vector of the layer; This represents the update function, which is responsible for effectively combining the node's own historical information with the newly aggregated neighborhood information. This function can be specifically implemented as the update gate of a gated recurrent unit, or a transformation function containing a nonlinear activation function.

[0044] go through After the information propagation iterations of the layers, each node in the graph Each obtains its final state vector. This vector has fully incorporated the structural and semantic information of its neighborhood. To obtain a single vector representation of the global state of the entire graph, the data processing module uses a readout function to integrate the final state vectors of all nodes in the graph to generate a graph-level representation vector. The readout function can be specifically implemented as global average pooling, global summation pooling, or attention-based pooling operations.

[0045] Subsequently, the resulting graph-level representation vector It is fed into a downstream classifier. This classifier, for example, a multilayer perceptron (MLP), is followed by a softmax activation function. The process is performed, and the predicted probability of the target category for different crop phenotypes and environmental states is finally calculated. This probability vector This is the result of the graph neural network model identifying the current field condition, which is then output to the result generation module.

[0046] In one embodiment of the present invention, the result generation module receives the predicted probability of the target category output from the data processing module. The predicted probability It is a vector, where the value of each dimension corresponds to the confidence level of a preset crop phenotype or environmental state.

[0047] Specifically, the function of the result generation module is to generate the predicted probability vector. The data is then parsed into a final, structured monitoring report. This process first determines the dimension with the largest value in the predicted probability vector; the target category corresponding to this dimension is the most likely crop phenotype or environmental state currently identified.

[0048] To ensure the reliability of the output, the result generation module also compares the maximum probability value with a preset confidence threshold. A comparison is then made. Only when the maximum probability value exceeds the stated confidence threshold... Only when the maximum probability value does the system confirm the validity of the identification and use it as the final result. If the maximum probability value does not exceed the threshold, the system determines that the current state is unclear or the identification confidence is insufficient, and does not generate a specific conclusion. This determination logic can be described by the following formula:

[0049] Where: Result represents the final output recognition result; This indicates obtaining the predicted probability vector. The index of the dimension with the largest median value, which corresponds to the specific target category; This indicates obtaining the predicted probability vector. The maximum value in; This indicates the preset confidence threshold.

[0050] This device consists of multiple sensors and processing units to achieve comprehensive monitoring and analysis of specific environmental parameters. The following is a detailed description of the functions of each major structural component: The data acquisition module includes: Photosynthetically active radiation sensor 1 is used to measure the photosynthetically active radiation at the top and is the direct source of data.

[0051] Temperature and humidity sensor A2 is used to measure temperature and humidity data at the top.

[0052] Wind speed sensor 3 is used to measure wind speed data.

[0053] The multi-mode camera 4 acquires color, depth, and thermal infrared data by capturing images.

[0054] The near-surface temperature and humidity sensor 7 is used to measure the temperature and humidity data of the near-surface layer.

[0055] PAR sensor 8, used to measure photosynthetically active radiation at the bottom, is a direct source of data.

[0056] Soil probe interface 9 is used to connect a soil probe (sensor) and is the channel for acquiring soil data, therefore it belongs to the data acquisition part.

[0057] The level 10 is used to ensure that the equipment is in a horizontal position to ensure the accuracy of the data acquired by other sensors. It is an auxiliary and prerequisite for data acquisition. The data processing module includes a microcomputer 6, which executes programs, processes all the data collected by the sensors, and performs calculations, analysis, and storage. The results generation module includes a touch screen 5 for displaying processed data, images, and a user interface. Users see the final results on the screen, which also has input functionality, but its role as an output is very clear.

[0058] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A real-time monitoring system for field crop phenotyping and environment based on multi-sensor fusion, characterized in that, The application relates to a field crop monitoring method and system. The method comprises the following steps: a data acquisition module is used to acquire real-time multi-sensor data in a field; a data processing module is connected with the data acquisition module and is used to pre-process the multi-sensor data to construct a structured semantic graph, and based on the structured semantic graph, a graph neural network model is used to identify crop phenotypes and environmental states; 2. The system of claim 1, wherein, a result generation module is connected with the data processing module and is used to generate a monitoring report according to the result of the identification of the crop phenotypes and the environmental states. The data processing module pre-processes the multi-sensor data to construct a structured semantic graph, specifically comprising the following steps:

3. The system of claim 2, wherein, element features are extracted from the multi-sensor data; internal correlation relationships between the element features are analyzed to construct the structured semantic graph. The structured semantic graph comprises a node set and an edge set; 4. The system of claim 1, wherein, the nodes correspond to the element features, and the edges correspond to the internal correlation relationships; 5. The system of claim 4, wherein, The iterative updating of the hidden state vectors of the nodes in the graph comprises an information aggregation step which computes an aggregated information vector by : ； in, This represents the current propagation layer. For the current node, For nodes in, This represents the current propagation layer. For the current node, For nodes The set of neighboring nodes, For neighboring nodes, Neighboring nodes In the The hidden state vector of the layer, For connecting nodes and The edge at the 1st The state vector of the layer, For aggregation functions that are invariant to substitution, This is the message function.

6. The system of claim 5, wherein, The iterative updating of the hidden state vector of each node in the graph, after the information aggregation step, also comprises a state updating step, which computes the new state of a node In a first layer of the new state : ; wherein, is a node In a first the hidden state vector of the layer, is an aggregated information vector, is an update function.

7. The system of claim 4, wherein, the data processing module is further used to label initial attribute vectors for the nodes and the edges.

8. The system of claim 7, wherein, The data processing module uses a graph neural network model to identify crop phenotypes and environmental states, specifically comprising the following steps:

9. The system of claim 1, wherein, the hidden state vectors of the nodes in the graph are iteratively updated through multi-layer information propagation.

10. The system of claim 1, wherein, After the multi-layer information propagation is completed, the data processing module is further used to integrate the final state vectors of all the nodes in the graph by using a readout function to generate a graph-level representation vector. The data processing module is further used to calculate the prediction probability of a target category through a classifier based on the graph-level representation vector to complete the identification of the crop phenotypes and the environmental states. The multi-sensor data comprises at least one selected from the following: visible light image data, multispectral image data, hyperspectral image data, laser radar point cloud data, temperature sensor data and humidity sensor data. The crop phenotypes comprise at least one selected from the following: crop plant height, leaf area index, canopy coverage, disease and pest stress state and nutrient stress state.

Images