Rice monitoring method, device and system based on multi-modal data fusion

Abstract

The invention provides a rice monitoring method, device and system based on multi-modal data fusion, and the method comprises the steps: obtaining the spectrum and image data of a rice canopy, respectively extracting the features of the data, inputting the features to a trained rice monitoring model, and obtaining a monitoring result; the rice monitoring model is obtained based on the following steps: preprocessing rice canopy spectrum and image data to obtain a feature vector and a pixel matrix; and training a multi-task learning model composed of a one-dimensional convolutional neural network and a two-dimensional coupon neural network based on the sample feature vector, the pixel matrix and the corresponding label matrix. Aiming at the problems of tedious flow, low efficiency, poor real-time performance and the like in the existing rice nitrogen and chlorophyll detection and pest and disease damage identification technology, rapid in-situ detection of the nitrogen and chlorophyll contents of rice leaves and efficient pest and disease damage identification are realized through a map fusion technology; the technical problems that an existing rice monitoring method is prone to misjudgment and missed judgment and low in efficiency can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of crop monitoring, in particular to a rice monitoring method, device and system based on multi-modal data fusion. BACKGROUND

[0002] Rice is a major food crop, and accurate supply of nutritional elements and effective prevention and control of biological stress are two key links to ensure rice production. Nitrogen and chlorophyll are key factors affecting the growth, yield and quality formation of rice, but their accurate regulation is still a difficulty in current agricultural management. At present, rice nutrient management mainly relies on the extensive mode of "experience-based fertilization", and it is difficult for growers to real-time grasp the dynamic changes of nitrogen and chlorophyll in rice leaves at different growth stages. The traditional method of sampling and sending to the laboratory for analysis is time-consuming, high-cost and result-lagging, which cannot guide the field precise fertilization in time. In terms of biological stress prevention and control, the identification of pests and diseases mainly relies on manual inspection, which is inefficient under large-scale planting conditions, and is easily affected by personnel experience, field environment and other factors, resulting in misjudgment and missed judgment, and often missing the best prevention period, which aggravates the risk of yield loss.

[0003] With the rapid development of smart agriculture technology, rice production has put forward new demands for real-time, precision and in-situ nutrient monitoring and biological stress identification. In terms of nutrient detection, traditional laboratory methods cannot realize in-situ continuous monitoring in the field, which is difficult to meet the technical requirements of variable rate fertilization and precise regulation. Although existing portable leaf detection equipment can achieve a certain degree of in-situ analysis, it is still limited by the detection range, data continuity and equipment flexibility, and it is difficult to support long-term dynamic monitoring of nitrogen and chlorophyll in large-area rice fields and regional precision management decision-making. In terms of pest and disease identification, the traditional manual inspection mode is not suitable for large-scale rice field management needs. The method relying on human eye recognition is low in efficiency and easy to be affected by subjective experience and environmental interference, and misidentification and missed identification occur frequently, which makes it difficult to achieve the goal of early detection and early prevention of smart plant protection.

[0004] In summary, the existing rice monitoring method has the problems of misjudgment, missed judgment and low efficiency. SUMMARY

[0005] Therefore, it is necessary to provide a rice monitoring method, device and system based on multi-modal data fusion to solve the technical problems of misjudgment, missed judgment and low efficiency of the existing rice monitoring method.

[0006] In order to solve the above problems, in a first aspect, the present application provides a rice monitoring method based on multi-modal data fusion, comprising: obtaining spectral data and image data of a rice canopy; preprocessing the spectral data to obtain a feature vector, and preprocessing the image data to obtain a pixel matrix; inputting the feature vector and the pixel matrix into a trained rice monitoring model to obtain a rice monitoring result; The rice monitoring model is obtained based on the following steps. preprocessing sample spectral data of a rice canopy to obtain a sample feature vector, and preprocessing sample image data of the rice canopy to obtain a sample pixel matrix; training a multi-task learning model composed of a one-dimensional convolutional neural network and a two-dimensional convolutional neural network based on the sample feature vector and the sample pixel matrix and a corresponding label matrix to obtain the rice monitoring model.

[0007] In a possible implementation, inputting the feature vector and the pixel matrix into a trained rice monitoring model to obtain a rice monitoring result includes: inputting the feature vector and the pixel matrix into the trained rice monitoring model; extracting spectral sequence features from the feature vector based on a one-dimensional convolutional neural network of the rice monitoring model; extracting image spatial features from the pixel matrix based on a two-dimensional convolutional neural network of the rice monitoring model; splicing and fusing the spectral sequence features and the image spatial features based on an intermediate layer of the rice monitoring model to obtain fused features; learning cross-modal correlation information of the fused features based on a full connection layer of the rice monitoring model; classifying the cross-modal correlation information based on an output head of the rice monitoring model to obtain the rice monitoring result.

[0008] In a possible implementation, the output head includes a linear activation output head and a softmax activation output head. classifying the cross-modal correlation information based on an output head of the rice monitoring model to obtain the rice monitoring result includes: classifying the cross-modal correlation information based on the linear activation output head to obtain prediction results of nitrogen content and chlorophyll content of the rice; classifying the cross-modal correlation information based on the softmax activation output head to obtain detection results of pest and disease identification of the rice.

[0009] In a possible implementation, training a multi-task learning model composed of a one-dimensional convolutional neural network and a two-dimensional convolutional neural network based on the sample feature vector and the sample pixel matrix and a corresponding label matrix to obtain the rice monitoring model includes: training a multi-task learning model composed of a one-dimensional convolutional neural network and a two-dimensional convolutional neural network based on the sample feature vectors and the sample pixel matrix and the corresponding label matrix, to obtain a trained model and nitrogen content prediction results, chlorophyll content prediction results and pest and disease identification results output by the trained model; measuring the prediction accuracy and the goodness of fit of the nitrogen content prediction results and the chlorophyll content prediction results output by the trained model, and optimizing the trained model based on the prediction accuracy and the goodness of fit to obtain a first intermediate model; determining the classification ability and the category distinguishing effect of the pest and disease identification results output by the trained model, and optimizing the first intermediate model based on the classification ability and the category distinguishing effect to obtain a rice monitoring model.

[0010] In a possible implementation, training a multi-task learning model composed of a one-dimensional convolutional neural network and a two-dimensional convolutional neural network based on the sample feature vectors and the sample pixel matrix and the corresponding label matrix, to obtain a trained model and nitrogen content prediction results, chlorophyll content prediction results and pest and disease identification results output by the trained model, includes: training a multi-task learning model composed of a one-dimensional convolutional neural network and a two-dimensional convolutional neural network based on the sample feature vectors and the sample pixel matrix and the corresponding label matrix, and using an early stopping callback mechanism to monitor the validation set loss and terminate training when there is no improvement for a plurality of consecutive rounds, and adjusting the loss weight to balance the learning progress of different tasks during training, to obtain a trained model and nitrogen content prediction results, chlorophyll content prediction results and pest and disease identification results output by the trained model.

[0011] In a possible implementation, the spectral data is preprocessed to obtain a feature vector, including: the spectral data is sequentially subjected to SG smoothing denoising, first and second derivative transformation, and standardization and normalization processing to obtain processed spectral data; feature wavelengths related to nitrogen, chlorophyll and pests and diseases are selected from the processed spectral data to form a feature vector.

[0012] In a possible implementation, the image data is preprocessed to obtain a pixel matrix, including: the image data is subjected to ROI segmentation to obtain a leaf region; the leaf region is subjected to size normalization and data enhancement processing to obtain a normalized and enhanced leaf image; the normalized and enhanced leaf image is subjected to random rotation, flipping and brightness adjustment, and then the pixels are normalized to obtain a pixel matrix.

[0013] In a second aspect, the present application provides a rice monitoring device based on multi-modal data fusion, comprising: an acquisition module configured to acquire spectral data and image data of a rice canopy; a preprocessing module configured to preprocess the spectral data to obtain a feature vector and preprocess the image data to obtain a pixel matrix; a monitoring module configured to input the feature vector and the pixel matrix into a trained rice monitoring model to obtain a rice monitoring result; wherein the rice monitoring model is obtained based on the following steps: preprocessing sample spectral data of a rice canopy to obtain a sample feature vector and preprocessing sample image data of the rice canopy to obtain a sample pixel matrix; training a multi-task learning model composed of a one-dimensional convolutional neural network and a two-dimensional convolutional neural network based on the sample feature vector, the sample pixel matrix, and a corresponding label matrix to obtain the rice monitoring model.

[0014] In a third aspect, the present application provides a rice monitoring system based on multi-modal data fusion, comprising a server, a camera, a hyperspectral sensor, and an Internet of Things communication module; wherein the camera is configured to acquire image data of a rice canopy and upload the image data to the server through the Internet of Things communication module; the hyperspectral sensor is configured to acquire spectral data of the rice canopy and upload the spectral data to the server through the Internet of Things communication module; the server is configured to implement the steps of the rice monitoring method based on multi-modal data fusion as described in any one of the above aspects.

[0015] In a possible implementation, the rice monitoring system based on multi-modal data fusion further comprises: a local monitoring terminal configured to receive the rice monitoring result sent by the processor through the Internet of Things communication module and display the result on a display screen; a manual telescopic rod configured to fix the camera and the hyperspectral sensor and adjust the height of the camera and the hyperspectral sensor; and the hyperspectral sensor is externally covered with optical glass; the hyperspectral sensor comprises a first hyperspectral sensor and a second hyperspectral sensor; the first hyperspectral sensor is configured with a cosine receiver arranged vertically upward to capture incident sunlight; the second hyperspectral sensor is arranged vertically downward to capture reflected light of the rice.

[0016] The beneficial effects of the above implementation manner are that the rice monitoring method, device and system based on multi-modal data fusion provided by the application obtain spectral data and image data of a rice canopy; the spectral data is preprocessed to obtain a feature vector, and the image data is preprocessed to obtain a pixel matrix; the feature vector and the pixel matrix are input into a trained rice monitoring model to obtain a rice monitoring result. The application adopts a modeling method of spectrum fusion. Compared with predicting the nitrogen content and chlorophyll of rice leaves by using single spectral data and identifying diseases and pests by using single image data, the method realizes complementary advantages by fusing the sensitivity of spectral data to internal chemical components of leaves and the capturing ability of image data to external morphology and spatial features of the canopy, and can achieve higher prediction accuracy. Through the fusion model of one-dimensional convolutional neural network and two-dimensional convolutional neural network, the nitrogen content, chlorophyll and disease and pest conditions of rice leaves can be accurately predicted at the same time. The traditional measurement method needs to go through multiple steps such as sampling, pretreatment and laboratory analysis, and the process is complicated, long and cannot reflect the real-time nutritional status in the field. The identification of diseases and pests mainly depends on manual inspection, which is inefficient under large-scale planting conditions and is prone to misjudgment and omission. The application adopts the spectrum fusion technology to quickly complete the synchronous detection of the nitrogen and chlorophyll content of leaves and the efficient identification of diseases and pests in the field, and does not rely on manual inspection, effectively solving the technical problems of the existing rice monitoring method that are prone to misjudgment, omission and low efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Figure 1 The flowchart of one embodiment of the rice monitoring method based on multi-modal data fusion provided by the application; Figure 2 The flowchart of the trained rice monitoring model provided by the application; Figure 3 The flowchart of another embodiment of the rice monitoring method based on multi-modal data fusion provided by the application; Figure 4 The principle block diagram of one embodiment of the rice monitoring device based on multi-modal data fusion provided by the application; Figure 5 The structural schematic diagram of one embodiment of the rice monitoring system based on multi-modal data fusion provided by the application; Figure 6 A structural schematic diagram of a case provided by the present application is shown in the figure. Figure 7 A workflow diagram of a rice monitoring system based on multi-modal data fusion provided by the present application is shown in the figure. Figure 8 An embodiment structural schematic diagram of an electronic device provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0020] In the description of the embodiments of the present application, the meaning of “a plurality of” is two or more, unless otherwise specified.

[0021] In the embodiments of the present application, the terms “comprising” and “having” and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, device, product or equipment comprising a series of steps or modules does not have to be limited to the clearly listed steps or modules, but can include other steps or modules that are not clearly listed or inherent to the process, method, product or equipment.

[0022] The naming or numbering of steps appearing in the embodiments of the present application does not mean that the steps in the method flow must be performed in the time / logical order indicated by the naming or numbering. The flow steps that have been named or numbered can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved.

[0023] In this document, the term “embodiment” means that the specific features, structures or characteristics described in connection with the embodiment can be included in at least one embodiment of the present application. The phrase appears at various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily independent or alternative to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0024] The application adopts atlas fusion technology to cooperatively analyze hyperspectral data and canopy image, which can significantly improve the accurate perception of the growth state of rice. The method realizes complementary advantages by fusing the sensitivity of spectral data to the internal chemical composition of leaves and the capture ability of image data to the external morphology and spatial features of the canopy. On the one hand, when inverting key physiological parameters such as nitrogen and chlorophyll, the spatial and texture information provided by the fused canopy image can effectively correct the spectral fluctuations caused by background interference and canopy structure differences, thereby establishing a more robust and accurate quantitative analysis model. On the other hand, when identifying biological stress such as diseases and pests, the spectral features that are extremely sensitive to physiological changes can capture early physiological abnormalities before visual symptoms appear, thereby realizing early warning and accurate identification of stress, significantly improving the sensitivity and reliability of the identification model.

[0025] The atlas fusion technology constructs an integrated perception system for the nitrogen, chlorophyll content and disease and pest situation of rice by deep combination of spectrum and image, which can simultaneously realize accurate quantification of key physiological parameters and early intelligent diagnosis of biological stress, and provides more comprehensive and reliable technical support for precision agricultural management of rice.

[0026] The application provides a rice monitoring method, device and system based on multi-modal data fusion, which are described below.

[0027] The application provides a rice monitoring method based on multi-modal data fusion, which can be realized by executing an application program on a server, and the server is a cloud server. Figure 1 As shown in the figure, the method comprises: S101, acquiring spectral data and image data of a rice canopy.

[0028] It can be understood that the image data can be collected by a camera, and the spectral data can be collected by a configured hyperspectral sensor. Specifically, the camera and the hyperspectral sensor can automatically collect canopy hyperspectral data and image data through a timing task, and upload the collected data to the server.

[0029] More specifically, the self-designed rice monitoring device is used to simultaneously acquire spectral data and image data of the rice canopy in the range of 400-1000nm, and rice samples covering different growth stages, health conditions and fertilization levels are collected to ensure sample diversity. The actual nitrogen content and chlorophyll content of the corresponding leaves are determined by laboratory chemical method, and the specific disease and pest types are determined by artificial diagnosis as the model true value label.

[0030] S102, pre-processing the spectral data to obtain a feature vector, and pre-processing the image data to obtain a pixel matrix.

[0031] It can be understood that the pre-processing of the spectral data can be denoising, standardization and normalization, and then the characteristic wavelength is extracted.

[0032] The pre-processing of the image data can be size normalization and data enhancement processing after the leaf region is segmented, and the pixels in the image are normalized.

[0033] S103, input the feature vector and the pixel matrix into the trained rice monitoring model to obtain a rice monitoring result; As shown in Figure 2 The rice monitoring model is obtained based on the following steps: The sample spectral data of the rice canopy is pre-processed to obtain a sample feature vector, and the sample image data of the rice canopy is pre-processed to obtain a sample pixel matrix; Based on the sample feature vector and the sample pixel matrix, and a corresponding label matrix, a multi-task learning model composed of a one-dimensional convolutional neural network and a two-dimensional convolutional neural network is trained to obtain a rice monitoring model.

[0034] It can be understood that the label matrix is constructed: each sample needs to establish a complete label vector containing all prediction targets, including nitrogen content label, chlorophyll content label, and pest type label, forming a multi-dimensional label system. The data set is divided: the entire paired data set (spectrum, image, [nitrogen, chlorophyll, pest and disease]) is randomly divided into a training set, a validation set and a test set according to a certain proportion.

[0035] The present application aims at the deficiencies of the existing rapid and accurate detection of nitrogen and chlorophyll in rice leaves, and efficient identification technology of pests and diseases, and provides a fixed multi-modal data fusion rice monitoring scheme.

[0036] The present application provides a fixed multi-modal data fusion rice monitoring scheme, aiming at solving the limitations of the existing detection technology. In terms of nitrogen and chlorophyll detection, the traditional laboratory method needs to go through complex pretreatment processes such as sample collection, organ separation, fixation, drying, and powdering, which is time-consuming, high-cost, and the results are seriously lagging behind, and cannot reflect the real-time nutritional status in the field; although the existing portable equipment supports partial in-situ detection, it is limited by the detection range, data continuity and mobility, and it is difficult to realize long-term dynamic monitoring of large-area paddy fields. In terms of pest and disease identification, it currently mainly relies on manual inspection, which is low in efficiency and easy to be affected by subjective experience, and misjudgment and omission are frequent, which cannot meet the requirements of intelligent agriculture for early detection and early prevention of biological stress.

[0037] As Figure 3As shown, the artificial intelligence model adopted by the present application includes a 1D-2D CNN (i.e. one-dimensional convolutional neural network and two-dimensional convolutional neural network) fusion model for both biological pest and disease identification and model for leaf nitrogen and chlorophyll content prediction. Compared with a single one-dimensional convolutional neural network (1D-CNN) for predicting leaf nitrogen and chlorophyll content through spectral reflectance data or a two-dimensional convolutional neural network model (2D-CNN) for identifying biological pests and diseases through image data, the method realizes complementary advantages by fusing the sensitivity of spectral data to the internal chemical composition of the leaf and the capture ability of image data to the external morphology and spatial features of the canopy.

[0038] Although the rice canopy in the visible-near infrared band will exhibit specific spectral response characteristics due to differences in nitrogen and chlorophyll content in the leaf, the 1D-CNN model can be used to predict the nitrogen and chlorophyll content. However, the spectrum mainly reflects the internal chemical composition information of the leaf, which is an indirect measurement of the internal physiological state, and it cannot capture the external morphological manifestations related to these internal states. For example, nitrogen deficiency or lack of chlorophyll will cause the leaf to lose green and turn yellow, and lack of nutrition will cause the leaf to become smaller and the leaf texture to change. Such color changes and spatial distribution cannot be reflected through one-dimensional spectral data.

[0039] Similarly, when the rice is attacked and competed by pests and diseases, the canopy will exhibit typical changes in color, texture, and morphology. The 2D-CNN model can be used to identify the type of pests and diseases. For example, rice blast, sheath blight, and rice leaf roller can be accurately identified through image information due to their unique symptoms and obvious characteristics. However, pictures mainly reflect the external visual features of the leaf, such as the shape, color, texture, and distribution of the disease spot. For diseases with similar symptoms and extremely similar appearances, such as water-stained stripes of white leaf blight and bacterial leaf spot, it is difficult to accurately identify the disease condition through images and the 2D-CNN model. In fact, as early as the initial stage of disease infection, the chemical composition of the leaf, such as chlorophyll and cellulose content, will change, thereby changing its unique spectral reflectance characteristics. However, no obvious disease spots are visible to the naked eye. The internal structural changes cannot be reflected through pictures.

[0040] In view of the above difficulties in accurately predicting and identifying the nitrogen, chlorophyll, and pests and diseases of rice leaves through single-dimensional data, the present application provides a 1D-2D CNN fusion model. The model uses dual-branch feature extraction to model one-dimensional spectral data and two-dimensional image data, and performs feature fusion in the middle layer. Through a fusion model, multiple indicators of rice can be more accurately predicted.

[0041] In some embodiments, the feature vector and the pixel matrix are input into a trained rice monitoring model to obtain a rice monitoring result, including: inputting the feature vector and the pixel matrix into a trained rice monitoring model; extracting spectral sequence features from the feature vector based on a one-dimensional convolutional neural network of the rice monitoring model; extracting image space features from the pixel matrix based on a two-dimensional convolutional neural network of the rice monitoring model; splicing and fusing the spectral sequence features and the image space features based on an intermediate layer of the rice monitoring model to obtain fused features; learning cross-modal correlation information of the fused features based on a fully connected layer of the rice monitoring model; classifying the cross-modal correlation information based on an output head of the rice monitoring model to obtain a rice monitoring result.

[0042] It can be understood that the core of the multi-task learning model construction is to design a unified architecture that can simultaneously process spectral and image dual-modal inputs and cooperatively complete multiple prediction tasks. The model first extracts spectral sequence features and image space features through independent 1D-CNN and 2D-CNN branches, respectively, then splices and fuses the feature vectors output by the two branches in the intermediate layer, and finally further learns cross-modal correlation information through a shared fully connected layer of neurons, and finally performs regression and classification tasks through three independent output heads.

[0043] In some embodiments, the output head includes a linear activation output head and a softmax activated output head. classifying the cross-modal correlation information based on an output head of the rice monitoring model to obtain a rice monitoring result, including: classifying the cross-modal correlation information based on the linear activation output head to obtain prediction results of nitrogen content and chlorophyll content of the rice; classifying the cross-modal correlation information based on the softmax activated output head to obtain detection results of pest and disease identification of the rice.

[0044] It can be understood that regression and classification tasks are performed through three independent output heads: nitrogen content and chlorophyll content prediction use a single neuron with linear activation to output continuous numerical values, while pest and disease identification uses multiple neurons with softmax activation to output category probability distribution; during model compilation, corresponding loss functions are configured for different task types and the weights of each task are balanced, so that multiple related tasks are efficiently learned in parallel based on shared underlying features.

[0045] In some embodiments, based on the sample feature vector and the sample pixel matrix, and the corresponding label matrix, a multi-task learning model composed of a one-dimensional convolutional neural network and a two-dimensional convolutional neural network is trained to obtain a rice monitoring model, comprising: Based on the sample feature vector and the sample pixel matrix, and the corresponding label matrix, a multi-task learning model composed of a one-dimensional convolutional neural network and a two-dimensional convolutional neural network is trained to obtain a trained model and nitrogen content prediction results, chlorophyll content prediction results and pest and disease identification results output by the trained model. The prediction accuracy and goodness of fit of the nitrogen content prediction results and chlorophyll content prediction results output by the trained model are measured, and based on the prediction accuracy and the goodness of fit, the trained model is optimized to obtain a first intermediate model. The classification ability and category discrimination effect of the pest and disease identification results output by the trained model are determined, and based on the classification ability and the category discrimination effect, the first intermediate model is optimized to obtain a rice monitoring model.

[0046] It can be understood that the model evaluation stage respectively evaluates the comprehensive performance of the three tasks on the independent test set, wherein the nitrogen content and chlorophyll content prediction adopts RMSE (Root Mean Square Error), MAE (Mean Absolute Error) and R² regression index to measure the prediction accuracy and goodness of fit, and the pest and disease identification comprehensively evaluates the classification ability and category discrimination effect of the model through accuracy, confusion matrix and each category F1-Score index, to ensure that the model meets the practical requirements on each task. The structure of the model that does not meet the requirements is optimized and the parameters are adjusted.

[0047] The qualified model is deployed on a cloud inference platform, and a regular updating mechanism is established to continuously improve the generalization ability and robustness of the model.

[0048] In some embodiments, based on the sample feature vector and the sample pixel matrix, and the corresponding label matrix, a multi-task learning model composed of a one-dimensional convolutional neural network and a two-dimensional convolutional neural network is trained to obtain a trained model and nitrogen content prediction results, chlorophyll content prediction results and pest and disease identification results output by the trained model, comprising: Based on the sample feature vector and the sample pixel matrix, and the corresponding label matrix, a multi-task learning model composed of a one-dimensional convolutional neural network and a two-dimensional convolutional neural network is trained, and an early stopping callback mechanism is used to monitor the validation set loss and terminate the training when there is no improvement for continuous multiple rounds. During the training process, the learning progress of different tasks is balanced by adjusting the loss weight to obtain a trained model and nitrogen content prediction results, chlorophyll content prediction results and pest and disease identification results output by the trained model.

[0049] It can be understood that the model training adopts a multi-task learning framework, simultaneously processes the spectral and image data through double input streams, monitors the validation set loss using an early stopping callback mechanism and automatically terminates the training when there is no improvement for multiple rounds in succession, while restoring the best weights; during the training process, the learning progress of different tasks is balanced by adjusting the loss weight, and the network structure and learning rate and other hyperparameters are optimized to ensure that the three prediction tasks can converge cooperatively and achieve the best performance.

[0050] In some embodiments, the spectral data is preprocessed to obtain a feature vector, including: The spectral data is sequentially subjected to SG smoothing denoising (Savitzky-Golay filtering), first and second derivative transformation, and standardization and normalization processing to obtain processed spectral data; Feature wavelengths related to nitrogen, chlorophyll and plant diseases and insect pests are screened from the processed spectral data to form a feature vector.

[0051] It can be understood that the spectral data preprocessing: sequentially performing SG smoothing denoising, first and second derivative transformation to eliminate baseline drift, standardization and normalization processing to unify the dimension, and then combining random forest and XGBoost algorithm to screen feature wavelengths sensitive to nitrogen, chlorophyll and plant diseases and insect pests to form a feature vector.

[0052] In some embodiments, the image data is preprocessed to obtain a pixel matrix, including: The image data is subjected to ROI (region of interest) segmentation to obtain a leaf region; The leaf region is subjected to size normalization and data enhancement processing to obtain a normalized and enhanced leaf image; The normalized and enhanced leaf image is subjected to random rotation, flipping and brightness adjustment, and then the pixels are subjected to normalization processing to obtain a pixel matrix.

[0053] It can be understood that the image data preprocessing: first performing ROI segmentation to accurately segment the leaf region from the image; then performing size normalization to adjust all ROI images to a fixed size; then performing data enhancement processing to randomly rotate, flip and adjust the brightness of the training set images to increase robustness; finally, the pixels are subjected to normalization processing to scale the pixel values to [0, 1].

[0054] In some embodiments, the present application provides a step of training a rice monitoring model, including: Step 1: Data collection. Use the self-designed rice monitoring device to simultaneously obtain spectral data and image data of the rice canopy in the range of 400-1000 nm, with a data volume of more than 600. Collect rice samples covering different growth stages, health conditions, and fertilization levels to ensure sample diversity. Simultaneously, use laboratory chemical methods to determine the actual nitrogen content and chlorophyll content of the corresponding leaves, and determine the specific disease and pest types through artificial diagnosis as the model true value label.

[0055] Step 2: Data preprocessing. Spectral data preprocessing: sequentially perform SG smoothing denoising, first and second derivative transformation to eliminate baseline drift, standardization and normalization processing to unify the dimension, and then combine random forest and XGBoost algorithm to screen sensitive feature wavelengths of nitrogen, chlorophyll, and diseases and pests to form a feature vector. Image data preprocessing: first perform ROI segmentation to accurately segment the leaf area from the image; then perform size normalization to adjust all ROI images to a fixed size; then perform data enhancement processing to randomly rotate, flip, and adjust the brightness of the training set images to increase robustness; finally, normalize the pixels to scale the pixel values to [0, 1].

[0056] Step 3: Data set construction and label encoding. Construct the label matrix: each sample needs to establish a complete label vector containing all prediction targets, including nitrogen content label, chlorophyll content label, and disease and pest type label, forming a multi-dimensional label system. Data set division: randomly divide the entire paired data set (spectrum, image, [nitrogen, chlorophyll, disease and pest]) into training set, validation set, and test set according to a certain proportion.

[0057] Step 4: Multi-task learning model construction, the flow chart is as follows Figure 2The input layer adopts a dual-modal input design. The spectral input channel inputs one-dimensional spectral reflectance data, and the input shape is (S,). The image input channel inputs two-dimensional spectral data, and the input shape is (H, W, C), where H is the image height, W is the image width, and C is the number of channels. The spectral feature extraction and image feature extraction adopt a dual-branch feature extraction, and modeling is performed respectively. The spectral feature extraction uses a 1D CNN model. The one-dimensional spectral data is reshaped first, and the one-dimensional spectral data is reshaped from (S,) to a two-dimensional sequence format of (S, 1). After two layers of convolution and two layers of pooling, the multi-dimensional feature vector is flattened into a one-dimensional vector, and then passes through a fully connected layer and a Droput layer. The convolution layer uses ReLU as the activation function, and the Dropout layer is a regularization technique used in deep learning to prevent model overfitting. By randomly turning off part of the neurons during training, the network is forced to learn redundant features to improve the generalization ability. The image feature extraction uses a 2D CNN model. After two layers of convolution and two layers of pooling, the multi-dimensional feature vector is flattened into a one-dimensional vector, and then passes through a fully connected layer and a Droput layer. The convolution layer uses ReLU as the activation function. In the middle layer, the features of the two models are fused to form a 1D-2D CNN fusion model. The feature vectors output by the spectral branch and the image branch are spliced to form a fusion feature vector. The fused feature vector is then input into the shared feature learning layer, and ReLU is used as the activation function. The fused features are further learned through the shared fully connected layer of neurons to learn the cross-modal correlation information. Finally, three independent output heads are used to perform regression and classification tasks respectively: the nitrogen content and chlorophyll content prediction use a single neuron with linear activation to output continuous numerical values, while the disease and pest identification uses multiple neurons with softmax activation to output category probability distribution; during model compilation, the corresponding loss function is configured for different task types and the task weights are balanced, so that multiple related tasks can be learned efficiently and in parallel based on the shared bottom layer features.

[0058] Step 5: Model training and optimization. The model training adopts a multi-task learning framework, which simultaneously processes spectral and image data through dual input streams, uses an early stopping callback mechanism to monitor the validation set loss and automatically terminates training when there is no improvement for 20 consecutive rounds, and restores the best weights at the same time; during the training process, the learning progress of different tasks is balanced by adjusting the loss weight, and the network structure and learning rate and other hyperparameters are optimized to ensure that the three prediction tasks can converge cooperatively and achieve the best performance.

[0059] Step 6: Model evaluation. The model evaluation stage comprehensively evaluates the performance of the three tasks on the independent test set, respectively. The prediction of nitrogen content and chlorophyll content uses RMSE, MAE and R² regression indicators to measure the prediction accuracy and goodness of fit. The disease and pest identification comprehensively evaluates the classification ability and class distinction effect of the model through accuracy, confusion matrix and F1-Score indicators of each class to ensure that the model meets the practical requirements on each task. For the models that do not meet the requirements, return to step 5 for structure optimization and parameter adjustment.

[0060] Step 7: Deploy the qualified model on the cloud reasoning platform and establish a regular updating mechanism to continuously improve the generalization ability and robustness of the model.

[0061] The present application has the following advantages: (1) The present application realizes the accurate in-situ detection of nitrogen and chlorophyll content of rice and the efficient identification of diseases and pests. Traditional measurement methods need to go through multiple steps such as sampling, pretreatment and laboratory analysis, which is complicated and time-consuming, and cannot reflect the real-time nutritional status in the field. The identification of diseases and pests mainly relies on manual inspection, which is inefficient under large-scale planting conditions and is prone to misjudgment and omission. The present application uses a fixed monitoring device and non-destructive spectroscopy technology to quickly complete the synchronous detection of leaf nitrogen and chlorophyll content and the efficient identification of diseases and pests in the field, effectively solving the problem of lag and extensive traditional methods.

[0062] (2) Compared with using a hyperspectral unmanned aerial vehicle to obtain crop hyperspectral data, the cost is high because the unmanned aerial vehicle is expensive and needs multiple people to cooperate to complete equipment debugging, flight planning and other operations, and whiteboard correction is required. The present application uses an upper and lower dual-probe hyperspectral sensor to simultaneously collect solar incident light and crop reflected light, directly calculates the canopy reflectance, and avoids the operation of frequent whiteboard correction. The system has low cost and is easy to deploy, and can realize long-term in-situ hyperspectral data measurement of crops.

[0063] (3) The present application uses a graph fusion modeling method, which is superior to single spectral data for predicting leaf nitrogen and chlorophyll content or single image data for identifying biological diseases and pests. The method combines the sensitivity of spectral data to leaf internal chemical composition and the capture ability of image data to canopy external morphology and spatial features to achieve complementary advantages. The method can achieve higher prediction accuracy, and through a 1D-2D CNN fusion model, the leaf nitrogen, chlorophyll and disease and pest conditions of rice can be accurately predicted simultaneously.

[0064] (4) A height-adjustable adaptive monitoring structure is designed. In view of the difference in canopy height of rice at different growth stages or of different varieties, the present application uses a manual telescopic rod structure to flexibly adjust the sensor detection height according to the actual growth conditions, effectively improving the accuracy and applicability of the nutrient and stress identification model.

[0065] As Figure 4 shown, the application also provides a rice monitoring device 400 based on multi-modal data fusion, comprising: a data acquisition module 401 for acquiring spectral data and image data of the rice canopy; a preprocessing module 402 for preprocessing the spectral data to obtain a feature vector, and preprocessing the image data to obtain a pixel matrix; a monitoring module 403 for inputting the feature vector and the pixel matrix into a trained rice monitoring model to obtain a rice monitoring result; wherein the rice monitoring model is obtained based on the following steps: preprocessing sample spectral data of the rice canopy to obtain a sample feature vector, and preprocessing sample image data of the rice canopy to obtain a sample pixel matrix; training a multi-task learning model composed of a one-dimensional convolutional neural network and a two-dimensional convolutional neural network based on the sample feature vector and the sample pixel matrix, and a corresponding label matrix, to obtain a rice monitoring model.

[0066] The application also provides a rice monitoring system based on multi-modal data fusion, as Figure 5 shown, comprising: a server (not shown in the figure), and a camera 510, a hyperspectral sensor, and an Internet of Things communication module 512; wherein the camera 510 is used to collect image data of the rice canopy and upload the image data to the server through the Internet of Things communication module 512; the hyperspectral sensor is used to collect spectral data of the rice canopy and upload the spectral data to the server through the Internet of Things communication module 512; the server is used to implement the steps of the rice monitoring method based on multi-modal data fusion as described in any one of the above.

[0067] It can be understood that the hyperspectral sensor includes a first hyperspectral sensor 506 and a second hyperspectral sensor 508.

[0068] The first hyperspectral sensor 506 is configured with a cosine receiver and is arranged vertically upward for capturing the incident sunlight; specifically, after diffuse reflection, the light incident at different angles can be uniformly accepted, and the sky is oriented to capture the incident sunlight in real time.

[0069] The second hyperspectral sensor 508 is arranged vertically downward for capturing the reflected light of the rice.

[0070] The second hyperspectral sensor 508 is a fiber-optic interface, vertically downward to capture crop reflected light in real time, and through the acquisition of leaf reflectance spectrum data and solar incident spectrum data, the leaf reflectance can be calculated by the ratio of the two without additional whiteboard calibration.

[0071] In some embodiments, the rice monitoring system based on multi-modal data fusion further comprises: The local monitoring terminal 511 receives the rice monitoring result sent by the processor through the Internet of Things communication module 512. The manual telescopic rod 503 is used to fix the camera 510 and the hyperspectral sensor, and to adjust the height of the camera 510 and the hyperspectral sensor. The hyperspectral sensor is externally covered with optical glass 505.

[0072] The local monitoring terminal 511 receives the rice monitoring result sent by the processor through the Internet of Things communication module 512, and displays it through the display screen 507. The local monitoring terminal 511 has storage and certain edge computing capabilities, and can store image data and spectrum data.

[0073] The manual telescopic rod 503 is used to fix the camera 510 and the hyperspectral sensor, and to adjust the height of the camera 510 and the hyperspectral sensor.

[0074] The hyperspectral sensor is externally covered with optical glass 505.

[0075] In addition, Figure 5 The case 501, cross beam 502, solar panel 504, and battery 509 are also shown.

[0076] It can be understood that the present application integrates spectrum acquisition equipment, multi-modal data acquisition module and artificial intelligence analysis model by fusing visible light-near infrared spectrum technology and machine vision technology, realizes rapid and accurate detection of rice leaf nitrogen and chlorophyll, and efficient identification of diseases and pests. The device system provided by the present application can be fixed and deployed in key areas of the rice field, and automatically collects canopy hyperspectral data and image data through timed tasks. Due to the high computational complexity of the fusion data, the leaf nitrogen and chlorophyll prediction and disease and pest identification will be uploaded to the cloud, and then the results will be returned to the local terminal. All analysis results can be displayed in real time on the terminal screen, so that users can obtain the nutritional and biological stress conditions of rice in the field. At the same time, the device relies on the Internet of Things technology to upload all data to the cloud management platform in real time, providing all-weather, in-situ and continuous rice growth dynamic information for growers and agricultural technicians, and providing integrated technical support for accurate nutrition regulation and efficient biological stress prevention and control. In some embodiments, the system provided by the application comprises five modules: data acquisition, local monitoring terminal, energy supply, communication transmission and mechanical structure. Specifically, it is composed of a camera 510, two hyperspectral sensors, a local monitoring terminal, a battery, a solar panel, an Internet of Things communication module 512 (DTU), a display screen, K9 optical glass, a manual telescopic rod, a crossbeam and a case. As shown in Figure 6 The case is provided with an incident light collection window 601, a data display window 602, an image collection window 603 and a reflected light collection window 604.

[0077] In terms of data acquisition, the camera 510 is responsible for acquiring rice canopy images for analyzing the pest and disease conditions of rice; the two groups of hyperspectral sensors are arranged vertically towards the sky and vertically downward, the upper spectral sensor is provided with a cosine receiver, which can uniformly receive light of different angles after diffuse reflection, and the lower spectral sensor is a fiber interface, which is vertically downward to capture crop reflected light in real time. By collecting leaf reflectance spectrum data and solar incident spectrum data, the leaf reflectance can be calculated by the ratio of the two without additional whiteboard calibration.

[0078] The Internet of Things communication module 512 (DTU) is responsible for uploading image data and spectral data to the cloud platform, i.e. the server. These data are calculated by the 1D-2D CNN fusion model deployed on the cloud, and the results are returned to the local monitoring terminal, so as to realize the synchronization and real-time grasp of the rice nutrient status and biological stress conditions by the field and remote users. The solar panel and the battery constitute the energy unit of the system, realizing energy collection and storage and ensuring the continuous operation of the equipment in the field environment.

[0079] In order to adapt to the change of rice canopy height in different growth periods, the system is equipped with a manual telescopic rod, which can flexibly adjust the sensor height according to the actual monitoring needs. The hyperspectral sensor probe is covered with ultra-thin K9 optical glass, which can ensure high transmittance of light while playing the roles of dustproof, moistureproof and physical protection, ensuring the reliability of long-term outdoor use. The overall structure of the system is integrated in the protective case.

[0080] As shown in Figure 7As shown, the system provided by this invention operates as follows: The console first sends a preset timed task instruction to the local terminal. Upon receiving the instruction, the terminal simultaneously activates the hyperspectral sensor and camera 510 to collect canopy spectral data and image data, respectively. The IoT communication module 512 uploads this data to the cloud. After calculation using a 1D-2DCNN fusion model deployed in the cloud, the calculated results are returned to the local terminal and displayed on the screen. This mechanism allows users to simultaneously obtain information on the nutrient status and biological stress of rice in the field. Furthermore, all analysis results, including leaf nitrogen and chlorophyll content, and pest and weed identification results, are transmitted in real time to the cloud management platform via IoT technology, forming an all-weather, in-situ, and continuous dynamic monitoring data system for rice growth, providing accurate and timely decision support for growers and agricultural technicians.

[0081] The rice monitoring device based on multimodal data fusion provided in the above embodiments can realize the technical solutions described in the above embodiments of the rice monitoring method based on multimodal data fusion. The specific implementation principles of each module or unit can be found in the corresponding content of the above embodiments of the rice monitoring method based on multimodal data fusion, which will not be repeated here.

[0082] like Figure 8 As shown, the present invention also provides an electronic device 800, which may be a server. The electronic device 800 includes a processor 801, a memory 802, and a display 803. Figure 8 Only some components of the electronic device 800 are shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0083] In some embodiments, memory 802 may be an internal storage unit of electronic device 800, such as a hard disk or memory of electronic device 800. In other embodiments, memory 802 may also be an external storage device of electronic device 800, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 800.

[0084] Furthermore, the memory 802 may include both internal storage units of the electronic device 800 and external storage devices. The memory 802 is used to store application software and various types of data installed on the electronic device 800.

[0085] The processor 801 may, in some embodiments, be a central processing unit (CPU), a microprocessor, or other data processing chip, for running program codes stored in the memory 802 or processing data, such as the method for monitoring rice based on multi-modal data fusion in the present application.

[0086] The display 803 may, in some embodiments, be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, or the like. The display 803 is used to display information of the electronic device 800 and to display a visualized user interface. The components 801-803 of the electronic device 800 communicate with each other through a system bus.

[0087] In some embodiments of the present application, when the processor 801 executes the program for monitoring rice based on multi-modal data fusion in the memory 802, the following steps can be implemented: Obtaining spectral data and image data of a rice canopy; Preprocessing the spectral data to obtain a feature vector and preprocessing the image data to obtain a pixel matrix; Inputting the feature vector and the pixel matrix into a trained rice monitoring model to obtain a rice monitoring result; The rice monitoring model is obtained based on the following steps: Preprocessing sample spectral data of a rice canopy to obtain a sample feature vector and preprocessing sample image data of a rice canopy to obtain a sample pixel matrix; Training a multi-task learning model composed of a one-dimensional convolutional neural network and a two-dimensional convolutional neural network based on the sample feature vector and the sample pixel matrix, and a corresponding label matrix to obtain a rice monitoring model.

[0088] It should be understood that, when the processor 801 executes the program for monitoring rice based on multi-modal data fusion in the memory 802, it can also implement other functions in addition to the above functions. For details, please refer to the description of the corresponding method embodiments.

[0089] Further, the embodiments of the present application do not make specific limitation on the type of the electronic device 800 mentioned above, and the electronic device 800 can be a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, or the like. Exemplary embodiments of the portable electronic device include, but are not limited to, a portable electronic device running an IOS, an android, a microsoft, or other operating system. The portable electronic device described above can also be other portable electronic devices, such as a laptop computer having a touch-sensitive surface (e.g., a touch panel), and the like. It should also be understood that in some other embodiments of the present application, the electronic device 800 can also not be a portable electronic device, but a desktop computer having a touch-sensitive surface (e.g., a touch panel).

[0090] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the method for monitoring rice based on multi-modal data fusion provided by the above method, the method comprising: obtaining spectral data and image data of a rice canopy; preprocessing the spectral data to obtain a feature vector, and preprocessing the image data to obtain a pixel matrix; inputting the feature vector and the pixel matrix into a trained rice monitoring model to obtain a rice monitoring result; wherein the rice monitoring model is obtained based on the following steps: preprocessing sample spectral data of a rice canopy to obtain a sample feature vector, and preprocessing sample image data of the rice canopy to obtain a sample pixel matrix; training a multi-task learning model composed of a one-dimensional convolutional neural network and a two-dimensional convolutional neural network based on the sample feature vector and the sample pixel matrix, and a corresponding label matrix, to obtain the rice monitoring model.

[0091] Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. The computer readable storage medium includes a magnetic disk, an optical disk, a read-only memory, a random access memory, and the like.

[0092] The above describes in detail the rice monitoring method, device and system based on multi-modal data fusion provided by the present application. The principles and implementation manners of the present application are described by using specific examples. The above description of the examples is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges can be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A rice monitoring method based on multi-modal data fusion, characterized in that, The method comprises the following steps: acquiring spectral data and image data of a rice canopy; preprocessing the spectral data to obtain a feature vector and preprocessing the image data to obtain a pixel matrix; inputting the feature vector and the pixel matrix into a trained rice monitoring model to obtain a rice monitoring result; wherein the rice monitoring model is obtained based on the following steps: preprocessing sample spectral data of a rice canopy to obtain a sample feature vector and preprocessing sample image data of a rice canopy to obtain a sample pixel matrix; training a multi-task learning model composed of a one-dimensional convolutional neural network and a two-dimensional convolutional neural network based on the sample feature vector and the sample pixel matrix and a corresponding label matrix to obtain the rice monitoring model.

2. The method for rice monitoring based on multi-modal data fusion according to claim 1, characterized in that, inputting the feature vector and the pixel matrix into the trained rice monitoring model to obtain the rice monitoring result, comprising: inputting the feature vector and the pixel matrix into the trained rice monitoring model; extracting spectral sequence features from the feature vector based on the one-dimensional convolutional neural network of the rice monitoring model; extracting image spatial features from the pixel matrix based on the two-dimensional convolutional neural network of the rice monitoring model; splicing and fusing the spectral sequence features and the image spatial features based on the intermediate layer of the rice monitoring model to obtain fused features; learning cross-modal correlation information of the fused features based on the fully connected layer of the rice monitoring model; classifying the cross-modal correlation information based on the output head of the rice monitoring model to obtain the rice monitoring result. 3.The rice monitoring method based on multi-modal data fusion according to claim 2, characterized in that, The output head comprises a linear activation output head and a softmax activation output head. Classifying the cross-modal correlation information based on the output head of the rice monitoring model to obtain the rice monitoring result, comprising: classifying the cross-modal correlation information based on the linear activation output head to obtain prediction results of nitrogen content and chlorophyll content of the rice; classifying the cross-modal correlation information based on the softmax activation output head to obtain detection results of pest and disease identification of the rice.

4. The method for monitoring rice based on multi-modal data fusion according to claim 1, characterized in that, Training a multi-task learning model composed of a one-dimensional convolutional neural network and a two-dimensional convolutional neural network based on the sample feature vector and the sample pixel matrix and a corresponding label matrix to obtain the rice monitoring model, comprising: training a multi-task learning model composed of a one-dimensional convolutional neural network and a two-dimensional convolutional neural network based on the sample feature vector and the sample pixel matrix and a corresponding label matrix to obtain a trained model and nitrogen content prediction results, chlorophyll content prediction results and pest and disease identification results output by the trained model; measuring prediction accuracy and goodness of fit of the nitrogen content prediction results and the chlorophyll content prediction results output by the trained model, and optimizing the trained model based on the prediction accuracy and the goodness of fit to obtain a first intermediate model; determining the classification ability and the category discrimination effect corresponding to the pest and disease identification results output by the trained model, and optimizing the first intermediate model based on the classification ability and the category discrimination effect to obtain the rice monitoring model.

5. The method for rice monitoring based on multi-modal data fusion according to claim 4, characterized in that, training a multi-task learning model composed of a one-dimensional convolutional neural network and a two-dimensional convolutional neural network based on the sample feature vector and the sample pixel matrix, and a corresponding label matrix, to obtain a trained model and nitrogen content prediction results, chlorophyll content prediction results, and pest and disease identification results output by the trained model, including: training a multi-task learning model composed of a one-dimensional convolutional neural network and a two-dimensional convolutional neural network based on the sample feature vector and the sample pixel matrix, and a corresponding label matrix, and using an early stopping callback mechanism to monitor the validation set loss and terminate training when there is no improvement for multiple consecutive rounds, and adjusting the loss weight to balance the learning progress of different tasks during training, to obtain a trained model and nitrogen content prediction results, chlorophyll content prediction results, and pest and disease identification results output by the trained model. 6.The rice monitoring method based on multi-modal data fusion of claim 1, wherein, preprocessing the spectral data to obtain a feature vector, including: performing SG smoothing denoising, first and second derivative transformation, and standardization and normalization processing on the spectral data in sequence to obtain processed spectral data; selecting feature wavelengths related to nitrogen, chlorophyll, and pests and diseases from the processed spectral data to form a feature vector.

7. The method for rice monitoring based on multi-modal data fusion according to any one of claims 1-6, characterized in that, preprocessing the image data to obtain a pixel matrix, including: performing ROI segmentation on the image data to obtain a leaf region; performing size normalization and data enhancement processing on the leaf region to obtain a normalized and enhanced leaf image; performing random rotation, flipping, and brightness adjustment on the normalized and enhanced leaf image, and then performing pixel normalization processing to obtain a pixel matrix.

8. A rice monitoring device based on multi-modal data fusion, characterized in that, including: a data acquisition module for acquiring spectral data and image data of a rice canopy; a preprocessing module for preprocessing the spectral data to obtain a feature vector, and preprocessing the image data to obtain a pixel matrix; a monitoring module for inputting the feature vector and the pixel matrix into a trained rice monitoring model to obtain a rice monitoring result; wherein the rice monitoring model is obtained based on the following steps: preprocessing sample spectral data of a rice canopy to obtain a sample feature vector, and preprocessing sample image data of a rice canopy to obtain a sample pixel matrix; training a multi-task learning model composed of a one-dimensional convolutional neural network and a two-dimensional convolutional neural network based on the sample feature vector and the sample pixel matrix, and a corresponding label matrix, to obtain a rice monitoring model.

9. A rice monitoring system based on multi-modal data fusion, characterized in that, including a server, a camera, a hyperspectral sensor, and an Internet of Things communication module; wherein the camera is configured to acquire image data of a rice canopy and upload the image data to the server through the Internet of Things communication module; the hyperspectral sensor is configured to acquire spectral data of a rice canopy and upload the spectral data to the server through the Internet of Things communication module; the server is configured to implement the steps of the rice monitoring method based on multi-modal data fusion according to any one of claims 1 to 7.

10. The multi-modal data fusion based rice monitoring system according to claim 9, wherein, further including: The local monitoring terminal is used for receiving the rice monitoring result sent by the Internet of Things communication module and displaying the rice monitoring result through a display screen. The manual telescopic rod is used for fixing the camera and the hyperspectral sensor and adjusting the height of the camera and the hyperspectral sensor. The hyperspectral sensor is externally covered with optical glass. The hyperspectral sensor comprises a first hyperspectral sensor and a second hyperspectral sensor. The first hyperspectral sensor is provided with a cosine receiver and is vertically arranged upwards to capture incident sunlight. The second hyperspectral sensor is vertically arranged downwards to capture reflected light of the rice.

Images