Method for analyzing and managing crop phenotype information based on multi-modal information heterogeneous fusion

Abstract

The application discloses a kind of based on multimodal information isomeric crop phenotype information analysis and management method, belong to crop monitoring and agricultural intelligent sensing technical field, steps are as follows: using unmanned aerial vehicle along the planned path synchronous acquisition obtains the hyperspectral image data, visible light image data and infrared image data of crop;Respectively pre-process the image data of crop, obtain multimodal image data;Feature extraction is carried out, and the classification features of multimodal image data are obtained;Optimized multimodal pulse signal is obtained;Real-time environmental data is acquired, and combined with the optimized modal pulse signal, simulate brain confidence assessment, weight distribution, weighted fusion and maximum probability selection, obtain crop phenotype information analysis result;Crop phenotype information analysis result is superimposed on farmland remote sensing map, and stress condition is marked, and stress management scheme is automatically generated.The application solves the problem that crop state analysis has information limitation, and lacks adaptability and flexibility.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of crop monitoring and agricultural intelligent sensing, and particularly relates to a multi-modal information heterogeneous fusion crop phenotype information analysis and management method. BACKGROUND

[0002] In the field of crop monitoring and agricultural intelligent sensing, the commonly used sensing means mainly include hyperspectral images, visible light images and infrared images. The hyperspectral images can obtain fine spectral characteristics to identify crop physiological states and stress types. The visible light images can intuitively reflect the spatial form and growth structure of crops. The infrared images are sensitive to temperature and moisture conditions and can be used to characterize crop heat stress. Although each single modality has its own advantages, it can only cover part of the three key information of "spectrum-morphology-heat", and it is difficult to achieve comprehensive and accurate analysis of crop states, which may lead to one-sided and inaccurate identification of crop stress states such as water deficiency, nitrogen deficiency, diseases and pests.

[0003] Multi-modal information fusion technology is the mainstream research direction of current agricultural remote sensing intelligent phenotype decision. At present, it is mainly based on traditional deep learning framework, and fusion is carried out through feature splicing and attention mechanism. Although the accuracy can be improved, there are still some defects, such as the processing model used usually has large parameters and high calculation demand, which is difficult to deploy to edge devices with limited computing resources in the field, and lacks the ability to dynamically adjust according to different stress types, data quality and environmental conditions. Such fusion mechanism cannot effectively utilize the complementarity and redundancy between multi-element information. SUMMARY

[0004] In view of the above problems in the prior art, the multi-modal information heterogeneous fusion crop phenotype information analysis and management method provided by the application solves the problems of information limitation and lack of adaptability and flexibility in crop state analysis.

[0005] In order to achieve the above-mentioned application purposes, the technical scheme adopted by the application is as follows:

[0006] The multi-modal information heterogeneous fusion crop phenotype information analysis and management method provided by the application comprises the following steps:

[0007] The method comprises the following steps:

[0008] S1, using a UAV control platform to plan a path for a target area, and using a UAV to synchronously collect hyperspectral image data, visible light image data and infrared image data of crops along the planned path;

[0009] S2, respectively pre-processing the hyperspectral image data, visible light image data and infrared image data of crops to obtain multi-modal image data;

[0010] S3, feature extraction is performed on the multi-modal image data to obtain multi-modal image data classification features;

[0011] S4, according to the direct encoding method, the multi-modal image data classification features are mapped into the target firing rate of the spiking neurons in the spiking neural network, and through dynamic adjustment of the connection strength between the spiking neurons and Kalman filtering, the optimized multi-modal spiking signals are obtained;

[0012] S5, real-time environmental data are obtained, and combined with the optimized multi-modal spiking signals, brain confidence assessment and weight distribution are performed, and through weighted fusion and maximum probability selection, crop phenotype information analysis results are obtained;

[0013] S6, the crop phenotype information analysis results are superimposed on the corresponding position of the farmland remote sensing map, and the stress type and degree are marked, and the stress management scheme is automatically generated according to the stress rule base.

[0014] Further, the S1 comprises the following steps:

[0015] S11, the hyperspectral image sensor, the visible light image sensor and the infrared image sensor are carried on the unmanned aerial vehicle, and the hyperspectral image sensor, the visible light image sensor and the infrared image sensor are set to be in a synchronous acquisition state;

[0016] S12, the target area is path planned by using the unmanned aerial vehicle control platform to obtain a planned path;

[0017] S13, the unmanned aerial vehicle is driven to collect the hyperspectral image data, the visible light image data and the infrared image data of the crops by using the hyperspectral image sensor, the visible light image sensor and the infrared image sensor along the planned path.

[0018] Further, the S2 comprises the following steps:

[0019] S21, the hyperspectral image data of the crops are sequentially subjected to radiation correction, invalid noise band elimination, dimension reduction and filtering to obtain preprocessed hyperspectral image data;

[0020] S22, the visible light image data of the crops are subjected to color correction to obtain preprocessed visible light image data;

[0021] S23, the infrared image data of the crops are sequentially subjected to non-uniform correction, radiation calibration, thermal drift compensation and contrast enhancement to obtain preprocessed infrared image data;

[0022] S24, the preprocessed hyperspectral image data, visible light image data and infrared image data are aligned at the same time step to form multi-modal image data.

[0023] Further, the S3 comprises the following steps:

[0024] S31, based on the improved compression and excitation network, 3D convolutional neural network, self-attention mechanism Transformer, global average pooling layer, fully connected layer and softmax function, the feature extraction of the preprocessed hyperspectral image data in the multi-modal image data is carried out, and the hyperspectral image data classification feature is obtained;

[0025] S32, combined with the improved compression and excitation network, ResNet-50, attention mechanism ECA-Net, global average pooling layer, fully connected layer and softmax function, the feature extraction of the preprocessed visible light image data in the multi-modal image data is carried out, and the visible light image data classification feature is obtained;

[0026] S33, combined with the improved compression and excitation network, ResNet-50, attention mechanism ECA-Net, and global average pooling layer, the feature extraction of the preprocessed infrared image data in the multi-modal image data is carried out, and the infrared image data classification feature is obtained;

[0027] S34, based on the pulse timing dependent plasticity algorithm, the hyperspectral image data classification feature, the visible light image data classification feature and the infrared image data classification feature after Kalman filtering are time-synchronized and space-time aligned, and the noise is suppressed through Kalman filtering algorithm, and the multi-modal image data classification feature is obtained.

[0028] Further, the S31 comprises the following steps:

[0029] S311, the improved compression and excitation network is used to identify and amplify the wave band related to crop stress in the preprocessed hyperspectral image data, and irrelevant wave bands are filtered out, and the crop stress weighted hyperspectral image feature is obtained;

[0030] S312, the 3D convolutional neural network is used to repeatedly perform 3D convolution, activation and global average pooling on the crop stress weighted hyperspectral image feature, and the correlation weight between all feature points is directly calculated combined with the self-attention mechanism Transformer, and a one-dimensional vector of the hyperspectral image data is obtained;

[0031] S313, the one-dimensional vector of the hyperspectral image data is mapped to the feature dimension by using the fully connected layer, and the softmax function is inputted, and the hyperspectral image data classification feature is obtained.

[0032] Further, the S32 comprises the following steps:

[0033] S321, analyze the global color distribution of the preprocessed visible light image data using the improved compression and excitation network, and call the embedded knowledge base to analyze the color channel characteristics, and obtain the visible light image features weighted by crop stress;

[0034] S322, use the ResNet-50 with the attention mechanism ECA-Net to first perform convolution, pooling and down-sampling on the visible light image features weighted by crop stress to obtain low-level biophysical features, and then perform feature combination and multiple down-sampling feature extraction through the residual block to obtain enhanced visible light image features;

[0035] S323, compress the enhanced visible light image features using the global average pooling layer to obtain global visible light image features;

[0036] S324, map the global visible light image features to the category dimension through the fully connected layer, and input the softmax function to obtain the visible light image data classification features.

[0037] Further, the S33 comprises the following steps:

[0038] S331, according to the crop thermal physiological mechanism, analyze the overall temperature distribution of the preprocessed infrared image data using the improved compression and excitation network, and perform spatial adaptive weighting based on the embedded knowledge base to obtain the thermal signal features weighted by crop stress;

[0039] S332, according to the mean replication method, expand the thermal signal features weighted by crop stress into a three-channel pseudo RGB image, and use the ResNet-50 with the attention mechanism ECA-Net to focus on the stress-related thermal response area to extract enhanced infrared image data features;

[0040] S333, compress the enhanced infrared image data features using the global average pooling layer to obtain global infrared image data features;

[0041] S334, map the global infrared image data features to the category dimension through the fully connected layer, and input the softmax function to obtain the infrared image data classification features.

[0042] Further, the S4 comprises the following steps:

[0043] S41, according to the direct encoding method, map the hyperspectral image data classification features, visible light image data classification features and infrared image data classification features in the multi-modal image data classification features to the target firing rate of the corresponding pulse neuron in the pulse neural network, respectively;

[0044] S42, generating a multi-modal pulse signal through a Poisson process according to a target firing rate of the pulse neuron;

[0045] S43, dynamically adjusting connection strength between the pulse neurons based on a timing correlation between the multi-modal pulse signals according to a pulse timing-dependent plasticity learning rule, to obtain a spatio-temporally aligned multi-modal pulse signal;

[0046] S44, performing recursive optimal estimation on the spatio-temporally aligned multi-modal pulse signal according to a Kalman filtering algorithm, to obtain an optimized multi-modal pulse signal.

[0047] Further, the S5 comprises the following steps:

[0048] S51, acquiring real-time environmental data, and extracting initial decision probability vectors of a hyperspectral image modality, a visible light image modality and an infrared image modality based on the optimized multi-modal pulse signal, to constitute a current stress type and an environmental context;

[0049] S52, inputting the current stress type and the environmental context into a fuzzy rule base to perform brain confidence evaluation, and dynamically generating a weight distribution scheme of each modality through a meta-weight network;

[0050] S53, performing weighted fusion on phenotypic decisions corresponding to the hyperspectral image modality, the visible light image modality and the infrared image modality respectively according to the weight distribution scheme based on a weighted fusion algorithm, to obtain a multi-modal phenotypic information probability distribution;

[0051] S54, selecting a decision corresponding to a class with the highest probability as a crop phenotypic information analysis result according to the multi-modal phenotypic information probability distribution.

[0052] Further, the S6 comprises the following steps:

[0053] S61, superimposing corresponding labels of crop stress categories onto a farmland remote sensing map according to the crop phenotypic information analysis result, and labeling stress types and degrees, to obtain a crop stress distribution map;

[0054] S62, traversing each different category of crop stress regions in the crop stress distribution map, and performing matching reasoning according to crop stress types based on a stress rule base, to generate operation management schemes for different regions, and to constitute a stress management scheme.

[0055] The application has the advantages that the method for analyzing and managing crop phenotype information based on multi-modal information heterogeneous fusion provided by the application can realize the advantages of retaining different modal information, obtaining spectral information, capturing spatial texture and thermal characteristics, effectively making up for the deficiency of single modal information, and comprehensively analyzing the crop state by synchronously and jointly acquiring hyperspectral image data, visible light image data and infrared image data; the application introduces a brain-like dynamic attention weight distribution mechanism, can dynamically adjust the weight of each modal according to different stress types and environmental context by simulating the confidence evaluation of the brain on multi-source information, improves the self-adaptive ability of the scheme in a complex farmland environment, realizes the contextualization and intelligentization of the stress management scheme, and significantly improves the accuracy and practicality of the crop phenotype information analysis and management.

[0056] Other advantages of the application will be analyzed in more detail in subsequent examples. DETAILED DESCRIPTION

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed to be used in the embodiments will be briefly introduced as follows. It should be understood that the following drawings only show some of the embodiments of the application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0058] Figure 1 The step flow chart of the method for analyzing and managing crop phenotype information based on multi-modal information heterogeneous fusion in the embodiments of the application.

[0059] Figure 2 The structural block diagram of the improved compression and excitation network in the embodiments of the application. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the application will be described clearly and completely in combination with the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, but not all the embodiments. The components of the embodiments of the application described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the application provided in the drawings is not intended to limit the scope of the claimed application, but only represents selected embodiments of the application. Based on the embodiments of the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of the application.

[0061] As Figure 1As shown, in one embodiment of the present application, the present application provides a multi-modal information heterogeneous fusion crop phenotype information analysis and management method, comprising the following steps:

[0062] S1, using a UAV control platform to plan a path for a target area, and using a UAV to synchronously collect hyperspectral image data, visible light image data and infrared image data of crops along the planned path;

[0063] The S1 comprises the following steps:

[0064] S11, mounting a hyperspectral image sensor, a visible light image sensor and an infrared image sensor on a UAV, and setting the hyperspectral image sensor, the visible light image sensor and the infrared image sensor in a synchronous collection state;

[0065] In this embodiment, a synchronization module is set as a unified sensor instruction source to ensure the synchronous collection of the hyperspectral image sensor, the visible light image sensor and the infrared image sensor. A high-precision clock inside the synchronization module generates a short electrical pulse signal, which is synchronously sent to external trigger interfaces of the spectral image sensor, the visible light image sensor and the infrared image sensor through different lines, so that the three sensors in a continuous waiting state for listening to external commands immediately respond and synchronously perform exposure and data collection work at the moment of receiving the same pulse signal, achieving time-synchronized collection work.

[0066] In this scheme, the hyperspectral image sensor, the visible light image sensor and the infrared image sensor capable of synchronous collection are mounted on the UAV, which can synchronously perform exposure and data collection work.

[0067] S12, using a UAV control platform to plan a path for a target area, to obtain a planned path;

[0068] S13, driving a UAV to synchronously collect hyperspectral image data, visible light image data and infrared image data of crops along the planned path using a hyperspectral image sensor, a visible light image sensor and an infrared image sensor.

[0069] S2, respectively pre-processing the hyperspectral image data, the visible light image data and the infrared image data of crops to obtain multi-modal image data;

[0070] The S2 comprises the following steps:

[0071] S21, sequentially performing radiation correction, invalid noise band elimination, dimension reduction and filtering on the hyperspectral image data of crops to obtain pre-processed hyperspectral image data;

[0072] In this embodiment, the hyperspectral image data is a three-dimensional data, which not only contains two-dimensional spatial information, but also contains one-dimensional spectral information; when the hyperspectral image data is preprocessed, first, radiation correction is performed to eliminate radiation distortion caused by factors such as the sensor itself, the atmosphere and solar radiation; then, invalid noise bands are removed to remove low signal-to-noise ratio data; then, principal component analysis is further used for spectral dimension reduction to compress the data volume and reduce the redundancy between bands; finally, the spectrum is smoothed by a Savitzky-Golay filter to achieve the suppression of random noise while retaining the key spectral absorption and reflection characteristics.

[0073] S22, color correction is performed on the visible light image data of the crop to obtain preprocessed visible light image data;

[0074] In this embodiment, the visible light image data is the most common and most intuitive visual information expression form, and most of the colors that can be perceived by the human eye can be simulated and presented through the superposition of the three color channels of red, green and blue; color correction is performed on the visible light image data to ensure the accuracy and consistency of the color, and the illumination and contrast adjustment highlights the details and levels of the image.

[0075] S23, the infrared image data of the crop is sequentially subjected to non-uniform correction, radiation calibration, thermal drift compensation and contrast enhancement to obtain preprocessed infrared image data;

[0076] In this embodiment, the infrared image data is a spatial distribution map of the thermal radiation intensity of the object surface, which directly reflects the radiation capacity of the object in the infrared waveband, and the radiation capacity is mainly through the surface temperature and the emissivity; the infrared image data is subjected to non-uniform correction and radiation calibration to ensure the physical accuracy and measurability of the data, the thermal drift compensation ensures the time stability of the data, and the contrast enhancement optimizes the feature expression ability of the infrared image data.

[0077] S24, the preprocessed hyperspectral image data, visible light image data and infrared image data are aligned according to the same time step to form multi-modal image data.

[0078] S3, feature extraction is performed on the multi-modal image data to obtain multi-modal image data classification features;

[0079] The S3 includes the following steps:

[0080] S31, based on the improved compression and excitation network, the 3D convolutional neural network, the self-attention mechanism Transformer, the global average pooling layer, the full connection layer and the softmax function, feature extraction is performed on the preprocessed hyperspectral image data in the multi-modal image data to obtain hyperspectral image data classification features;

[0081] The S31 includes the following steps:

[0082] S311, identifying and amplifying the wave bands related to crop stress in the preprocessed hyperspectral image data using the improved compression and excitation network, and filtering out irrelevant wave bands to obtain crop stress weighted hyperspectral image features;

[0083] As shown in Figure 2 The improved compression and excitation network includes a global average pooling layer, an excitation multi-layer perceptron connected with the global average pooling layer, a bottleneck structure and a non-bottleneck structure connected with the excitation multi-layer perceptron respectively, a weight vector layer connected with the bottleneck structure and the non-bottleneck structure, and a Scale layer connected with the weight vector layer.

[0084] The bottleneck structure includes a fully connected layer, a ReLu activation function layer, a fully connected layer and a Sigmoid activation function layer connected in sequence.

[0085] The non-bottleneck structure includes a fully connected layer and a Sigmoid activation function layer connected with the fully connected layer.

[0086] In this scheme, after the preprocessed hyperspectral image data, visible light image data and infrared image data in the multi-modal image data are input into the global evaluation pooling layer, the improved compression and excitation network automatically selects the bottleneck structure or the non-bottleneck structure according to the channel number of the input data without human intervention. The channel number of the preprocessed hyperspectral image data is generally dozens to hundreds, which belongs to high-dimensional data, and needs to be reduced through the bottleneck structure to reduce the calculation complexity; the preprocessed visible light image data is 3 channels, and the preprocessed infrared image data is a single channel, which all belong to low-dimensional data. If the bottleneck structure is selected, the key features such as the color channel of the visible light image data and the temperature channel of the infrared image data will be lost, so the non-bottleneck structure is automatically selected to retain the original dimension.

[0087] The effect difference between the splicing structure and the non-bottleneck structure is significant in calculation efficiency, channel calibration effect and applicable target. The bottleneck structure first convolves and reduces the dimension, then calibrates the positive bias, and then restores the dimension by convolution, which can reduce the parameter amount and calculation amount, suppress the redundant wave bands of the hyperspectral image, focus on screening the key wave bands in the high-dimensional spectrum in the channel calibration effect, and is more obvious in the suppression of redundancy, and in the applicable target, the bottleneck structure solves the dimension disaster of high-dimensional data and improves the training efficiency of the model.

[0088] In the scheme, the improved compression and excitation network processes and focuses the spectrum according to prior knowledge through exciting the multi-layer perception, uses one-dimensional convolution to browse the whole spectrum, first performs overall perception on the health status of the crop to determine whether the spectrum of the crop is more inclined to water deficiency, nitrogen deficiency or disease as a whole, then calls the built-in crop physiological knowledge base to compare the spectral features, such as the key of water stress is the absorption valley of water molecules, nitrogen deficiency stress causes chlorophyll change and thus causes red edge blue movement, and specific diseases form unique spectral texture, so as to intelligently determine the most possible stress type and allocate attention weight, if it is determined that the crop is more likely to be water deficient, the water feature band is given a higher weight, so as to significantly enhance the most relevant features of the suspected stress and weaken irrelevant information. The prior knowledge refers to known laws and professional information embedded in the system in advance based on crop physiological characteristics, modal data laws and model design experience, which is used to guide the model to focus on key features more accurately, and can help the model quickly locate key information in complex data and improve the accuracy of discrimination.

[0089] The traditional 2D convolutional neural network can only perform convolution in the spatial dimension, while the 3D convolutional neural network can simultaneously perform convolution in space and spectrum and extract joint features of space and spectrum; the 3D convolution kernel of the 3D convolutional neural network slides in three dimensions at the same time, extracts the spatial neighborhood of each pixel point, and can capture the correlation between a small spatial region and adjacent spectral bands; the 3D convolution kernel of the 3D convolutional neural network has a dimension of wherein, is a first spatial receptive field factor, is a second spatial receptive field factor, is a spectral receptive field factor; the 3D convolution kernel can directly learn the change mode of the spectral curve in the local spatial context through the convolution operation of the 3D convolution kernel;

[0090] The 3D convolutional neural network is limited by the size and stacking number of the convolution kernel, and it is difficult to capture long-distance and non-local spatio-temporal correlation, so the self-attention mechanism Transformer is used to directly calculate the correlation weight between all feature points, and the combination of the two realizes the cooperation of local fine perception and global correlation modeling.

[0091] In the embodiment, the crop stress weighted hyperspectral image features include spatial size, band number, normalized water index, nitrogen index, spectral curve shape parameter and other feature indexes;

[0092] In S312, the 3D convolutional neural network is used to perform repeated 3D convolution, activation and global average pooling on the crop stress weighted hyperspectral image features, and the self-attention mechanism Transformer is used to directly calculate the correlation weight between all feature points, to obtain a one-dimensional vector of the hyperspectral image data;

[0093] The calculation expression of 3D convolution and activation of the crop stress weighted hyperspectral image feature is as follows:

[0094] ,

[0095] Among them, represents the 3D convolution activation feature of the hyperspectral image of the first layer, represents a nonlinear activation function, represents the 3D convolution kernel weight parameter of the first layer, represents the crop stress weighted hyperspectral image feature, represents the 3D convolution kernel bias parameter of the first layer, represents the first layer of the 3D convolutional neural network, represents the height position of the crop in the hyperspectral image, represents the width position of the crop in the hyperspectral image, represents the spectral feature value at the waveband, represents the offset amount of the height position of the crop in the hyperspectral image, represents the offset amount of the width position of the crop in the hyperspectral image, represents the offset amount of the spectral feature value of the crop at the waveband, represents the total amount of the height position of the crop in the hyperspectral image that can be offset, represents the total amount of the width position of the crop in the hyperspectral image that can be offset, represents the total amount of the spectral feature value at the waveband that can be offset; in the present scheme, the 3D convolution feature extraction introduces the result after convolution into a nonlinear transformation, enhancing the expression ability of the feature;

[0096] Through the stacking of multiple layers of 3D convolution and activation, a three-dimensional feature tensor containing spectral dimension information, spatial dimension information, and global correlation information can be extracted, wherein the spectral dimension information includes the reflectivity, absorption rate and other spectral response data of the crop, the spatial dimension information includes the spatial form, texture details and the like of the crop leaves, plants and the like, and the global average pooling is compressed into a one-dimensional vector, that is, the hyperspectral image data feature;

[0097] The calculation expression of the global average pooling is as follows:

[0098] ,

[0099] Among them, represents the hyperspectral image data feature, represents the height of the hyperspectral image, a width of hyperspectral image data, a depth of hyperspectral image data, 3D convolutional activation features of the hyperspectral image; the 3D convolutional activation features of the hyperspectral image are compressed into one-dimensional vector features after global average pooling, providing a basis for further feature prediction.

[0100] S313, mapping the one-dimensional vector of the hyperspectral image data to the feature dimension by using the full connection layer, and inputting the softmax function to obtain the hyperspectral image data classification feature.

[0101] S32, combining the improved compression and excitation network, ResNet-50, attention mechanism ECA-Net, global average pooling layer, full connection layer and softmax function, the visible light image data in the preprocessed multi-modal image data is extracted to obtain the visible light image data classification feature;

[0102] The S32 comprises the following steps:

[0103] S321, using the improved compression and excitation network to analyze the global color distribution of the preprocessed visible light image data, and calling the embedded knowledge base to analyze the color channel characteristics, to obtain the crop stress weighted visible light image feature;

[0104] In this scheme, by analyzing the global color distribution, the health status of the crop can be preliminarily judged, and then the embedded knowledge base is used to analyze the color channel characteristics according to the global color distribution, so as to accurately determine the crop stress situation. For example, when there is water stress, there is no strong response of specific channel, when there is nitrogen deficiency stress, it will cause chlorophyll degradation and leaf yellowing phenomenon, which is specifically manifested as significant increase of red channel reflectance and characteristic decrease of green channel reflectance, and when there is specific disease infection, the characteristics of the color channel are constructed according to the specificity of the disease, for example, the red-brown spots of rust disease have response characteristics in the red channel. By selectively enhancing the color information related to the crop stress situation, the crop stress weighted visible light image feature is obtained, which can provide more discriminative features for subsequent analysis and processing.

[0105] S322, using the ResNet-50 with the attention mechanism ECA-Net to first perform convolution, pooling and down-sampling on the crop stress weighted visible light image feature to obtain low-level biophysical features, and then performing feature combination and multiple down-sampling feature extraction through the residual block to obtain the enhanced visible light image feature.

[0106] In the scheme, the low-level biophysical features include leaf edges, textures, green pigment distribution and other features related to photosynthesis; the deep processing through the residual block includes four stages, in which, in the first stage, the low-level biophysical features are combined and transformed through three residual blocks, so that the features are more abundant, to preliminarily distinguish the uniform texture of healthy leaves, and the fine mottling, chlorosis or texture break caused by early infection of pests and diseases or nutrient deficiency, in the second stage, down-sampling is performed through four residual blocks, so that the spatial size is halved, the number of channels is doubled, and the receptive field of each feature point is larger, which can find larger areas in the original image, to combine the low-level biophysical features into crop organ components, identify the outline of a single leaf and petiole, and detect larger range of local stress symptoms such as leaf blight class or insect feeding holes, in the third stage, further down-sampling is performed through six residual blocks, the spatial information is further compressed, and the semantic is highly condensed, which can identify the local model of crops, i.e. the symptoms of different diseases, the overall wilting form of crops caused by water stress, or preliminarily judge the growth of crops according to the density of canopy structure, in the fourth stage, further down-sampling is performed through three residual blocks, so that the receptive field almost covers the entire original image, and the output semantic features can integrate the health status of the entire canopy, biomass estimation, etc.; in the scheme, the residual connection is used to ensure the effective propagation of gradient and avoid the degradation of deep network, and the step-by-step down-sampling constantly compresses the information in the spatial dimension and enriches the semantics in the channel dimension, realizing the smooth transition from pixel-level details to high-level semantics and multiple feature extraction.

[0107] The attention mechanism ECA-Net is a lightweight and efficient channel attention mechanism, which is embedded into ResNet-50 in the scheme in a plug-and-play manner, so that ResNet-50 can automatically focus on and enhance the channels more important for the current classification task, such as near-infrared reflection features closely related to chlorophyll content, specific channels indicating water stress, unimportant and redundant channels; setting the attention mechanism ECA-Net makes different health conditions and nutrient levels of crops present key spectral differences in the RGB channel, such as the decrease of chlorophyll content changing the reflectivity of the green channel, and the attention mechanism ECA-Net can adaptively learn and give higher weights to the spectral channels most relevant to the current classification task, thereby amplifying the subtle signs of disease and suppressing the interference of irrelevant information such as soil background, shadow and light changes; different convolution kernels in the channel dimension of the feature map will extract different types of features, and the attention mechanism ECA-Net can effectively enhance the feature channels most indicative of the target disease by recalibrating the channel weights, so that the network can focus more on these key disease areas, thereby improving the ability of ResNet-50 to distinguish similar symptoms.

[0108] S323, compress the enhanced visible light image features by using a global average pooling layer to obtain global visible light image features;

[0109] S324, map the global visible light image features to a category dimension through a fully connected layer, and input a softmax function to obtain visible light image data classification features.

[0110] In this scheme, the probability distribution corresponding to the visible light image data features intuitively reflects the model's comprehensive phenotype decision result for crops, highlights the specific correlation between the probability distribution and the crop state, and provides a quantitative basis for decision-making.

[0111] S33, combined with the improved compression and excitation network, ResNet-50, attention mechanism ECA-Net, and global average pooling layer, feature extraction is performed on the preprocessed infrared image data in the multi-modal image data to obtain infrared image data classification features;

[0112] The S33 includes the following steps:

[0113] S331, according to the crop thermal physiological mechanism, the improved compression and excitation network is used to analyze the overall temperature distribution of the preprocessed infrared image data, and spatial adaptive weighting is performed based on the embedded knowledge base to obtain crop stress weighted thermal signal features;

[0114] In this embodiment, the thermal signal features obtained by analyzing the overall temperature distribution of the preprocessed infrared image data can preliminarily reflect the status of crop water, such as normal transpiration, water stress or potential disease; then the improved compression and excitation network dynamically adjusts the spatial attention weight according to the embedded knowledge base for crop stress; in this embodiment, the embedded knowledge base can dynamically adjust the spatial attention weight according to the systematic increase of canopy temperature caused by stomatal closure due to water stress, and the local abnormal hot spot caused by specific diseases; when detecting that the crop is under drought stress, the improved compression and excitation network automatically enhances the response strength of the medium and high temperature area, highlighting the overall temperature rise pattern of the canopy; when detecting that the crop is under disease stress, the improved compression and excitation network focuses on enhancing the local thermal anomaly features; through the adaptive weighting mechanism, the improved compression and excitation network effectively amplifies the key thermal features related to stress and suppresses environmental noise interference.

[0115] S332, according to the mean replication method, the crop stress weighted thermal signal features are expanded into three-channel pseudo RGB images, and the ResNet-50 with the attention mechanism ECA-Net is used to focus on the thermal response area related to stress to obtain enhanced infrared image data features;

[0116] In the scheme, since ResNet-50 requires input of three channels, the crop stress weighted thermal signal is expanded into a three-channel pseudo RGB image, which can adapt to the ResNet-50 network architecture while retaining the temperature information of the infrared image data;

[0117] S333, compress the enhanced infrared image data features by using a global average pooling layer to obtain global infrared image data features;

[0118] In the embodiment, the global infrared image data features comprehensively encode the thermal anomaly intensity and spatial distribution pattern of the entire crop canopy;

[0119] S334, map the global infrared image data features to the category dimension through a fully connected layer, and input a softmax function to obtain infrared image data classification features.

[0120] S34, based on the pulse timing dependent plasticity algorithm, time synchronizes and spatiotemporally aligns the hyperspectral image data classification features, visible light image data classification features and infrared image data classification features after Kalman filtering, and suppresses noise through Kalman filtering algorithm to obtain multi-modal image data classification features.

[0121] S4, according to the direct encoding method, map the multi-modal image data classification features to the target firing rate of the pulse neurons in the pulse neural network, and obtain the optimized multi-modal pulse signal by dynamically adjusting the connection strength between the pulse neurons and Kalman filtering;

[0122] The S4 includes the following steps:

[0123] S41, according to the direct encoding method, map the hyperspectral image data classification features, visible light image data classification features and infrared image data classification features in the multi-modal image data classification features to the target firing rate of the corresponding pulse neurons in the pulse neural network, respectively;

[0124] In the embodiment, according to the direct encoding method, the water sensitive index, the erythema position and other feature values in the hyperspectral image data classification features, the lesion area, the G / R ratio and other feature values in the visible light image data classification features, and the leaf temperature and other feature values in the infrared image data classification features can be corresponded to the target firing rate of the pulse neurons. The higher the feature value is, the higher the target firing rate of the corresponding pulse neuron is, otherwise the lower the feature value is, the lower the target firing rate of the corresponding pulse neuron is.

[0125] S42, generate the multi-modal pulse signal through the Poisson process according to the target firing rate of the pulse neuron;

[0126] In this embodiment, in a unit time, the pulse neuron randomly generates a pulse signal according to the probability of the target firing rate. The higher the target firing rate corresponding to the characteristic value, the more pulse signals generated in a unit time, thereby converting the continuous characteristic number of each modality into a discrete pulse sequence that can be processed by the pulse neural network, that is, a multi-modal pulse signal.

[0127] S43, according to the pulse timing-dependent plasticity learning rule, based on the timing correlation between the multi-modal pulse signals, dynamically adjusting the connection strength between the pulse neurons, obtaining the spatio-temporal aligned multi-modal pulse signals;

[0128] In this embodiment, when the moisture feature neuron corresponding to the classification feature of the hyperspectral image data and the high temperature feature neuron corresponding to the classification feature of the infrared image data frequently fire at the same time, the pulse timing-dependent plasticity algorithm will continuously enhance the connection weight between the moisture feature neuron and the high temperature feature neuron, thereby autonomously forming a joint representation of low moisture and high temperature.

[0129] The calculation expression of the pulse timing-dependent plasticity learning rule is as follows:

[0130] ,

[0131] Wherein, represents the change amount of the connection weight between the pre-synaptic first pulse neuron and the post-synaptic second pulse neuron, represents the change amount of the connection weight between the pre-synaptic first pulse neuron and the post-synaptic second pulse neuron, represents the change amount of the connection weight between the pre-synaptic first pulse neuron and the post-synaptic second pulse neuron, represents the first pulse timing-dependent plasticity learning rate, represents the time difference between the pre-synaptic first pulse neuron and the post-synaptic second pulse neuron, represents the first time constant, represents the second pulse timing-dependent plasticity learning rate, represents the second time constant;

[0132] S44, according to the Kalman filtering algorithm, recursively optimally estimating the spatio-temporal aligned multi-modal pulse signals, obtaining the optimized multi-modal pulse signals.

[0133] In this scheme, since the generation and transmission of pulse sequences contain noise, the Kalman filtering algorithm is introduced to recursively optimally estimate the pulse firing activity of the pulse neuron, thereby effectively suppressing the random noise in the pulse transmission process, to ensure that in the complex environment in the field, the real stress characteristics such as lack, nitrogen deficiency and disease can still be stably transmitted to the decision layer, and efficient, robust multi-modal information unified processing and integration can be realized under the low-power pulse paradigm, providing a reliable feature basis for subsequent brain-level decision-making.

[0134] S5, acquire real-time environment data, combine the optimized modal pulse signals, perform simulated brain confidence evaluation and weight distribution, and obtain crop phenotype information analysis results through weighted fusion and maximum probability selection;

[0135] The S5 includes the following steps:

[0136] S51, acquire real-time environment data, and extract initial decision probability vectors of hyperspectral image modal, visible light image modal and infrared image modal from the optimized modal pulse signals to form a current stress type and environment context;

[0137] In this embodiment, the real-time environment data includes real-time light conditions and atmospheric temperature and humidity, the initial decision probability vector of the hyperspectral image modal represents crop biochemical component abnormalities, the initial decision probability vector of the visible light image modal represents crop morphological structure and color changes, and the initial decision probability vector of the infrared image modal represents crop canopy temperature or transpiration changes;

[0138] S52, input the current stress type and environment context into a fuzzy rule base to perform simulated brain confidence evaluation, and dynamically generate a weight distribution scheme for each modal through a meta-weight network;

[0139] In this scheme, the meta-weight network includes an input layer, a fully connected layer, a ReLu activation function layer, a fully connected layer and a softmax layer connected in sequence; in this embodiment, the fuzzy rule base is used for simulated brain confidence evaluation, can perform fuzzy reasoning on the weight of each modal according to the current stress type and environment context, so that the modal most suitable for the current environment and stress type occupies the highest weight; when generating the initial weight distribution scheme, the fuzzy rule base dynamically evaluates the reliability of each image modal in the current environment based on prior agricultural knowledge, real-time environment data and the confidence of each modal, wherein the fuzzy rules corresponding to the prior agricultural knowledge are, for example, that for early crop diseases, spectral analysis based on hyperspectral image data classification features should be more reliable, and for determining drought, thermal imaging of infrared image data should be given priority; the fuzzy rules corresponding to the real-time environment data are, for example, that under strong light at noon, visible light images are prone to overexposure, so the weight corresponding to the visible light image data classification features will be automatically adjusted lower, and under the condition of high air humidity, the weight corresponding to the infrared image data will be automatically adjusted higher; the fuzzy rules corresponding to the confidence of each modal are obtained by calculating the confidence of the output results of each image modal, once the confidence of a certain modal is low due to image blur or occlusion, etc., the influence of the modal will be weakened, thereby effectively preventing interference of low-quality data on the final decision.

[0140] S53, according to a weighted fusion algorithm, weighting fuse the phenotype decisions corresponding to the hyperspectral image modal, the visible light image modal and the infrared image modal respectively according to the weight distribution scheme to obtain a multi-modal phenotype information probability distribution;

[0141] In this embodiment, the calculation expression of the multi-modal phenotype information probability distribution is as follows:

[0142] ,

[0143] wherein, denotes the probability of the i-th stress category in the multi-modal phenotype information, denotes the weight of the i-th modality, denotes the probability that the phenotype of the i-th modality sample belongs to the j-th stress category; in this embodiment, when i = 1, it corresponds to the hyperspectral image modality; when i = 2, it corresponds to the visible light image modality; when i = 3, it corresponds to the infrared image modality; is a natural number, corresponding to different crop state categories, including stress categories such as diseases, droughts, and nutrient deficiencies; S54, according to the multi-modal phenotype information probability distribution, the decision corresponding to the category with the highest probability is selected as the crop phenotype information analysis result. In this embodiment, the calculation expression of the crop phenotype information analysis result is as follows: ,

[0144] wherein, denotes the crop phenotype information analysis result,

[0145] denotes the stress category with the maximum probability in the multi-modal phenotype information.

[0146] ,

[0147] wherein, denotes the crop phenotype information analysis result, denotes the stress category with the maximum probability in the multi-modal phenotype information.

[0148] S6, the crop phenotype information analysis result is superimposed on the corresponding position of the farmland remote sensing map, and the stress type and degree are marked, and a stress management scheme is automatically generated according to the stress rule base.

[0149] The S6 includes the following steps:

[0150] S61, according to the crop phenotype information analysis result, the corresponding identifier is superimposed on the farmland remote sensing map according to the crop stress category, and the stress type and degree are marked to obtain a crop stress distribution map;

[0151] ​​​In this embodiment, the crop phenotypic analysis results are transformed into an intuitive, spatialized farmland stress distribution map. The crop stress type and degree information obtained from multimodal fusion analysis is accurately overlaid onto the farmland remote sensing map using georeferencing technology. For example, areas corresponding to crop diseases are overlaid in red on the farmland remote sensing map, drought-related crop stress is overlaid in blue, and nutrient deficiency-related crop stress is overlaid in purple, thus obtaining the crop stress distribution map.

[0152] S62. Traverse the crop stress regions of different categories in the crop stress distribution map, and perform matching reasoning according to crop stress type based on the stress rule base to generate operation management plans for different regions, thus forming a stress management plan.

[0153] In this embodiment, the stress rule base can provide corresponding operation management schemes according to the type of crop stress in order to reduce or eliminate crop stress. Common operation management schemes include precision irrigation, precision fertilization, and precision pesticide application. By controlling the drone to operate according to the digital stress management scheme, precise agricultural operations can be achieved, thereby completely constructing a closed loop of precision agricultural management from integrated monitoring to intelligent decision-making and then to automatic execution.

[0154] In this embodiment, to ensure the system can adapt to changes in crop growing seasons, the emergence of pests and diseases, and regional differences, the provided crop phenotypic information analysis and management method based on multimodal information heterogeneous fusion has a fixed iteration cycle. It continuously collects new crop image data via drones and integrates farmer feedback to construct a rigorously cleaned and labeled incremental dataset. Then, based on the new data, various models in the scheme are evaluated to identify which new crops and regions exhibit poor performance under stress types, thus determining the optimization direction for that cycle. Finally, incremental training and A / B rolling testing are used to conduct online comparative experiments between the old and new model versions. After verifying effectiveness, updates are deployed. This closed-loop process continuously absorbs fresh samples and domain knowledge of crop stress management through rolling optimization, possessing the evolutionary ability to continuously adapt to new varieties and regional differences, enabling the agricultural system's adaptability to crops and regions to increase spirally.

[0155] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for analyzing and managing crop phenotype information based on multi-modal information heterogeneous fusion, characterized in that, Comprise the following steps: S1, using unmanned aerial vehicle control platform to target area path planning, and using unmanned aerial vehicle along the planned path synchronous collection of crop hyperspectral image data, visible light image data and infrared image data; S2, respectively on crop hyperspectral image data, visible light image data and infrared image data respectively are preprocessed, obtain multi modal image data; S3, the feature extraction of multi modal image data, obtain multi modal image data classification feature; S4, according to direct encoding method multi modal image data classification feature is mapped into the target firing rate of pulse neuron in pulse neural network, and through the dynamic adjustment of the connection strength between pulse neurons and kalman filtering, obtain the optimized multi modal pulse signal; The S4 comprises the following steps: S41, according to direct encoding method, the hyperspectral image data classification feature, visible light image data classification feature and infrared image data classification feature in multi modal image data classification feature are respectively mapped into the target firing rate of corresponding pulse neuron in pulse neural network; S42, according to the target firing rate of pulse neuron, through poisson process generates multi modal pulse signal; S43, according to the timing dependence plasticity learning rule, based on the timing correlation between multi modal pulse signals, the connection strength between pulse neurons is dynamically adjusted, and the spatiotemporal alignment multi modal pulse signal is obtained; S44, according to kalman filtering algorithm, the recursive optimal estimation of spatiotemporal alignment multi modal pulse signal is carried out, and the optimized multi modal pulse signal is obtained; S5, obtain real-time environmental data, and combine the optimized modal pulse signal, carry out simulated brain confidence evaluation and weight distribution, and through weighted fusion and maximum probability selection, obtain crop phenotype information analysis result; The S5 comprises the following steps: S51, obtain real-time environmental data, and extract initial decision probability vector of hyperspectral image mode, visible light image mode and infrared image mode based on the optimized modal pulse signal, constitute current stress type and environmental context; S52, the current stress type and environmental context are input into fuzzy rule base to carry out simulated brain confidence evaluation, and the weight distribution scheme of each mode is dynamically generated through meta weight network; S53, according to weighted fusion algorithm, the phenotype decision corresponding to hyperspectral image mode, visible light image mode and infrared image mode is weighted fused according to weight distribution scheme, and multi modal phenotype information probability distribution is obtained; S54, according to multi modal phenotype information probability distribution, the decision corresponding to the class with the highest probability is selected as crop phenotype information analysis result; S6, the crop phenotype information analysis result is superimposed on the corresponding position of farmland remote sensing map, and the stress type and degree are marked, and the stress management scheme is automatically generated according to stress rule base. 2.The method for analyzing and managing crop phenotyping information based on multi-modal information heterogeneous fusion according to claim 1, characterized in that, The S1 comprises the following steps: S11, the hyperspectral image sensor, visible light image sensor and infrared image sensor are carried on the unmanned aerial vehicle, and the hyperspectral image sensor, visible light image sensor and infrared image sensor are set to be in synchronous collection state; S12, path planning of the target area is performed by using the unmanned aerial vehicle control platform to obtain a planned path; S13, the unmanned aerial vehicle is driven to synchronously collect hyperspectral image data, visible light image data and infrared image data of the crops along the planned path by using the hyperspectral image sensor, the visible light image sensor and the infrared image sensor. 3.The method for analyzing and managing crop phenotyping information based on multi-modal information heterogeneous fusion according to claim 2, characterized in that, The S2 comprises the following steps: S21, the hyperspectral image data of the crops are sequentially subjected to radiation correction, invalid noise band elimination, dimension reduction and filtering to obtain preprocessed hyperspectral image data; S22, the visible light image data of the crops are subjected to color correction to obtain preprocessed visible light image data; S23, the infrared image data of the crops are sequentially subjected to non-uniform correction, radiation calibration, thermal drift compensation and contrast enhancement to obtain preprocessed infrared image data; S24, the preprocessed hyperspectral image data, the preprocessed visible light image data and the preprocessed infrared image data are aligned at the same time step to form multi-modal image data. 4.The method for analyzing and managing crop phenotyping information based on multi-modal information heterogeneous fusion according to claim 3, characterized in that, The S3 comprises the following steps: S31, the preprocessed hyperspectral image data in the multi-modal image data are subjected to feature extraction based on the improved compression and excitation network, the 3D convolutional neural network, the self-attention mechanism Transformer, the global average pooling layer, the full connection layer and the softmax function to obtain hyperspectral image data classification features; S32, the preprocessed visible light image data in the multi-modal image data are subjected to feature extraction in combination with the improved compression and excitation network, ResNet-50, the attention mechanism ECA-Net, the global average pooling layer, the full connection layer and the softmax function to obtain visible light image data classification features; S33, the preprocessed infrared image data in the multi-modal image data are subjected to feature extraction in combination with the improved compression and excitation network, ResNet-50, the attention mechanism ECA-Net and the global average pooling layer to obtain infrared image data classification features; S34, the hyperspectral image data classification features, the visible light image data classification features and the infrared image data classification features after Kalman filtering are subjected to time synchronization and space-time alignment based on the pulse time-dependent plasticity algorithm, and noise is suppressed by the Kalman filtering algorithm to obtain multi-modal image data classification features.

5. The method for analyzing and managing crop phenotyping information based on multi-modal information heterogeneous fusion according to claim 4, characterized in that, The S31 comprises the following steps: S311, the preprocessed hyperspectral image data related to crop stress are identified and amplified by using the improved compression and excitation network, and irrelevant bands are filtered out to obtain crop stress weighted hyperspectral image features; S312, the crop stress weighted hyperspectral image features are subjected to repeated 3D convolution, activation and global average pooling by using the 3D convolutional neural network, and the correlation weight between all feature points is directly calculated in combination with the self-attention mechanism Transformer to obtain a one-dimensional vector of the hyperspectral image data; S313, the one-dimensional vector of the hyperspectral image data is mapped to a feature dimension by using the full connection layer, and the softmax function is inputted to obtain the hyperspectral image data classification features. 6.The method for analyzing and managing crop phenotyping information based on multi-modal information heterogeneous fusion according to claim 4, characterized in that, The S32 comprises the following steps: S321, analyze the global color distribution of the preprocessed visible light image data using the improved compression and excitation network, and call the embedded knowledge base to analyze the color channel characteristics, and obtain the visible light image features weighted by crop stress; S322, use the ResNet-50 with ECA-Net to first perform convolution, pooling and down-sampling on the visible light image features weighted by crop stress to obtain low-level biophysical features, and then perform feature combination and multiple down-sampling feature extraction through residual blocks to obtain enhanced visible light image features; S323, compress the enhanced visible light image features using the global average pooling layer to obtain global visible light image features; S324, map the global visible light image features to the category dimension through the fully connected layer, and input the softmax function to obtain the visible light image data classification features.

7. The method for analyzing and managing crop phenotyping information based on multi-modal information heterogeneous fusion according to claim 4, characterized in that, The S33 includes the following steps: S331, according to the crop thermal physiological mechanism, analyze the overall temperature distribution of the preprocessed infrared image data using the improved compression and excitation network, and perform spatial self-adaptive weighting based on the embedded knowledge base to obtain the thermal signal features weighted by crop stress; S332, according to the mean replication method, expand the thermal signal features weighted by crop stress into a three-channel pseudo RGB image, and use the ResNet-50 with ECA-Net to focus on the stress-related thermal response area to extract enhanced infrared image data features; S333, compress the enhanced infrared image data features using the global average pooling layer to obtain global infrared image data features; S334, map the global infrared image data features to the category dimension through the fully connected layer, and input the softmax function to obtain the infrared image data classification features. 8.The method for analyzing and managing crop phenotyping information based on multi-modal information heterogeneous fusion according to claim 1, characterized in that, The S6 includes the following steps: S61, according to the crop phenotypic information analysis result, superimpose the corresponding identifier to the farmland remote sensing map according to the crop stress category, and label the stress type and degree to obtain the crop stress distribution map; S62, traverse each different category of crop stress area in the crop stress distribution map, and match and reason according to the crop stress type based on the stress rule base to generate operation management schemes for different regions, and constitute the stress management scheme.

Images