An unmanned aerial vehicle multi-scale crop detection method based on a neural dynamics model

Abstract

The application discloses a kind of unmanned aerial vehicle multiscale crop detection methods based on neural dynamics model, belong to the technical field of crop detection, comprising: obtaining the visible light image of crop under unmanned aerial vehicle multiscale, long-wave infrared image and hyperspectral data;Visible light image, long-wave infrared image and hyperspectral data are registered and fused to obtain fused hyperspectral cube data;The neural dynamics model of crop detection is constructed, and fused hyperspectral cube data is input into the neural dynamics model of crop detection to train, and the crop detection result is output;Based on crop detection result, the loss function of the neural dynamics model of crop detection is calculated, and is fed back to the model to be constrained.The application combines visual processing, deep learning and agricultural scene perception technology, utilizes the parallel computing capability and anti-interference characteristic of neural dynamics model, realizes efficient, accurate crop monitoring.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of crop detection, and particularly relates to a multi-scale crop detection method based on a neural dynamics model. BACKGROUND

[0002] As a product of the deep integration of modern information technology and agricultural production, smart agriculture is gradually breaking through the extensive mode of traditional agriculture, which is "relying on the weather", and is taking "precision, intelligence and high efficiency" as the core to reconstruct the whole chain of agricultural production, management and decision-making. Multi-scale crop detection by unmanned aerial vehicles has become a key link in "precise perception of field information" in smart agriculture, and is not only an important means to solve the pain points of agricultural production, but also one of the core supports to promote the landing of smart agriculture.

[0003] In precision agriculture, crop detection mainly includes crop classification, crop growth monitoring, pest and disease identification, etc. Unmanned aerial vehicles have become an important data collection platform for crop detection due to their flexibility, efficiency and low cost. However, single-scale images often have one-sided information, such as difficulty in capturing diseases on the back of leaves and loss of details due to crop shading. Although multi-scale data can provide more comprehensive information, it faces challenges such as data heterogeneity (resolution, angle difference), dynamic interference (light change, crop growth), etc. SUMMARY

[0004] The purpose of the present application is to solve the problem of low precision and inability to perform multi-task parallelism in existing crop detection methods by providing a multi-scale crop detection method based on a neural dynamics model.

[0005] To achieve the above purpose, the technical solution adopted by the present application is as follows:

[0006] A multi-scale crop detection method based on a neural dynamics model includes the following steps:

[0007] S1, obtaining visible light images, long-wave infrared images and hyperspectral data of crops under multi-scale unmanned aerial vehicles;

[0008] S2, performing registration and fusion processing on the visible light images, long-wave infrared images and hyperspectral data to obtain fused hyperspectral cube data;

[0009] S3, constructing a neural dynamics model for crop detection, and inputting the fused hyperspectral cube data into the neural dynamics model for crop detection to train and output the crop detection result.

[0010] Further, S1 specifically includes:

[0011] Based on the unmanned aerial vehicle, visible light images, long-wave infrared images and hyperspectral data of crops of different categories, different periods, different water and nitrogen contents and different pest and disease degrees are acquired.

[0012] Different categories include corn, rice and wheat.

[0013] Different periods include the emergence stage, the heading stage and the mature stage.

[0014] Different water and nitrogen contents include no water deficiency, water deficiency, no nitrogen deficiency and nitrogen deficiency.

[0015] Different pest and disease degrees include no disease, light disease and severe disease.

[0016] Further, the S2 comprises the following steps:

[0017] S21, topographic correction, radiation correction and data splicing processing are performed on the hyperspectral data, and the 470nm blue, 550nm green and 650nm red three channels of the processed hyperspectral data are combined into an RGB composite image.

[0018] S22, the RGB composite image is taken as a registration reference image, and is registered with the visible light image and the long-wave infrared image respectively to obtain the registered visible light image and long-wave infrared image.

[0019] S23, image enhancement processing is performed on the registered visible light image and long-wave infrared image, and denoising and dimension reduction processing are performed on the hyperspectral data to obtain the image-enhanced visible light image and long-wave infrared image and the dimension-reduced hyperspectral data.

[0020] S24, the image-enhanced visible light image and long-wave infrared image and the dimension-reduced hyperspectral data are spliced and fused to obtain fused hyperspectral cube data.

[0021] Further, in the S23, adaptive local tone mapping is adopted to perform image enhancement processing on the registered visible light image, and the registered visible light image is converted to the YUV color space, wherein the brightness adjustment in the Y channel is represented as:

[0022]

[0023] In the formula, is a pixel point After brightness adjustment, the result is is a pixel point the input brightness value of the pixel point is the input maximum brightness value; is the logarithmic average value of the input brightness;

[0024] Based on the Y channel brightness adjustment, the color correction is performed on the registered visible light image, which is specifically represented as:

[0025]

[0026] In the formula, represents the correction value of the red channel of all pixel sets in the visible light image; represents the correction value of the blue channel of all pixel sets in the visible light image; represents the correction value of the green channel of all pixel sets in the visible light image; represents the red, blue and green channel values in the visible light image; represents the result of all pixel sets after brightness adjustment.

[0027] Further, in the S23, an adaptive double-threshold wavelet enhancement algorithm is used to perform image enhancement processing on the registered long-wave infrared image, which is specifically represented as:

[0028]

[0029] In the formula,

[0030]

[0031] In the formula, represents the long-wave infrared image after image enhancement; represents the inverse discrete wavelet transform; represents the adaptive gain adjustment on the low-frequency component to strengthen the low-frequency component after global thermal contrast processing; is the high-frequency component; is the low-frequency component; is the gain coefficient; is the mean value, is the standard deviation.

[0032] Further, in the S3, the crop detection neural dynamics model includes three branch convolutional neural networks and a graph convolutional network.

[0033] The fused hyperspectral cube data is input into the crop detection neural dynamics model for training, including the following steps:

[0034] ​​​​​​​​​​S31. The fused hyperspectral cube data is split into three parts: visible light image, long-wave infrared image, and hyperspectral data. These parts are then input into three branch convolutional neural networks, which output high-level feature maps of the visible light image, long-wave infrared image, and hyperspectral data, respectively.

[0035] S32. Spatial upsampling is performed on the high-level feature map of the visible light image, the high-level feature map of the long-wave infrared image, and the high-level feature map of the hyperspectral data to obtain the corresponding three regional dimensional features. The regional dimensional features are then divided into multiple different small regional dimensional features.

[0036] S33. Constructing the graph structure in a graph convolutional network ,in, For the number of nodes, Given the number of edges, the dimensional features of small regions are used as nodes in the graph structure, and feature propagation is performed to obtain the transformed feature map. ;

[0037] S34, Transformation Feature Map Global average pooling and regularization are performed to obtain the feature vector of the final classification layer input. The feature vector input to the final classification layer Inputting the data into a softmax layer yields the predicted output probability. It is represented as:

[0038]

[0039]

[0040] In the formula, This indicates regularization processing; Indicates global average pooling;

[0041] S35, Output probability based on classification prediction The three parallel output heads output the crop detection results.

[0042] Furthermore, in S34, the three parallel output heads output crop detection results, including:

[0043] The first output header outputs: crop category and period;

[0044] The second output head outputs: water nitrogen content;

[0045] The third output head outputs: the degree of pests and diseases.

[0046] Furthermore, S3 also includes: calculating the loss function of the neural dynamics model for crop detection based on the crop detection results, which is expressed as:

[0047]

[0048] wherein, represents the final loss function; represents the core main task loss; is a hyper-parameter; represents the inter-modal contrast loss.

[0049] Further, based on the uncertainty adaptive weighting method, the core main task loss is calculated as:

[0050]

[0051] wherein,

[0052]

[0053] wherein, is the original loss value of the i-th task; represents the crop category, the period, the water and nitrogen content, and the pest and disease degree respectively; is a learnable parameter; represents the original loss value of the first task, represents the original loss value of the second task, represents the original loss value of the third task, represents the original loss value of the fourth task; CE represents the cross-entropy loss; Huber represents the Huber Loss; and are the true label and the predicted value of the crop category, and are the true label and the predicted value of the crop period, and are the true label and the predicted value of the water and nitrogen content of the crop; and are the true label and the predicted value of the pest and disease degree of the crop. Further, the cosine similarity between the three modal features of the visible light image, the long-wave infrared image and the hyperspectral data in each sample is calculated, and then the inter-modal contrast loss

[0054] is calculated as:

[0055]

[0056] wherein, is an exponential function with the natural constant e as the base; and ​​for different modal features of the same sample; for negative samples; for hyperparameters; denotes the cosine similarity; denotes the number of samples in the training batch; denotes the summation index for traversing all feature vectors in the batch.

[0057] The unmanned aerial vehicle multi-scale crop detection method based on the neural dynamics model has the following beneficial effects:

[0058] 1. The multi-source multi-scale feature standard crop data of the application integrates visible light images, long-wave infrared images, and hyperspectral data three modal data, dynamically collects multi-modal data of crops in different growth stages, constructs a complete feature atlas from the seedling stage to the mature stage, covers different spatial resolutions and spectral bands, and forms a systematic crop feature benchmark library. This crop feature benchmark library not only provides high-quality labeled data for model training, but also solves the long-standing data island problem in the agricultural field through standardized data formats and acquisition protocols.

[0059] 2. The neural dynamics model constructed by the application integrates three modal data inputs, realizes information complementation and enhancement, and fundamentally improves the perception ability and robustness of the model. And a collaborative model of CNN (Convolutional Neural Network) + GCN (Graph Convolution) is adopted, in which the three modal feature vectors are used as the nodes of GCN, the cross-modal correlation information is contained between each feature, and GCN is used to help the network search the hidden correlation features between multiple tasks. The edge of GCN combines "physical proximity" and "state similarity", and for multi-modal input, the GCN constructed by this edge is more robust and interpretable, which can accurately capture the internal physical and physiological laws of crop growth, and help to improve the performance of downstream tasks.

[0060] 3. The loss function combination of the neural dynamics model of the application considers the whole process from feature to task of the model, ensures that the features of the three modalities are aligned and complementary, and through uncertainty weighting, it can automatically coordinate the learning progress of different nature tasks such as classification and regression, and construct a stable and efficient end-to-end multi-task learning system. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 is a flowchart of the unmanned aerial vehicle multi-scale crop detection method based on the neural dynamics model of the embodiment of the application.

[0062] Figure 2 is a flowchart of the multi-source multi-scale data registration and fusion process of the embodiment of the application.

[0063] Figure 3A network structure diagram of a neurodynamics model for crop detection of an embodiment of the present application.

[0064] Figure 4 A schematic diagram for dividing a feature map region of an embodiment of the present application. DETAILED DESCRIPTION

[0065] The specific embodiments of the present application are described below to facilitate the understanding of the present application for those skilled in the art, but it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application defined and determined by the appended claims, and all the inventions utilizing the concept of the present application are within the scope of protection.

[0066] The unmanned aerial vehicle multi-scale crop detection method based on the neurodynamics model of the present embodiment refers to Figure 1 which specifically includes the following contents:

[0067] S1, acquiring visible light images, long-wave infrared images and hyperspectral data of crops under unmanned aerial vehicles in multi-scale, and then constructing a multi-source multi-scale feature standard crop database;

[0068] Among them, the crop image includes high-altitude images taken by visible light and long-wave infrared cameras. The spectral data is obtained by a hyperspectral camera; the acquisition method is that the unmanned aerial vehicle gimbal simultaneously carries visible light, long-wave infrared and hyperspectral cameras to take pictures of crops in the same area. Visible light images, long-wave infrared images and hyperspectral data of crops of different categories, different periods, different water and nitrogen contents, and different disease and pest degrees are obtained by using unmanned aerial vehicles;

[0069] Among them, different categories include: corn, rice, and wheat;

[0070] Different periods include: emergence stage, heading stage, and mature stage;

[0071] Different water and nitrogen contents include: no water deficiency, water deficiency, no nitrogen deficiency, and nitrogen deficiency;

[0072] Different disease and pest degrees include: no disease, mild disease, and severe disease.

[0073] Specifically, the pre-collection work needs to determine the crop planting area and geographic coordinates, and calibrate the left upper and right lower corners in the experimental farmland area. And collect data all day long for crops in the morning and evening, and use ground measured data to calibrate the disease, pest, water content, and nitrogen content of crops in the experimental field.

[0074] The unmanned aerial vehicle load system integrated with hyperspectral, visible light and long-wave infrared multi-sensor is used for data acquisition. The hyperspectral camera is a full-frame, push-broom and real-time imaging device with a spectral band of 400-1000 nm. The visible light and long-wave infrared cameras are integrated in an optoelectronic pod. The visible light camera acquires clear ground feature texture and color information, and the long-wave infrared camera is used to extract the long-wave infrared reflection characteristics of the vegetation canopy and can be used for night observation. The sensors are mounted on the gimbal of a multi-rotor unmanned aerial vehicle. Before flight, the integral time and shooting interval of the spectrometer are uniformly set according to the environmental brightness. After the dark current correction and whiteboard correction are completed, the shooting state is entered.

[0075] The unmanned aerial vehicle controls the flight parameters and attitude through the ground station and radio. After entering the set height of 100 m, the unmanned aerial vehicle automatically flies according to the pre-planned route in the test area, and records the flight height, weather condition and time information at the time of shooting. The flight route is reasonably planned to ensure a certain proportion of heading overlap and lateral overlap, so as to facilitate subsequent image stitching. For rice, corn and wheat, the hyperspectral, visible light and long-wave infrared image data of three development stages of seedling stage, heading stage and mature stage are collected in one year, and the water and nitrogen content and pest information are recorded through field experiments.

[0076] S2, the visible light image, the long-wave infrared image and the hyperspectral data are registered and fused to obtain fused hyperspectral cube data, referring to Figure 2 , which specifically includes the following contents;

[0077] S21, the hyperspectral data are subjected to terrain correction, radiation correction and data stitching;

[0078] Specifically, the GCPs coordinates are imported from the PhotoScan software to correct the terrain of the hyperspectral data, so as to eliminate the influence of terrain undulation on the image position. The Cube-pilot software is used for radiation correction, and the PhotoScan software is used for data stitching;

[0079] In order to better align the visible light image and the long-wave infrared image with the hyperspectral data, the 470 nm blue, 550 nm green and 650 nm red three channels of the processed hyperspectral data are combined into an RGB composite image .

[0080] S22, the RGB composite image is taken as a registration reference image, the visible light image and the long-wave infrared image are taken as floating images, and the RGB composite image is registered with the visible light image and the long-wave infrared image respectively to obtain the registered visible light image and long-wave infrared image;

[0081] The algorithm for registration is prior art, and reference can be made to the following documents:

[0082] Gao C, Li W, Tao R, et al. MS-HLMO: Multiscale histogram of local main orientation for remote sensing image registration[J]. IEEE Transactionson Geoscience and Remote Sensing, 2022, 60: 1-14.

[0083] S23, image enhancement processing is performed on the registered visible light image and long-wave infrared image, and denoising and dimensionality reduction processing are performed on the hyperspectral data, to obtain the visible light image and long-wave infrared image after image enhancement, and the hyperspectral data after dimensionality reduction;

[0084] For the visible light image, it contains visible light band (400-760 nm) signals, which is consistent with the principle of human eye vision, has high spatial resolution and high visual recognition, and has advantages for crop growth macroscopic monitoring and field crop morphology differentiation; the long-wave infrared image contains long-wave infrared band (8-14 μm) thermal radiation signals, which are positively correlated with object temperature, can quantify the difference of object surface temperature, and are not affected by light, and have advantages for crop water stress monitoring, early warning of diseases and pests, and night crop evaluation. Due to the different advantages of the characteristics of the two modal images, the purpose of image enhancement is to further highlight the advantages so that it can meet the detection needs of higher precision. This part only needs to enhance the visible light and long-wave infrared images.

[0085] In a specific embodiment, for the image enhancement processing of the visible light image, the following is specifically performed:

[0086] The visible light image needs to have high color fidelity and complete light correction. For this purpose, adaptive local tone mapping is adopted. The RGB image is converted to the YUV color space. YUV is a nonlinear color space, and its luminance component Y and chrominance component UV are separated, which has great advantages for adjusting the brightness of the image. The component value of the Y channel is extracted, and the global maximum element value of the image in the Y channel and the corresponding position are obtained. The luminance adaptive algorithm is as follows:

[0087]

[0088] In the formula, is the input luminance value of the pixel point After luminance adjustment, the result is is the input luminance value of the pixel point is the input luminance value of the pixel point is the input maximum luminance value; is the logarithmic average of the input luminance;

[0089] is the logarithmic average of the input luminance is:

[0090]

[0091] wherein, represents the sum of all pixels of the image, is a minimum value set to avoid the function meaningless when the input pixel point is 0.

[0092] Since the picture only has Y channel value change, the output picture will be color saturated due to the increase of only brightness, and in order to make up for the lack of color, color correction is performed:

[0093]

[0094] wherein, represents the correction value of the red channel of all pixel sets in the visible light image; represents the correction value of the blue channel of all pixel sets in the visible light image; represents the correction value of the green channel of all pixel sets in the visible light image; , , represents the red, blue and green channel values in the visible light image; represents the result of all pixel sets after brightness adjustment.

[0095] In a specific embodiment, the image enhancement processing of the long-wave infrared image is as follows:

[0096] The long-wave infrared image needs to reflect the difference of object thermal radiation, for which an adaptive double-threshold wavelet enhancement algorithm is adopted to enhance the contrast and suppress the noise. Daubechies-6 (Db6) wavelet basis is adopted for 3-level decomposition, and the image is decomposed into low-frequency component (wherein and represent the height and width of the image, respectively) and high-frequency component (global thermal radiation profile) and high-frequency component (horizontal, vertical and diagonal thermal edges and noise). The noise standard deviation is estimated from the low-frequency component :

[0097]

[0098] wherein, represents the median value; represents the absolute value of .

[0099] Adaptive dual-threshold calculation:

[0100]

[0101] wherein, is the pixel ratio of the high frequency component of the nth level, represents the suppression of strong noise, represents the preservation of weak edges; Dual-threshold processing, to

[0102] represent any high frequency component:

[0103]

[0104] wherein, represents the result of the dual-threshold processing of the high frequency component of the nth level at the pixel point ; is the value of the high frequency component of the nth level at the pixel point ; is the sign function;

[0105] Adaptive gain adjustment is performed on the low frequency component to strengthen the global thermal contrast:

[0106]

[0107] The processed low frequency component and the high frequency component are substituted into the inverse discrete wavelet transform to obtain the enhanced LWIR image, which is specifically represented as:

[0108]

[0109] wherein, represents the enhanced long-wave infrared image of the image; represents the inverse discrete wavelet transform; represents the low frequency component after adaptive gain adjustment to strengthen the global thermal contrast processing; is the high frequency component; is the low frequency component; is the gain coefficient; ​​​​​​​​​​​​The mean value is The standard deviation is

[0110] For the processing of hyperspectral data, due to the large number of wavebands, high detector sensitivity, and easy noise pollution and dimension redundancy, the subsequent analysis accuracy is reduced and the calculation cost is increased, so it is necessary to denoise and reduce dimension. Savitzky-Golay (SG) is used for convolution smoothing and baseline correction.

[0111] S24, the visible light image and long wave infrared image after image enhancement, and the hyperspectral data after dimension reduction are spliced and fused to obtain fused hyperspectral cube data.

[0112] S3, a neural dynamics model for crop detection is constructed, and the fused hyperspectral cube data is input into the neural dynamics model for crop detection for training, and a crop detection result is output;

[0113] Reference Figure 3 , the neural dynamics model for crop detection, the input is the fused hyperspectral cube with high space and high resolution composed of visible light image, long wave infrared image and hyperspectral data; the intermediate network layer of the neural dynamics model for crop detection includes a three-branch convolutional neural network (CNN) model and a graph convolutional network (GCN) model composed of feature nodes generated by the CNN. The output is the crop category, including: corn, rice, wheat; the period includes: emergence stage, heading stage, mature stage; water and nitrogen content includes: no water shortage, water shortage, no nitrogen deficiency, nitrogen deficiency; pest and disease degree: no disease, mild disease, severe disease.

[0114] In a specific embodiment, the fused hyperspectral cube data is input into the neural dynamics model for crop detection for training, including the following steps:

[0115] S31, the fused hyperspectral cube data is split, and the dimension of the hyperspectral cube data is (wherein , and respectively represent the height, width and channel number of the cube), wherein , is the number of hyperspectral channels. Splitting is: visible light image, long wave infrared image, and hyperspectral data three parts (three modalities) input;

[0116] At the same time, three different CNNs are matched for three modalities to adapt to their respective features.

[0117] For the visible light image of 3 channels (R, G, B), the matched CNN model is Xception. The depth separable convolution of Xception can strengthen the feature extraction of the local discriminant area while reducing the amount of calculation. Before training, the ImageNet dataset is used for pre-training, and the pre-training weight is imported into Xception. Then, the global average pooling (GAP) is used.

[0118] For the single-channel long-wave infrared image, MobileNetV2 is used for training, and the ImageNet pre-training weight is also imported. The lightweight architecture of MobileNetV2 can avoid parameter redundancy of the infrared branch, and the “bottleneck convolution” can effectively compress the feature dimension of the single-channel infrared data, which is suitable for the low information complexity of the infrared data.

[0119] For hyperspectral data, HyperSIGMA is used for training. The double-branch structure and unique attention mechanism of the model can fully excavate the spatial and spectral features of the data. First, on the HyperGlobal-450K dataset, the spatial and spectral branch network (ViT) is pre-trained using the mask image modeling method Masked AutoEncoder (MAE). The spatial image block or spectral channel is converted into token through the embedding layer. After pre-training, the sparse sampling attention (SSA) is designed to replace the complete self-attention of a specific layer in the ViT network. Finally, the spatial-spectral features are fused through the spectral enhancement module (SEM).

[0120] Based on the above process, the three branches of CNN output the visible light image high-level feature map, the long-wave infrared image high-level feature map, and the hyperspectral data high-level feature map.

[0121] S32, respectively, the visible light image high-level feature map, the long-wave infrared image high-level feature map, and the hyperspectral data high-level feature map are spatially up-sampled to obtain corresponding three regional dimension features, and the regional dimension features are divided into multiple different small regional dimension features.

[0122] In a specific embodiment, the compressed high-level feature maps of the three branches are not suitable for describing local non-overlapping regions. Therefore, the feature maps output by the three branch CNNs are respectively spatially up-sampled to obtain corresponding regional dimension features , and they are divided into different small regions, and the small regional dimension features are denoted as , as shown in Figure 4 These regions represent complementary information in different spatial contexts. Each small regional dimension feature becomes a node in the graph structure.

[0123] S33, constructing a graph structure in a graph convolution network ,in, For the number of nodes, Given the number of edges, the dimensional features of small regions are used as nodes in the graph structure, and feature propagation is performed to obtain the transformed feature map. ;

[0124] In one specific embodiment, in a graph convolutional network, a graph is constructed. (in For the number of nodes, (The number of edges) is used for feature propagation, and GCN is used to propagate the graph. Establish spatial relationships between features. Nodes in the diagram. Characterized by a deep feature map, the output is the convolutional feature of each node; edges A joint weighting of "spatial distance + multimodal feature similarity" is adopted. Description, used to represent node-level interactions:

[0125]

[0126] In the formula, Indicates the first The node and the first Edge weights between nodes; Represents a node and Spatial distance similarity (the closer the distance, the greater the weight); Represents a node and Multimodal feature cosine similarity (e.g., the more similar the infrared temperature and hyperspectral nitrogen features are, the greater the weight). To balance the parameters, it is recommended to perform parameter tuning through the verification set. .

[0127] The layer-by-layer feature propagation rule is defined as follows:

[0128]

[0129] In the formula, Indicates after the first The node feature matrix output after the layer GCN operation; Indicates the number of floors. For the first The weight matrix of the layer, Represents a nonlinear activation function (such as ReLU). Symmetric normalized adjacency matrix. (in , for The diagonal node degree matrix, the binary adjacency matrix Indicates self-connected , is a unit matrix).

[0130] S34, the converted feature map is globally averaged and regularized to obtain a feature vector input to the final classification layer The feature vector input to the final classification layer is input to a softmax layer to calculate the output probability of the prediction The output probability of the prediction , where corresponds to the true label of the target domain , which is expressed as:

[0131]

[0132]

[0133] In the formula, denotes the regularization processing; denotes global average pooling;

[0134] S35, based on the output probability of the classification prediction , three parallel output heads output crop detection results.

[0135] The three parallel output heads output crop detection results, including:

[0136] The first output head outputs: crop category (corn, rice, wheat corresponding output dimension 3);

[0137] The second output head outputs: water and nitrogen content (no water deficiency, water deficiency, no nitrogen deficiency, nitrogen deficiency corresponding output dimension 4);

[0138] The third output head outputs: the degree of disease and pest (no disease, mild disease, severe disease corresponding output dimension 3).

[0139] To constrain model training, based on the crop detection result, the loss function of the neural dynamics model for crop detection is calculated.

[0140] The present application designs a set of hierarchical loss function combination for the multi-modal multi-task neural dynamics model, in order to comprehensively improve the model performance. The combination contains core main task loss and auxiliary constraint loss, aiming to optimize the multi-task prediction accuracy and multi-modal feature fusion quality at the same time, which specifically includes:

[0141] Core main task loss ​To automatically balance the differences in loss magnitude among four tasks—crop type, growth period, water and nitrogen content, and pest and disease status—and avoid tedious manual parameter tuning, this invention employs an uncertainty-based adaptive weighting method as the core method for calculating the loss of the main task. This loss function can dynamically adjust the importance of each task, and its definition is as follows:

[0142]

[0143] In the formula, It is the first The original loss value for each task, These represent four tasks: crop type, growing season, water and nitrogen content, and pest and disease status. These are learnable parameters, corresponding to the task. Uncertainty The larger the value, the more difficult the task. The model will automatically reduce the loss weight of this difficult task to prevent its huge gradient from interfering with the learning of other tasks. It is a regularization penalty term, as Enlargement and enlargement, to prevent It increases without limit. This forces the model to find a balance between reducing task weights and incurring regularization penalties.

[0144] in,

[0145]

[0146] In the formula, This represents the original loss value for the first task. This represents the original loss value for the second task. This represents the original loss value for the third task. This represents the original loss value for the fourth task; and For the true label and predicted value of crop species, and For the actual label and predicted value of the crop period, and For the true label and predicted values ​​of crop water and nitrogen content, and The values ​​represent the true labels and predicted values ​​of crop pests and diseases; CE represents cross-entropy loss; Huber represents Huber Loss.

[0147] Intermodal contrast loss This loss can bring different modal features of the same sample closer together, and widen the gaps between features of different samples. For each sample's three modal features... Calculate the cosine similarity. The calculation is as follows:

[0148]

[0149] wherein, is an exponential function with base of natural constant and are different modal features of the same sample; is a negative sample; is a hyper-parameter; denotes cosine similarity; is the number of samples in the training batch; is the summation index for traversing all feature vectors in the batch.

[0150] The loss function of the neural dynamics model for crop detection is calculated, which is denoted as:

[0151]

[0152] wherein, denotes the final loss function; denotes the core main task loss; is a hyper-parameter; denotes the contrast loss between modalities.

[0153] The final loss function : The overall optimization objective for model training is composed of the core main task loss and the modal contrast loss, which is defined as follows:

[0154]

[0155] wherein, is a hyper-parameter for balancing the importance between the main task loss and the modal consistency constraint.

[0156] Although the specific embodiments of the invention are described in detail with reference to the accompanying drawings, it should not be understood as limiting the scope of protection of the patent. Various modifications and variations made by those skilled in the art within the scope described in the claims are still within the scope of protection of the patent.​

Claims

1. A method for multi-scale crop detection of unmanned aerial vehicle based on a neural dynamics model, characterized in that, The method comprises the following steps: S1, obtaining visible light images, long-wave infrared images and hyperspectral data of crops under multi-scale of a UAV; S2, performing registration and fusion processing on the visible light images, long-wave infrared images and hyperspectral data to obtain fused hyperspectral cube data; S3, constructing a neural dynamics model for crop detection, and inputting the fused hyperspectral cube data into the neural dynamics model for crop detection for training to output a crop detection result; In S3, the neural dynamics model for crop detection comprises three branch convolutional neural networks and a graph convolutional network; The training of the fused hyperspectral cube data in the neural dynamics model for crop detection comprises the following steps: S31, splitting the fused hyperspectral cube data into visible light images, long-wave infrared images and hyperspectral data, and inputting them into the three branch convolutional neural networks respectively to output visible light image high-level feature maps, long-wave infrared image high-level feature maps and hyperspectral data high-level feature maps respectively; S32, performing spatial upsampling on the visible light image high-level feature maps, the long-wave infrared image high-level feature maps and the hyperspectral data high-level feature maps respectively to obtain corresponding three regional dimension features, and dividing the regional dimension features into a plurality of different small regional dimension features; S33, constructing a graph structure in a graph convolution network wherein, is the number of nodes, is the number of edges, the small area dimension feature is taken as a node in the graph structure, and feature propagation is performed to obtain a converted feature map ; S34, the converted feature map Global average pooling and regularization are performed to obtain the feature vector input to the final classification layer The feature vector input to the final classification layer is obtained The softmax layer is input to obtain the predicted output probability which is expressed as: In the formula, denotes a regularization process; denotes a global average pooling; S35, output probability based on classification prediction three parallel output heads output crop detection results; In S3, a loss function of the neural dynamics model for crop detection is calculated based on the crop detection result, and is expressed as: wherein, represents the final loss function; represents the core main task loss; is a hyper-parameter; represents the contrastive loss between modalities. 2.The UAV multiscale crop detection method based on a neural dynamics model of claim 1, wherein, S1 specifically comprises: Based on the UAV, visible light images, long-wave infrared images and hyperspectral data of crops of different categories, different periods, different water and nitrogen contents and different disease and pest degrees are obtained; Different categories include corn, rice and wheat; Different periods include the emergence stage, the heading stage and the mature stage; Different water and nitrogen contents include no water deficiency, water deficiency, no nitrogen deficiency and nitrogen deficiency; Different disease and pest degrees include no disease, mild disease and severe disease. 3.The UAV multiscale crop detection method based on a neural dynamics model of claim 1, wherein, S2 comprises the following steps: S21, performing terrain correction, radiation correction and data splicing processing on the hyperspectral data, and merging the 470nm blue, 550nm green and 650nm red channels of the processed hyperspectral data into an RGB composite image; S22, taking the RGB composite image as a registration reference image, and performing registration with the visible light images and the long-wave infrared images respectively to obtain registered visible light images and long-wave infrared images; S23, performing image enhancement processing on the registered visible light images and long-wave infrared images, and performing denoising and dimension reduction processing on the hyperspectral data to obtain image-enhanced visible light images and long-wave infrared images and dimension-reduced hyperspectral data; S24, splicing and fusing the image-enhanced visible light images and long-wave infrared images and the dimension-reduced hyperspectral data to obtain fused hyperspectral cube data.

4. The UAV multiscale crop detection method based on a neural dynamics model according to claim 3, wherein, In S23, adaptive local tone mapping is used for image enhancement processing on the registered visible light images, and the registered visible light images are converted to the YUV color space, wherein the brightness adjustment in the Y channel is expressed as: wherein, is a pixel point After brightness adjustment result; is a pixel point input brightness value of, is the input maximum brightness value; is the logarithmic average value of input brightness; Based on the brightness adjustment in the Y channel, color correction is performed on the registered visible light images, and is specifically expressed as: wherein represents the corrected value of the red channel for all pixels in the visible light image; represents the corrected value of the blue channel for all pixels in the visible light image; represents the corrected value of the green channel for all pixels in the visible light image; represents the red, blue, green channel values in the visible light image; represents the result of the luminance adjustment for all pixels.​​ 5.The UAV multiscale crop detection method based on a neural dynamics model according to claim 3, wherein, In the S23, an adaptive double-threshold wavelet enhancement algorithm is used to perform image enhancement processing on the registered long-wave infrared image, and is specifically represented as: Wherein: In the formula, represents the long-wave infrared image after image enhancement; represents inverse discrete wavelet transform; represents adaptive gain adjustment on the low-frequency component to strengthen the low-frequency component after global thermal contrast processing; 、 、 、 、 、 、 、 、 is a high-frequency component; is a low-frequency component; is a gain coefficient; is a mean value, is a standard deviation.

6. The method of claim 1, wherein the method is based on a neural dynamics model. In the S34, three parallel output heads output the crop detection results, including: The first output head outputs: crop category, period; The second output head outputs: water and nitrogen content; The third output head outputs: pest and disease degree.

7. The method of claim 1, wherein the method is based on a neural dynamics model. Adaptive weighting method based on uncertainty, computing core main task loss which is expressed as: Wherein: In the formula, is the original loss value of the first task; is the original loss value of the second task; respectively represent the crop category, the period, the water and nitrogen content, and the pest and disease degree of the four tasks; is the learnable parameter; represents the original loss value of the first task, represents the original loss value of the second task, represents the original loss value of the third task, represents the original loss value of the fourth task; CE represents the cross-entropy loss; Huber represents the Huber Loss; and are the true label and the predicted value of the crop category, and are the true label and the predicted value of the crop period, and are the true label and the predicted value of the water and nitrogen content of the crop; and are the true label and the predicted value of the pest and disease degree of the crop.

8. The method of claim 1, wherein the method is based on a neural dynamics model. The cosine similarity between the three modal features of the visible light image, the long-wave infrared image and the hyperspectral data in each sample is calculated, and then the contrast loss between the modes is calculated which is expressed as: wherein, is an exponential function with the natural constant as the base; and are different modal features of the same sample; is a negative sample; is a hyper-parameter; denotes the cosine similarity; is the number of samples in the training batch; is the summation index for traversing all feature vectors in the batch.

Images