Multi-modal data fusion agricultural crop intelligent monitoring system and method

Abstract

The invention discloses a multi-modal data fusion agricultural crop intelligent monitoring system and method, and aims to realize accurate monitoring of crop growth and soil health through comprehensive analysis of image and audio data. The system comprises an image processing module, an audio acquisition module, an audio processing module and an early warning module. The image processing module acquires crop images in real time and analyzes the growth stage of crops so as to determine the active depth range of the root system. The audio acquisition module acquires audio signals in soil, and the audio processing module performs preprocessing, screening and feature extraction on the signals through an audio separation technology so as to obtain target audio information related to root activity. And further calculating an evaluation parameter based on a spectrum parameter obtained by audio analysis. And the early warning module generates corresponding early warning information and coping strategies according to whether the state parameters are abnormal or not. According to the invention, the accuracy and real-time performance of crop and soil health monitoring are improved, and more accurate decision support can be provided for agricultural management.

Description

Technical Field

[0001] The present invention relates to the technical field of crop monitoring, and in particular to a multi-modal data fusion agricultural crop intelligent monitoring system and method. Background Art

[0002] In the agricultural production process, the health of crops and the quality of soil have a decisive influence on the growth and yield of crops. Traditional soil and crop monitoring methods mostly rely on a single data source and technical means, which cannot fully and real-timely reflect the status of soil and crops. Traditional soil monitoring methods mainly rely on soil sensors, especially moisture sensors, temperature sensors and pH sensors. Although these methods can provide some information about the soil, they cannot fully monitor all biological and physical properties of the soil, such as soil microbial activity, root health status, etc. In addition, the working environment of soil sensors is easily disturbed by external factors (such as temperature, humidity, etc.), so the accuracy and real-time nature of their monitoring results are limited. Most of the existing monitoring systems rely on manual data acquisition or data collection through static sensors. These methods are usually difficult to monitor changes in crops and soil in real time. For example, traditional soil moisture sensors usually provide instantaneous data, while the health of crops and the quality of soil are a dynamically changing process. The real-time nature and accuracy of a single sensor are difficult to meet the needs of modern agriculture for data monitoring. Summary of the invention

[0003] The purpose of the present invention is to provide an agricultural crop intelligent monitoring system with multimodal data fusion, which has the advantage of high monitoring accuracy.

[0004] The above technical objectives of the present invention are achieved through the following technical solutions:

[0005] A multi-modal data fusion agricultural crop intelligent monitoring system, comprising:

[0006] An image processing module, the image processing module includes an image acquisition submodule and an image analysis submodule, the image acquisition submodule is used to monitor crops in real time to obtain crop images; the image analysis submodule is used to analyze crop images to determine crop growth stages, and determine the active root depth range according to the crop growth stages;

[0007] An audio acquisition module, wherein the audio acquisition module is disposed in the soil and is used to collect audio signals in the soil;

[0008] An audio processing module, the audio processing module includes an audio separation submodule and an audio analysis submodule, the audio separation submodule pre-processes the audio signal and then screens and separates the audio signal according to the root active depth range to obtain target audio information; the audio analysis submodule extracts features from the target audio information and analyzes it to obtain state parameters;

[0009] The early warning module generates corresponding early warning information and response strategies based on whether there is an abnormality in the state parameters.

[0010] Further configuration: the image analysis submodule is used to analyze the crop image to determine the crop growth stage, specifically including the following steps:

[0011] Preprocess the crop images;

[0012] The preprocessed crop image is segmented into growing areas and non-growing areas based on an image segmentation algorithm;

[0013] Extracting features from the growth area to obtain morphological feature data;

[0014] Determine the crop growth stage based on morphological characteristic data and preset historical data.

[0015] Further configuration: the audio separation submodule pre-processes the audio signal and then screens and separates the audio signal according to the root active depth range to obtain the target audio information, specifically including the following steps:

[0016] Pre-process the audio signal through filtering technology to remove environmental noise;

[0017] Performing time-frequency analysis on the preprocessed audio signal to generate a spectrogram;

[0018] Determine the corresponding frequency range according to the root active depth range, and filter the spectrum graph according to the root active depth range and frequency range to obtain the audio component;

[0019] The audio separation technology based on deep neural network performs signal separation processing on the spectrum graph according to the obtained audio components to obtain the target audio information.

[0020] Further configuration: the time-frequency analysis includes performing time-frequency domain conversion on the audio signal using short-time Fourier transform or wavelet transform.

[0021] Further configuration: the state parameters include spectrum parameters and evaluation parameters, and the audio analysis submodule extracts features from the target audio information and analyzes the state parameters to obtain the following specific parameters:

[0022] Extracting features of target audio information to obtain several audio features;

[0023] Spectral parameters are calculated based on several audio feature analyses, wherein the spectral parameters specifically include spectral power density, low frequency energy ratio, amplitude-humidity ratio, root activity index, frequency modulation index, spectral roll-off and high frequency energy ratio;

[0024] Evaluation parameters are calculated based on the spectrum parameters, wherein the evaluation parameters specifically include a crop health index and a soil health index.

[0025] Further setting: The specific calculation formula of the soil health index is:

[0026]

[0027] Among them, SHI is the soil health index, f i and w i is the spectrum parameter affecting the soil health index and its corresponding weight parameter, where f i Includes spectral power density, low frequency energy ratio, amplitude to humidity ratio, root activity index and spectral roll-off.

[0028] Further setting: The specific calculation formula of the crop health index is:

[0029]

[0030] Among them, CHI is the crop health index, f i and w i is the spectrum parameter affecting the crop health index and its corresponding weight parameter, where f i Including root activity index, frequency modulation index and high frequency energy ratio, α ij is the quadratic coefficient, which quantifies the interaction effect between different spectral parameters.

[0031] Further configuration: the warning module generates corresponding warning information and response strategies according to whether there is an abnormality in the state parameters, if there is an abnormality, specifically including:

[0032] If the soil health index SHI is less than the preset first soil threshold, and the spectrum roll-off SR is greater than the preset spectrum roll-off threshold, the soil is judged to be too wet and abnormal, and the corresponding warning information and response strategy are generated;

[0033] If the soil health index SHI is less than the preset second soil threshold, and the amplitude humidity ratio ARH is greater than the preset humidity ratio threshold, the soil is judged to be too dry and abnormal, and the corresponding warning information and response strategy are generated;

[0034] If the crop health index CHI is less than the preset crop threshold and the high-frequency energy ratio HFER is greater than the preset energy ratio threshold, it is judged that there are pests and diseases and abnormalities, and corresponding early warning information and response strategies are generated.

[0035] Another object of the present invention is to provide an intelligent monitoring method for agricultural crops using multimodal data fusion, which is applied to the intelligent monitoring system for agricultural crops using multimodal data fusion described above.

[0036] The above technical objectives of the present invention are achieved through the following technical solutions:

[0037] A method for intelligent monitoring of agricultural crops using multimodal data fusion, comprising the following steps:

[0038] (1) Acquire crop image data, monitor crops in real time through the image acquisition submodule, and acquire crop images;

[0039] (2) analyzing crop images to determine the growth stage of the crop and determining the active root depth range based on the crop growth stage;

[0040] (3) Acquire the audio signal in the soil and collect the audio data in the soil through the audio acquisition submodule;

[0041] (4) Preprocess the audio signal to remove environmental noise, use time-frequency analysis technology to convert the audio signal into time-frequency domain, and generate a spectrum diagram;

[0042] (5) According to the root active depth range and frequency range, the spectrum graph is screened to obtain the audio components;

[0043] (6) Based on the deep neural network audio separation technology, the spectrum is processed for signal separation according to the audio components to obtain the target audio information;

[0044] (7) Extracting features of the target audio information and calculating spectrum parameters, including spectrum power density, low-frequency energy ratio, amplitude-humidity ratio, root activity index, frequency modulation index, spectrum roll-off, and high-frequency energy ratio;

[0045] (8) further calculating evaluation parameters based on the calculated spectrum parameters;

[0046] (9) Determine whether there is an abnormality based on the spectrum parameters and evaluation parameters. If there is an abnormality, generate warning information and formulate a response strategy.

[0047] In summary, the present invention has the following beneficial effects:

[0048] 1. Multimodal data fusion improves monitoring accuracy. The present invention breaks through the limitations of the traditional single data source by combining multimodal data fusion of image and audio signals. Image data can reflect the growth status of the crop surface, while audio signals can deeply monitor the dynamic changes of root activity in the soil. The growth stage of the crop is judged by image data, and the actual active depth range of the crop root system is determined according to its growth stage. The audio signal is directionally screened according to the active depth range of the root system to obtain the audio signal within the active depth range for subsequent analysis and processing. The fusion of image and audio provides more accurate monitoring results.

[0049] 2. It can monitor the underground root system activity. Different from the existing technology that only relies on the ground part image monitoring, the present invention collects the audio signal in the soil in real time through the audio acquisition module to effectively monitor the activity of the underground root system. The health of the root system directly affects the water and nutrient absorption capacity of the crop. Therefore, it can deeply understand the impact of root system activity on crop growth, thereby improving the accuracy of crop management.

[0050] 3. Real-time early warning mechanism improves agricultural management efficiency. The present invention calculates the crop health index and soil health index by analyzing the spectral parameters in the audio signal, and compares them with the preset threshold value. It can identify abnormal changes in soil or crop status in real time and generate timely early warning information. This function can help agricultural managers to find problems in time and take countermeasures to prevent potential losses and improve crop production efficiency.

[0051] 4. Efficient audio signal processing technology. The present invention uses deep neural network audio separation technology to effectively filter environmental noise and extract audio components related to root activity. By deeply processing the audio signal through time-frequency analysis technology, it can more accurately capture the subtle changes in biological activity in the soil, improving the accuracy and sensitivity of soil and crop monitoring.

[0052] 5. Comprehensive health assessment: the system not only provides image analysis results of crop growth stages, but also provides comprehensive assessment based on soil and crop health indexes. Through the calculation of spectral parameters, the crop health index and soil health index can fully reflect the health status of crops and soil, which is helpful for optimizing agricultural production decisions and management strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is an overall flow chart of the embodiment. DETAILED DESCRIPTION

[0054] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0055] Example:

[0056] like Figure 1As shown, a multimodal data fusion agricultural crop intelligent monitoring system includes:

[0057] An image processing module, the image processing module includes an image acquisition submodule and an image analysis submodule, the image acquisition submodule is used to monitor crops in real time to obtain crop images; the image analysis submodule is used to analyze crop images to determine crop growth stages, and determine the active root depth range according to the crop growth stages;

[0058] The audio acquisition module is set in the soil and is used to collect audio signals in the soil. Various vibrations and sounds will be generated in the soil. For example, overly wet or overly dry soil will produce sound characteristics of different frequency bands, and the biological and microbial activities in the soil and the activities of the crop roots will produce various sounds. These audio information can be used to reflect the soil conditions and crop growth conditions, so as to judge the growth status of the crops and whether the soil is suitable for the current state of crop growth. Although the relevant data of the soil can also be monitored by humidity sensors, temperature sensors and other related sensors, the sensors reflect quantitative data, which is the general situation within a certain range. It is impossible to perform accurate detection based on the specific growth conditions of the crops. There may be a situation where the surrounding soil is too dry but the soil within the root activity range still has moisture, and it is difficult to control the movement of various organisms in the soil, which also has a great impact on the crops.

[0059] An audio processing module includes an audio separation submodule and an audio analysis submodule. The audio separation submodule pre-processes the audio signal and then filters and separates the audio signal according to the root active depth range to obtain target audio information; the audio analysis submodule extracts features from the target audio information and analyzes it to obtain state parameters.

[0060] The early warning module generates corresponding early warning information and response strategies based on whether there is an abnormality in the state parameters.

[0061] The image analysis submodule is used to analyze crop images and determine the crop growth stage, which specifically includes the following steps:

[0062] Preprocess crop images, including denoising, contrast enhancement, and image size standardization, to prepare for image segmentation;

[0063] The preprocessed crop image is segmented into a growing area and a non-growing area based on an image segmentation algorithm. In this embodiment, a U-Net model is used to segment the crop image.

[0064] The morphological feature data is obtained by extracting the features of the growth area.

[0065] Determine the crop growth stage based on morphological characteristic data and preset historical data.

[0066] The area of ​​the growing region is calculated based on the segmented growing region, and the morphological feature data extracted from the growing region specifically includes morphological features of the crop such as aspect ratio, roundness, leaf area index, etc.

[0067] Aspect ratio: Calculate the aspect ratio (i.e. the ratio of the width to the height of the rectangle) based on the circumscribed rectangle of the crop's growing area. In the early stages, the crop may appear long and narrow. In the mature stage, the crop's growing area tends to be more circular or uniform, with an aspect ratio close to 1.

[0068] Roundness is an indicator used to measure the similarity of an object's shape to a circle. A higher roundness value usually means that the crop shape is close to maturity. In the early stages of the crop, the roundness is lower and the shape is more irregular.

[0069] Leaf Area Index (LAI): The leaf area index (LAI) is calculated by extracting the crop leaf area from the segmented image. LAI represents the total area of ​​leaves per unit area. In the early stages of crop growth, LAI is low, and as the crop grows, LAI gradually increases. The LAI value can also be estimated based on vegetation indices (such as NDVI) in multispectral images.

[0070] Combined with the analysis results of the above morphological characteristics, the growth stage of the crop can be quantitatively judged by comparing the relationship between the crop growth area, morphological characteristics and historical data. For accurate calculation, a growth stage scoring model can be defined to quantify the characteristics of each growth stage through a series of rules. According to the segmentation results and morphological analysis provided by the image analysis module, the growth stage of the crop can be quantified into three specific indicators (initial stage, growth stage and maturity stage), and the model can be calibrated in combination with historical data to make the judgment more accurate.

[0071] Early stage: In this stage, the crop has a small growing area, a length-to-width ratio greater than 1, low roundness, and a low leaf area index. The plant's roots are mainly in the surface soil and grow slowly.

[0072] Growth period: The growing area of ​​the crop expands rapidly, the aspect ratio approaches 1, the roundness gradually increases, and the leaf area index rises. At this time, the roots of the plants gradually penetrate into the soil and begin to absorb a large amount of water and nutrients.

[0073] Maturity: The crop's growing area is saturated, the aspect ratio is close to 1, the roundness is close to the maximum, the leaf area index tends to be stable or decreases, the root depth reaches the maximum, and it is ready to be harvested.

[0074] It is also possible to establish more detailed growth stage judgment indicators based on relevant data according to different standards for different crops. To ensure more accurate judgment of the growth stage, the model can be trained in combination with historical data, and machine learning methods (such as support vector machines, random forests, etc.) can be used to optimize the weight coefficients and improve the accuracy of the model. For example, a classification model can be trained through historical crop growth data (such as the area of ​​the growing area, morphological characteristics, climatic conditions, etc.), and the weights of various parameters can be automatically adjusted to ultimately form an accurate growth stage prediction model.

[0075] After obtaining the specific growth stage of the crop, the approximate length and distribution range of the root system of the crop at the current growth stage can be determined based on the crop type, the local climate, and the type of soil in which it is planted. Based on these data, the active root depth range of the crop can be obtained, that is, the root system of the crop at this growth stage is mainly active or grows within this range. The root active depth range can provide more accurate screening conditions for the audio processing module, ensuring that the audio acquisition module only focuses on audio signals within the root active depth range, thereby improving the accuracy of audio signal analysis.

[0076] After obtaining the root active depth range, the audio signal is collected and processed. The audio separation submodule pre-processes the audio signal and then filters and separates the audio signal according to the root active depth range to obtain the target audio information, which specifically includes the following steps:

[0077] Pre-process the audio signal through filtering technology to remove environmental noise;

[0078] Performing time-frequency analysis on the preprocessed audio signal to generate a spectrogram. By analyzing the signal in different time and frequency ranges, a spectrogram containing multi-level depth information is obtained. The time-frequency analysis includes converting the audio signal into the time-frequency domain using short-time Fourier transform or wavelet transform.

[0079] Determine the corresponding frequency range according to the root active depth range, and filter the spectrum graph according to the root active depth range and frequency range to obtain the audio component;

[0080] The audio separation technology based on deep neural network performs signal separation processing on the spectrum graph according to the obtained audio components to obtain the target audio information.

[0081] Sounds at different soil depths have different frequency characteristics. Root activity, microbial activity, and soil physical properties (such as soil particle size and density) will affect the frequency range of the sound. Assuming that we know the frequency range of sound at different depths through experiments or soil acoustic models (for example, the sound within the root active depth range is between 100Hz and 3000Hz), based on the frequency range calculated based on the root active depth Dactive, the collected spectrum signal S(t, f) is filtered according to the frequency range, and the frequency components S that meet the root active depth range are retained. active (t, f). A Gaussian filter is used to filter out signals outside the frequency range. After filtering, the filtered audio components are separated from the original spectrum using audio separation technology based on a deep neural network.

[0082] The state parameters include spectrum parameters and evaluation parameters. The audio analysis submodule extracts and analyzes the target audio information to obtain state parameters, which specifically include:

[0083] Extracting features of target audio information to obtain several audio features;

[0084] The spectrum parameters are calculated based on several audio feature analyses, wherein the spectrum parameters specifically include spectrum power density, low frequency energy ratio, amplitude-humidity ratio, root activity index, frequency modulation index and high frequency energy ratio;

[0085] Evaluation parameters are calculated based on the spectrum parameters, wherein the evaluation parameters specifically include a crop health index and a soil health index.

[0086] The spectral power density (SPD) reflects the activity of the root system in the soil, such as water absorption. When there is a lack of water, the root system activity weakens and the spectral power density decreases accordingly. The calculation formula is:

[0087]

[0088] X (fi) Indicates frequency f i is the signal strength at , where N is the total number of frequency components.

[0089] Amplitude-Relative to Humidity (ARH): By analyzing the amplitude change of the signal and combining it with the humidity sensor data, an indication of water shortage can be obtained. The calculation formula is:

[0090]

[0091] Amplitude: The amplitude of the audio signal, which can be calculated using the maximum value or RMS value of the signal. Frequency Ratio: Moist soil usually has higher energy in the high frequency band, while dry soil tends to be in the low frequency band. This ratio reflects the difference in the spectrum of soils with different humidity. Soil Acoustic Impedance: The acoustic impedance of the soil, which affects the propagation of audio signals in the soil. This parameter can be estimated by the physical properties of the soil, such as porosity and density.

[0092] Spectral roll-off (SR) is an indicator of frequency distribution in the spectrum, which can reflect the energy changes of audio signals in different frequency ranges. When the soil is too wet, the energy of the low-frequency part will increase, causing the frequency distribution in the spectrum to change. Therefore, the SR value can be calculated to determine whether the soil is too wet. The calculation formula is:

[0093]

[0094] M is the number of frequency points in the low-frequency part (such as 20Hz to 200Hz), and N is the total number of frequency points of the signal.

[0095] The Root Activity Index (RAI) is used to measure the growth and activity intensity of roots in the soil. Root activity will produce audio signals in a specific frequency range, so we calculate the root activity index by analyzing the frequency characteristics in the audio signal. The calculation formula is:

[0096]

[0097] F1 and F2 are the lower and upper limits of the root activity frequency band (usually 20Hz to 1kHz), F low and F high It is the low and high frequency range of the entire spectrum.

[0098] High-Frequency Energy Ratio (HFER): Pests usually generate audio signals in the high frequency band. By calculating the energy ratio of the high frequency part (for example, above 2kHz), it can help identify signs of pests and diseases. The calculation formula is:

[0099]

[0100] High-Frequency Range refers to the frequency points within the high-frequency range.

[0101] Low-Frequency Energy Ratio (LFER): When the humidity is too high, the proportion of low-frequency energy increases. Over-humidity can be further confirmed by calculating the ratio of the energy ratio of the low-frequency part (for example, 0-200Hz) to the overall energy. The calculation formula is:

[0102]

[0103] Low-Frequency Range refers to the frequency points within the low-frequency range.

[0104] Frequency Modulation Index (FMI): Pests and diseases often manifest as specific audio patterns, especially in the mid- and high-frequency ranges of the spectrum. Frequency fluctuations caused by pest and disease activity can be measured by the frequency modulation index. The calculation formula is:

[0105]

[0106] △f is the change in frequency.

[0107] The specific calculation formula of the soil health index is:

[0108]

[0109] Among them, SHI is the soil health index, f i and w i is the spectrum parameter affecting the soil health index and its corresponding weight parameter, where f i Includes spectral power density, low frequency energy ratio, amplitude to humidity ratio, root activity index and spectral roll-off.

[0110] Further setting: The specific calculation formula of the crop health index is:

[0111]

[0112] Among them, CHI is the crop health index, f i and w i is the spectrum parameter affecting the crop health index and its corresponding weight parameter, where f i Including root activity index, frequency modulation index and high frequency energy ratio, α ij is the quadratic coefficient, which quantifies the interaction effect between different spectral parameters.

[0113] The warning module generates corresponding warning information and response strategies according to whether there is an abnormality in the state parameters. If there is an abnormality, the specific ones include:

[0114] If the soil health index SHI is less than the preset first soil threshold, and the spectrum roll-off SR is greater than the preset spectrum roll-off threshold, the soil is judged to be too wet and abnormal, and the corresponding early warning information and response strategy are generated; the drainage system can be started to reduce the accumulation of water in the soil and promote ventilation. The irrigation system is started to ensure that the soil has sufficient moisture to meet the needs of crop growth.

[0115] If the soil health index SHI is less than the preset second soil threshold, and the amplitude humidity ratio ARH is greater than the preset humidity ratio threshold, the soil is judged to be too dry and abnormal, and the corresponding warning information and response strategy are generated;

[0116] If the crop health index CHI is less than the preset crop threshold, and the high-frequency energy ratio HFER is greater than the preset energy ratio threshold, it is judged that there are pests and diseases, and there are abnormalities, and corresponding warning information and response measures are generated. Initiate pest and disease control measures.

[0117] The above warning information only represents the warning information implemented in this embodiment, i.e., the response strategy, including but not limited to these strategies. More warning and response measures can be judged through some data. For example, if the root activity index RAI shows abnormal root growth, it may be due to soil problems or pests and diseases. The system should provide root growth stimulation measures (such as fertilization, loosening the soil, etc.).

[0118] The analysis of audio information can detect some soil problems earlier, especially those related to the physical state of the soil or microbial activity. For example, audio information may be able to issue an early warning by detecting specific sounds emitted by the soil when it is dry or overly wet (such as the formation of soil cracks, increased pressure on plant roots, etc.), while temperature and humidity sensors may only respond when the moisture changes significantly. The source of various sounds in the soil is first the growth and activity of the roots in the soil. During the growth process, the roots will cause tiny vibrations in the surrounding soil, especially when absorbing water, breathing, expanding or undergoing tiny growth movements, which will generate sound waves. These vibration waves propagate in the soil and are manifested in the audio signal. Secondly, the change in moisture in the soil is an important factor affecting the audio signal. Humidity has a direct impact on the propagation speed and attenuation characteristics of sound waves. In soils with higher moisture, sound waves propagate more smoothly and attenuate less, while sound waves in dry soil propagate more slowly and attenuate faster. In addition, the activities of organisms and microorganisms are also one of the sources of audio signals, especially in the process of organic matter degradation and microbial metabolism.

[0119] This embodiment also provides a method for intelligent monitoring of agricultural crops using multimodal data fusion, comprising the following steps:

[0120] (1) Acquire crop image data, monitor crops in real time through the image acquisition submodule, and acquire crop images;

[0121] (2) analyzing crop images to determine the growth stage of the crop and determining the active root depth range based on the crop growth stage;

[0122] (3) Acquire the audio signal in the soil and collect the audio data in the soil through the audio acquisition submodule;

[0123] (4) Preprocess the audio signal to remove environmental noise, use time-frequency analysis technology to convert the audio signal into time-frequency domain, and generate a spectrum diagram;

[0124] (5) According to the root active depth range and frequency range, the spectrum graph is screened to obtain the audio components;

[0125] (6) Based on the deep neural network audio separation technology, the spectrum is processed for signal separation according to the audio components to obtain the target audio information;

[0126] (7) Extracting features of the target audio information and calculating spectrum parameters, including spectrum power density, low-frequency energy ratio, amplitude-humidity ratio, root activity index, frequency modulation index, spectrum roll-off, and high-frequency energy ratio;

[0127] (8) further calculating evaluation parameters based on the calculated spectrum parameters;

[0128] (9) Determine whether there is an abnormality based on the spectrum parameters and evaluation parameters. If there is an abnormality, generate warning information and formulate a response strategy.

[0129] In summary, the present invention has the following beneficial effects:

[0130] 1. Multimodal data fusion improves monitoring accuracy. The present invention breaks through the limitations of the traditional single data source by combining multimodal data fusion of image and audio signals. Image data can reflect the growth status of the crop surface, while audio signals can deeply monitor the dynamic changes of root activity in the soil. The growth stage of the crop is judged by image data, and the actual active depth range of the crop root system is determined according to its growth stage. The audio signal is directionally screened according to the active depth range of the root system to obtain the audio signal within the active depth range for subsequent analysis and processing. The fusion of image and audio provides more accurate monitoring results.

[0131] 2. It can monitor the underground root system activity. Different from the existing technology that only relies on the ground part image monitoring, the present invention collects the audio signal in the soil in real time through the audio acquisition module to effectively monitor the activity of the underground root system. The health of the root system directly affects the water and nutrient absorption capacity of the crop. Therefore, it can deeply understand the impact of root system activity on crop growth, thereby improving the accuracy of crop management.

[0132] 3. Real-time early warning mechanism improves agricultural management efficiency. The present invention calculates the crop health index and soil health index by analyzing the spectral parameters in the audio signal, and compares them with the preset threshold value. It can identify abnormal changes in soil or crop status in real time and generate timely early warning information. This function can help agricultural managers to find problems in time and take countermeasures to prevent potential losses and improve crop production efficiency.

[0133] 4. Efficient audio signal processing technology. The present invention uses deep neural network audio separation technology to effectively filter environmental noise and extract audio components related to root activity. By deeply processing the audio signal through time-frequency analysis technology, it can more accurately capture the subtle changes in biological activity in the soil, improving the accuracy and sensitivity of soil and crop monitoring.

[0134] 5. Comprehensive health assessment: the system not only provides image analysis results of crop growth stages, but also provides comprehensive assessment based on soil and crop health indexes. Through the calculation of spectral parameters, the crop health index and soil health index can fully reflect the health status of crops and soil, which is helpful for optimizing agricultural production decisions and management strategies.

[0135] The above-described implementation methods do not constitute a limitation on the protection scope of the technical solution. Any modification, equivalent replacement and improvement made within the spirit and principle of the above-described implementation methods shall be included in the protection scope of the technical solution.

Claims

1. A multimodal data fusion agricultural crop intelligent monitoring system, characterized in that: include: An image processing module, the image processing module includes an image acquisition submodule and an image analysis submodule, the image acquisition submodule is used to monitor crops in real time to obtain crop images; The image analysis submodule is used to analyze the crop image to determine the crop growth stage, and determine the active root depth range according to the crop growth stage; An audio acquisition module, wherein the audio acquisition module is disposed in the soil and is used to collect audio signals in the soil; An audio processing module, the audio processing module includes an audio separation submodule and an audio analysis submodule, the audio separation submodule pre-processes the audio signal and then screens and separates the audio signal according to the root active depth range to obtain target audio information; The audio analysis submodule extracts features from the target audio information and analyzes to obtain state parameters; The early warning module generates corresponding early warning information and response strategies based on whether there is an abnormality in the state parameters.

2. The agricultural crop intelligent monitoring system with multimodal data fusion according to claim 1 is characterized in that: The image analysis submodule is used to analyze the crop image to determine the crop growth stage, and specifically includes the following steps: Preprocess the crop images; The preprocessed crop image is segmented into growing areas and non-growing areas based on an image segmentation algorithm; Extracting features from the growth area to obtain morphological feature data; Determine the crop growth stage based on morphological characteristic data and preset historical data.

3. The agricultural crop intelligent monitoring system with multimodal data fusion according to claim 1 is characterized in that: The audio separation submodule pre-processes the audio signal and then screens and separates the audio signal according to the root active depth range to obtain the target audio information, which specifically includes the following steps: Pre-process the audio signal through filtering technology to remove environmental noise; Performing time-frequency analysis on the preprocessed audio signal to generate a spectrogram; Determine the corresponding frequency range according to the root active depth range, and filter the spectrum graph according to the root active depth range and frequency range to obtain the audio component; The audio separation technology based on deep neural network performs signal separation processing on the spectrum graph according to the obtained audio components to obtain the target audio information.

4. The agricultural crop intelligent monitoring system with multimodal data fusion according to claim 3 is characterized in that: The time-frequency analysis includes converting the audio signal into time-frequency domain by using short-time Fourier transform or wavelet transform.

5. The agricultural crop intelligent monitoring system with multimodal data fusion according to claim 1 is characterized in that: The state parameters include spectrum parameters and evaluation parameters. The audio analysis submodule extracts and analyzes the target audio information to obtain state parameters, which specifically include: Extracting features of target audio information to obtain several audio features; Spectral parameters are calculated based on several audio feature analyses, wherein the spectral parameters specifically include spectral power density, low frequency energy ratio, amplitude-humidity ratio, root activity index, frequency modulation index, spectral roll-off and high frequency energy ratio; Evaluation parameters are calculated based on the spectrum parameters, wherein the evaluation parameters specifically include a crop health index and a soil health index.

6. The multimodal data fusion agricultural crop intelligent monitoring system according to claim 5, characterized in that: The specific calculation formula of the soil health index is: Among them, SHI is the soil health index, f i and w i is the spectrum parameter affecting the soil health index and its corresponding weight parameter, where f i Includes spectral power density, low frequency energy ratio, amplitude to humidity ratio, root activity index and spectral roll-off.

7. The agricultural crop intelligent monitoring system with multimodal data fusion according to claim 5 is characterized in that: The specific calculation formula of the crop health index is: Among them, CHI is the crop health index, f i and w i is the spectrum parameter affecting the crop health index and its corresponding weight parameter, where f i Including root activity index, frequency modulation index and high frequency energy ratio, α ij is the quadratic coefficient, which quantifies the interaction effect between different spectral parameters.

8. The agricultural crop intelligent monitoring system with multimodal data fusion according to claim 1, characterized in that: The warning module generates corresponding warning information and response strategies according to whether there is an abnormality in the state parameters. If there is an abnormality, the specific ones include: If the soil health index SHI is less than the preset first soil threshold, and the spectrum roll-off SR is greater than the preset spectrum roll-off threshold, the soil is judged to be too wet and abnormal, and the corresponding warning information and response strategy are generated; If the soil health index SHI is less than the preset second soil threshold, and the amplitude humidity ratio ARH is greater than the preset humidity ratio threshold, the soil is judged to be too dry and abnormal, and the corresponding warning information and response strategy are generated; If the crop health index CHI is less than the preset crop threshold and the high-frequency energy ratio HFER is greater than the preset energy ratio threshold, it is judged that there are pests and diseases and abnormalities, and corresponding early warning information and response strategies are generated.

9. A method for intelligent monitoring of agricultural crops by multimodal data fusion, applied to a system for intelligent monitoring of agricultural crops by multimodal data fusion as claimed in claims 1-8, characterized in that: The following steps are involved: (1) Acquire crop image data, monitor crops in real time through the image acquisition submodule, and acquire crop images; (2) analyzing crop images to determine the growth stage of the crop and determining the active root depth range based on the crop growth stage; (3) Acquire the audio signal in the soil and collect the audio data in the soil through the audio acquisition submodule; (4) Preprocess the audio signal to remove environmental noise, use time-frequency analysis technology to convert the audio signal into time-frequency domain, and generate a spectrum diagram; (5) According to the root active depth range and frequency range, the spectrum graph is screened to obtain the audio components; (6) Based on the deep neural network audio separation technology, the spectrum is processed for signal separation according to the audio components to obtain the target audio information; (7) extracting features from the target audio information and calculating spectrum parameters, including spectrum power density, low-frequency energy ratio, amplitude-humidity ratio, root activity index, frequency modulation index spectrum roll-off, and high-frequency energy ratio; (8) further calculating evaluation parameters based on the calculated spectrum parameters; (9) Determine whether there is an abnormality based on the spectrum parameters and evaluation parameters. If there is an abnormality, generate warning information and formulate a response strategy.

Images