Agricultural scene identification and verification method and system based on multi-modal data

Abstract

The invention discloses an agricultural scene recognition and verification method and system based on multi-modal data, and belongs to the technical field of agricultural scene recognition. Multi-dimensional agricultural scene data such as images, weather and soil are collected through multiple sensors, and a standardized data set is formed through preprocessing such as data cleaning, calibration and space-time registration; using a soft voting integrated classifier to carry out agricultural scene identification on the image and time sequence features extracted from the standardized data set; based on the recognition result, the scene analysis model management library combines the scene matching degree and system resource dynamic selection, adjusts a scene analysis model and parameters thereof, and then completes multi-modal analysis to obtain a multi-modal data set containing rich information; and finally, carrying out multi-dimensional quality evaluation on the multi-modal data set. According to the method and the system, multi-modal data is effectively fused, the agricultural scene recognition accuracy is improved, computing resources are reasonably allocated, comprehensive and reliable data support is provided for agricultural production decisions, and the adaptability of the system to complex agricultural scenes is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of agricultural scene recognition, and in particular to an agricultural scene recognition and verification method and system based on multi-modal data. BACKGROUND

[0002] With the development of agricultural modernization, precision agriculture has become the key to improving agricultural production efficiency, ensuring food security and sustainable development. In precision agriculture, accurate recognition and understanding of agricultural scenes are the basis for fine management. Traditional agricultural production management mainly relies on manual experience, which is highly subjective and inefficient, and cannot meet the needs of large-scale, modern agricultural production.

[0003] In recent years, with the rapid development of sensor technology, computer technology and artificial intelligence technology, it has become possible to use multi-source data for agricultural scene analysis. However, agricultural scenes are highly complex and diverse, influenced by factors such as climate, soil, crop varieties, and planting methods. Single modal data often cannot fully reflect the true situation of agricultural scenes, resulting in low accuracy and reliability of scene recognition and analysis. At the same time, existing agricultural scene recognition and analysis methods also have shortcomings in data processing, model selection and resource utilization, making it difficult to adapt to complex and changing agricultural production environments. For example, some methods are inefficient in processing large-scale data, some models lack universality and adaptability for different agricultural scenes, and some systems cannot effectively integrate and utilize multi-modal data. Therefore, developing a method and system that can comprehensively utilize multi-modal data, accurately recognize and verify agricultural scenes, and efficiently utilize computing resources has important practical significance and application value.

[0004] To this end, an agricultural scene recognition and verification method and system based on multi-modal data are proposed. SUMMARY

[0005] The present application aims to solve the problems raised in the background art and provides an agricultural scene recognition and verification method and system based on multi-modal data.

[0006] The specific technical solutions are as follows:

[0007] An agricultural scene recognition and verification method based on multi-modal data, comprising the following steps:

[0008] Data preprocessing is performed on multi-dimensional agricultural scene data to output a standardized data set, the data preprocessing at least covers data cleaning, multi-source data calibration, time series synchronization and spatial registration, and the standardized data set is a multi-dimensional tensor containing agricultural data completely aligned in time, space and spectral dimensions;

[0009] Based on the standardized data set, an agricultural scene recognition is performed to obtain an agricultural scene recognition result. Specifically, image features and time sequence features are extracted from the standardized data set, and the image features and the time sequence features are connected to form a first comprehensive feature vector. Then, a soft voting integrated classifier is used to perform probability prediction on the first comprehensive feature vector to obtain the agricultural scene recognition result. The soft voting integrated classifier includes multiple basic classifiers. The first comprehensive feature vector is input into different basic classifiers to obtain agricultural scene prediction results, and the agricultural scene prediction results are weighted and averaged to obtain the agricultural scene recognition result. The calculation formula is as follows:

[0010]

[0011] wherein P(y) represents an agricultural scene category probability distribution; w i represents the weight of the i-th basic classifier; P i (y) represents the agricultural scene prediction result of the i-th basic classifier for the first comprehensive feature vector.

[0012] The scene analysis model management library manages and maintains different scene analysis models. Based on the agricultural scene recognition result, a matched set of scene analysis models and parameters thereof are dynamically selected, and the set of scene analysis models is adjusted in real time. The process of dynamically selecting the matched set of scene analysis models and parameters thereof is as follows: according to the agricultural scene category probability distribution, the scene matching degree score of each scene recognition model is calculated. The scene matching degree calculation formula is as follows:

[0013]

[0014] wherein P(yj) represents the probability of the j-th scene category; C m (yj) is the coverage of the j-th scene category by the scene recognition model m. Based on the calculation complexity of each scene recognition model and the current available calculation resources of the system, the feasibility score of each scene recognition model is calculated. The weighted sum of the scene matching degree score and the feasibility score of each scene recognition model is calculated, and the top a scene analysis models and parameters thereof with the highest scores are selected. The set of scene analysis models is adjusted in real time. When the evaluation index of the scene recognition model in the set changes while being continuously monitored, the model switching process is triggered, and the scene recognition model in the set is adjusted.

[0015] Based on the set of scene analysis models and their parameters, a multi-modal analysis is performed on the standardized data set to obtain a multi-modal data set of the agricultural scene data, specifically including feature recognition on the standardized data set, extracting a second comprehensive feature vector of the agricultural scene data, extracting agricultural features of the agricultural scene data based on the second comprehensive feature vector, and obtaining a multi-modal data set of the agricultural scene data based on the agricultural features. The data information in the multi-modal data set at least includes: identifier of the data in the standardized data set, list of applied enhancement operations and parameters thereof, feature importance map, list of agricultural features and confidence thereof, and feature consistency check result.

[0016] Quality evaluation is performed on the multi-modal data set, and the quality evaluation at least includes rule verification, multi-dimensional quality evaluation, credibility score, and anomaly detection.

[0017] The above-mentioned agricultural scene recognition and verification method based on multi-modal data, wherein before data preprocessing, it further includes: collecting agricultural scene data from multiple dimensions through multiple sensors, the multiple dimensions at least including image, meteorology, soil, and crop growth status.

[0018] The above-mentioned agricultural scene recognition and verification method based on multi-modal data, wherein the base classifier in the soft voting integrated classifier at least includes a random forest classifier, a support vector machine classifier, and a naive Bayes classifier, and the weight w of different base classifiers is determined according to the accuracy, recall rate, and F1 value of each base classifier on the training set. i According to the accuracy, recall rate, and F1 value of each base classifier on the training set.

[0019] The above-mentioned agricultural scene recognition and verification method based on multi-modal data, wherein when calculating the feasibility score of the scene recognition model, the computational complexity factors considered at least include the parameter quantity, the number of calculation layers, and the memory occupation of the model, and the current available computing resources of the system at least include CPU usage, GPU memory remaining amount, and memory remaining amount.

[0020] The above-mentioned agricultural scene recognition and verification method based on multi-modal data, wherein when performing feature recognition on the standardized data set, a deep learning model is used, the deep learning model at least includes a convolutional neural network and a recurrent neural network, and the deep learning model is pre-trained on a large-scale agricultural data set through transfer learning. The present application also proposes an agricultural scene recognition and verification system based on multi-modal data, the system comprising:

[0021] An agricultural scene data preprocessing module is configured to preprocess multi-dimensional agricultural scene data to output a standardized data set, wherein the data preprocessing at least includes data cleaning, multi-source data calibration, time series synchronization, and spatial registration, and the standardized data set is a multi-dimensional tensor containing agricultural data completely aligned in time, space, and spectral dimensions.

[0022] An agricultural scene recognition module is configured to recognize the standardized data set to obtain an agricultural scene recognition result, specifically, image features and time series features are extracted from the standardized data set, the image features and the time series features are connected to form a first comprehensive feature vector, and a soft voting ensemble classifier is used to probabilistically predict the first comprehensive feature vector to obtain the agricultural scene recognition result, wherein the soft voting ensemble classifier includes multiple base classifiers, the first comprehensive feature vector is input into different base classifiers to obtain agricultural scene prediction results, and the agricultural scene prediction results are weighted and averaged to obtain the agricultural scene recognition result, and the calculation formula is as follows:

[0023]

[0024] wherein P(y) represents an agricultural scene category probability distribution; w i represents the weight of the i-th base classifier; P i (y) represents the agricultural scene prediction result of the i-th base classifier for the first comprehensive feature vector.

[0025] A scene analysis model management library is configured to manage and maintain different scene analysis models, dynamically select a matched set of scene analysis models and parameters based on the agricultural scene recognition result, and adjust the set of scene analysis models in real time, and the process of dynamically selecting a matched set of scene analysis models and parameters is as follows: according to the agricultural scene category probability distribution, the scene matching degree score of each scene recognition model is calculated, and the calculation formula of the scene matching degree is as follows:

[0026]

[0027] wherein P(yj) represents the probability of the j-th scene category, and C m(yj) is the coverage of the jth scene recognition model m on the scene class; based on the computational complexity of each scene recognition model and the current available computing resources of the system, the feasibility score of each scene recognition model is calculated; the weighted sum of the scene matching degree score and the feasibility score of each scene recognition model is calculated, and the set of the top a scene analysis models with the highest score and their parameters are selected, and the set of scene analysis models is adjusted in real time. When the evaluation index changes when the scene recognition models in the set are continuously monitored, the model switching process will be triggered, and the scene recognition models in the set are adjusted;

[0028] The scene analysis model set module is configured to perform multi-modal analysis on the standardized data set based on the set of scene analysis models and their parameters, to obtain a multi-modal data set of the agricultural scene data. Specifically, the module is configured to perform feature recognition on the standardized data set, extract a second comprehensive feature vector of the agricultural scene data, extract agricultural features of the agricultural scene data based on the second comprehensive feature vector, and obtain a multi-modal data set of the agricultural scene data based on the agricultural features. The data information in the multi-modal data set at least includes identifiers of data in the standardized data set, a list of applied enhancement operations and their parameters, a feature importance map, a list of agricultural features and their confidence, and a feature consistency check result.

[0029] The quality evaluation module is configured to perform quality evaluation on the multi-modal data set. The quality evaluation at least includes rule verification, multi-dimensional quality evaluation, credibility scoring, and anomaly detection.

[0030] The above-mentioned agricultural scene recognition and verification system based on multi-modal data further comprises a data acquisition module. The data acquisition module acquires agricultural scene data from multiple dimensions through multiple sensors. The multiple dimensions at least include images, weather, soil, and crop growth conditions.

[0031] The above-mentioned agricultural scene recognition and verification system based on multi-modal data, wherein the weights w of the different base classifiers in the soft voting integrated classifier in the agricultural scene recognition module at least include random forest classifier, support vector machine classifier, and naive Bayes classifier. i The weights w of the different base classifiers are determined based on the accuracy, recall rate, and F1 value of each base classifier on the training set.

[0032] The above-mentioned agricultural scene recognition and verification system based on multi-modal data, wherein when calculating the feasibility score of the scene recognition model, the scene analysis model management library considers the computational complexity factors at least including the number of parameters, the number of calculation layers, and the memory occupation of the model. The current available computing resources of the system at least include CPU usage, GPU memory remaining amount, and memory remaining amount.

[0033] The agricultural scene recognition and verification system based on multi-modal data, wherein the scene analysis model set module adopts a deep learning model when performing feature recognition on the standardized data set, the deep learning model at least includes a convolutional neural network and a recurrent neural network, and is pre-trained on a large-scale agricultural data set through transfer learning.

[0034] The present application has the following beneficial effects:

[0035] 1. Improve the accuracy of agricultural scene recognition: through multi-modal data fusion, advanced feature extraction methods and integrated classifiers, etc. Technical means, fully excavate various information in agricultural scene, comprehensive analysis and judgment, effectively improve the accuracy of agricultural scene recognition, reduce misjudgment and omission.

[0036] 2. Optimize resource utilization: the scene analysis model management library can dynamically select and real-time adjust the scene analysis model and its parameters according to the scene recognition result, model calculation complexity and system available computing resources, realize the reasonable allocation and efficient utilization of computing resources, avoid resource waste, improve the system running efficiency.

[0037] 3. Provide comprehensive data support: collect agricultural scene data from multiple dimensions, and generate multi-modal data set containing rich information through multi-modal analysis, provide comprehensive and in-depth data support for agricultural production decision-making, which helps farmers and agricultural managers to make more scientific and reasonable production plan and decision.

[0038] 4. Ensure data quality: strict data preprocessing is performed on the collected data, and multi-dimensional quality evaluation is performed on the generated multi-modal data set, which ensures the accuracy, integrity and reliability of the data, and lays a solid foundation for the stable operation and effective application of the whole system.

[0039] 5. Enhance system adaptability: the system can adapt to different agricultural scenes and complex and changeable environmental conditions, continuously optimize the recognition and analysis process by dynamically adjusting the model and parameters, improve the universality and adaptability of the system, and can be widely applied to different regions and different types of agricultural production scenes. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 The flowchart of the scientific research knowledge retrieval method based on RAG and hallucination correction provided by the embodiment of the present application. DETAILED DESCRIPTION

[0041] The technical solutions of the present application will be further illustrated by specific embodiments in combination with the accompanying drawings.

[0042] In the drawings, only for example, the representation is a schematic diagram, not a physical diagram, and cannot be understood as a limitation of the patent; in order to better illustrate the embodiments of the present application, some components of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0043] The same or similar reference numerals in the drawings of the embodiments of the present application correspond to the same or similar components; in the description of the present application, it should be understood that if the terms "upper", "lower", "left", "right", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the positional relationship described in the drawings is only for example, and cannot be understood as a limitation of the patent, for those skilled in the art, the specific meaning of the above terms can be understood according to the specific situation.

[0044] In the description of the present application, unless otherwise explicitly specified and limited, if the term "connection" and the like appear to indicate the connection relationship between components, the term should be broadly understood, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrated; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication or interaction relationship between two components. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific situation.

[0045] Embodiment 1

[0046] The present embodiment provides an agricultural scene recognition and verification method based on multi-modal data, comprising the following steps:

[0047] S1: data preprocessing is performed on multi-dimensional agricultural scene data, and a standardized data set is output, the data preprocessing at least covers data cleaning, multi-source data calibration, time series synchronization and spatial registration, the standardized data set is a multi-dimensional tensor, containing agricultural data completely aligned in time, space and spectral dimensions;

[0048] S2: Based on the standardized data set, the agricultural scene recognition result is obtained, and the specific operation is to extract the image features and time sequence features from the standardized data set, connect the image features and time sequence features to form a first comprehensive feature vector, and then perform probability prediction on the first comprehensive feature vector through a soft voting integrated classifier to obtain the agricultural scene recognition result. The soft voting integrated classifier includes a plurality of basic classifiers, the first comprehensive feature vector is input into different basic classifiers, and an agricultural scene prediction result is obtained respectively. The agricultural scene prediction results are weighted and averaged to obtain the agricultural scene recognition result, and the calculation formula is:

[0049]

[0050] Wherein: P(y) represents the probability distribution of the agricultural scene category; w i represents the weight of the i-th basic classifier; P i (y) represents the agricultural scene prediction result of the i-th basic classifier for the first comprehensive feature vector.

[0051] S3: The scene analysis model management library manages and maintains different scene analysis models, dynamically selects a matched set of scene analysis models and their parameters based on the agricultural scene recognition result, and adjusts the set of scene analysis models in real time. The process of dynamically selecting a matched set of scene analysis models and their parameters is as follows: according to the agricultural scene category probability distribution, the scene matching degree score of each scene recognition model is calculated, and the scene matching degree calculation formula is as follows:

[0052]

[0053] Wherein: P(yj) represents the probability of the j-th scene category, C m (yj) is the coverage of the j-th scene category by the scene recognition model m; based on the calculation complexity of each scene recognition model and the current available computing resources of the system, the feasibility score of each scene recognition model is calculated; the weighted sum of the scene matching degree score and the feasibility score of each scene recognition model is calculated, the top a scene analysis model set and its parameters with the highest score are selected, and the set of scene analysis models is adjusted in real time. When the evaluation index changes when the scene recognition models in the set are continuously monitored, the model switching process will be triggered, and the scene recognition models in the set are adjusted.

[0054] S4: based on the set of scene analysis models and their parameters, multi-modal analysis is performed on the standardized data set to obtain a multi-modal data set of agricultural scene data, specifically including feature recognition on the standardized data set, extraction of a second comprehensive feature vector of the agricultural scene data, extraction of agricultural features of the agricultural scene data based on the second comprehensive feature vector, and obtaining a multi-modal data set of the agricultural scene data based on the agricultural features, the data information in the multi-modal data set at least including: identifiers of data in the standardized data set, a list of applied enhancement operations and their parameters, a feature importance map, a list of agricultural features and their confidence, and feature consistency check results;

[0055] S5: quality evaluation is performed on the multi-modal data set, which at least includes rule verification, multi-dimensional quality evaluation, credibility scoring, and anomaly detection.

[0056] By using the above technical solutions, comprehensive data preprocessing is adopted to ensure accurate alignment of agricultural data of different sources and different types in time, space, and spectral dimensions, to provide a high-quality, standardized data set for subsequent analysis, and to greatly improve the usability of data and the accuracy of analysis. The soft voting integrated classifier composed of multiple basic classifiers combines the advantages of different classifiers and improves the reliability and accuracy of the agricultural scene recognition result through weighted average probability prediction. The scene analysis model management library dynamically selects models and parameters according to the recognition result and adjusts them in real time, realizes accurate matching of models and actual agricultural scenes, improves analysis efficiency and effect, reasonably utilizes computing resources, and avoids resource waste. Multi-modal analysis can deeply mine the features of agricultural scene data to obtain a multi-modal data set containing rich information, providing more comprehensive and in-depth data support for agricultural production decision-making. Multi-dimensional quality evaluation is performed on the multi-modal data set to ensure the reliability, credibility, and integrity of the data, and to improve the quality of the output results of the entire system.

[0057] Specifically, in this embodiment, before data preprocessing, it also includes: collecting agricultural scene data from multiple dimensions through multiple sensors, the multiple dimensions at least including images, weather, soil, and crop growth conditions. By using the above technical solutions, agricultural scene data is collected from multiple dimensions through multiple sensors, covering images, weather, soil, crop growth conditions, etc., greatly enriching the data sources, enabling the system to obtain comprehensive agricultural scene information, providing sufficient data basis for subsequent accurate analysis and decision-making, and improving the understanding and grasping ability of complex agricultural scenes.

[0058] Specifically, in this embodiment, the basic classifiers in the soft voting integrated classifier at least include a random forest classifier, a support vector machine classifier, and a naive Bayes classifier, and the weights w of different basic classifiers satisfy w iThe accuracy, recall rate and F1 value of each base classifier on the training set are comprehensively determined. By using the technical scheme, the soft voting integrated classifier contains multiple base classifiers, such as a random forest classifier, a support vector machine classifier, a naive Bayes classifier and the like, the weights are comprehensively determined in combination with the accuracy, recall rate and F1 value of different base classifiers on the training set, the advantages of each classifier can be fully played, the performance of the integrated classifier is optimized, and the precision and stability of the agricultural scene recognition are further improved.

[0059] Specifically, in the embodiment, when calculating the feasibility score of the scene recognition model, the considered calculation complexity factors include at least the parameter quantity, the calculation layer number and the memory occupation of the model, and the currently available calculation resources of the system include at least the CPU usage rate, the GPU video memory remaining amount and the memory remaining amount. By using the above technical scheme, when calculating the feasibility score of the scene recognition model, the calculation complexity factors such as the model parameter quantity, the calculation layer number and the memory occupation, and the currently available calculation resources of the system such as the CPU usage rate, the GPU video memory remaining amount and the memory remaining amount are comprehensively considered, the model selection can be dynamically adjusted according to the actual resource situation of the system under the premise of ensuring the model performance, the efficient use of calculation resources is realized, and the system can be stably and efficiently operated under different hardware environments.

[0060] Specifically, in the embodiment, when performing feature recognition on the standardized data set, a deep learning model is used, which at least includes a convolutional neural network and a recurrent neural network, and is pre-trained on a large-scale agricultural data set through transfer learning. By using the above technical scheme, the deep learning model (such as a convolutional neural network and a recurrent neural network) is used and pre-trained on a large-scale agricultural data set through transfer learning, which can quickly and accurately perform feature recognition on the standardized data set, effectively extract key features of agricultural scene data, improve the efficiency and accuracy of feature extraction, and further improve the performance and quality of the entire multi-modal analysis process.

[0061] The specific data processing scheme is as follows:

[0062] 1. Data acquisition and preprocessing technical scheme

[0063] 1.1 Multi-source data acquisition specification

[0064] Sensor type:

[0065] Image sensor: DJI Mavic3 Multispectral unmanned aerial vehicle (resolution 5 cm / pixel, containing 5 wavebands of red, green, blue, red edge and near infrared).

[0066] Weather station: AWS-400 automatic weather station (monitoring temperature, humidity, light, wind speed and rainfall, sampling frequency 1 time / 10 minutes).

[0067] Soil sensors: EC-5 soil moisture sensors (buried at 10 cm, 30 cm, 50 cm, sampling frequency 1 time / hour).

[0068] Temporal and spatial synchronization mechanism:

[0069] Time synchronization: All sensors are calibrated by NTP server, error ≤5 ms; POS (position, attitude) data is recorded during UAV flight, with centimeter-level accuracy.

[0070] Spatial positioning: RTK-GNSS technology is used, with GCP (ground control point) layout density ≥1 per hectare, ensuring image geographic registration error ≤1 pixel.

[0071] 1.2 Data cleaning and calibration core algorithm

[0072] 1.2.1 Data cleaning

[0073] Image denoising: Bilateral filtering (spatial weight σ_s=5, pixel value weight σ_r=0.2) is used to remove remote sensing image noise and preserve edge details.

[0074] Time series denoising: Wavelet transform (db4 wavelet, 3-level decomposition) is used to remove high-frequency noise from meteorological data, and Kalman filter is used to predict and correct jump values for soil data.

[0075] 1.2.2 Multi-source calibration

[0076] Spectral calibration: UAV multispectral data is radiometrically calibrated using a laboratory calibration panel (SpectraVue2000) and converted to reflectance data.

[0077] Format unification: All data are converted to HDF5 format, and metadata include sensor ID, sampling time, coordinate system (WGS84 / UTM).

[0078] 1.3 Temporal and spatial registration technology

[0079] Time series synchronization:

[0080] For different frequency data (such as meteorological data 10 minutes / time, crop growth 1 hour / time), sliding window interpolation (window size=15 minutes) is used, and cubic spline interpolation is used to generate equally spaced time series.

[0081] Spatial gridding:

[0082] Point data (soil sensors, ground sampling points) are interpolated to generate 10m×10m grids using inverse distance weighting (IDW), aligned with remote sensing image pixels; Gaussian kernel function (bandwidth=20m) is used to optimize spatial smoothness.

[0083] How the base classifier works and the classification method

[0084] 2. Soft voting ensemble classifier technical details

[0085] 2.1 Random Forest Classifier (RF)

[0086] Core mechanism:

[0087] Build 100 decision trees, each tree randomly selects 1 / 3 features (such as NDVI, soil moisture, temperature, etc.) and 50% samples for training, and reduces variance through Bagging.

[0088] Node splitting criterion: Gini impurity is used (suitable for multi-classification scenarios), and the splitting is stopped when the number of node samples <10 or Gini gain <0.01.

[0089] Agricultural scene application:

[0090] Good at handling high-dimensional nonlinear features, such as identifying crop growth stages (sowing, seedling, flowering, etc.) through canopy structure features (leaf area index, plant height).

[0091] 2.2 Support Vector Machine Classifier (SVM)

[0092] Core mechanism:

[0093] Radial basis function (RBF) kernel is used, with parameters C=10, γ=0.1, to map features to high-dimensional space to find the optimal segmentation hyperplane.

[0094] The one-vs-one strategy is used to handle multi-classification problems (such as 10 binary classifiers for 5-class scenarios).

[0095] Agricultural scene application:

[0096] Sensitive to spectral features, can distinguish different crop types (wheat, corn, soybeans) through vegetation indices (such as EVI, SAVI).

[0097] Example 2

[0098] The embodiment provides an agricultural scene recognition and verification system based on multi-modal data, characterized in that the system comprises an agricultural scene data preprocessing module, an agricultural scene recognition module, a scene analysis model management library, a scene analysis model set module and a quality evaluation module, wherein:

[0099] The agricultural scene data preprocessing module is configured to perform data preprocessing on multi-dimensional agricultural scene data, output a standardized data set, and the data preprocessing at least covers data cleaning, multi-source data calibration, time series synchronization, and spatial registration, the standardized data set is a multi-dimensional tensor, and contains agricultural data completely aligned in time, space, and spectral dimensions.

[0100] The agricultural scene recognition module is configured to perform agricultural scene recognition on the standardized data set, obtain an agricultural scene recognition result, and specifically perform image feature and time series feature extraction on the standardized data set, connect the image feature and the time series feature to form a first comprehensive feature vector, and perform probability prediction on the first comprehensive feature vector through a soft voting integrated classifier to obtain the agricultural scene recognition result, wherein the soft voting integrated classifier includes multiple basic classifiers, the first comprehensive feature vector is input into different basic classifiers, and an agricultural scene prediction result is obtained, the agricultural scene prediction results are weighted and averaged to obtain the agricultural scene recognition result, and the calculation formula is as follows:

[0101]

[0102] wherein P(y) represents an agricultural scene category probability distribution; w i represents the weight of the i-th basic classifier; P i (y) represents the agricultural scene prediction result of the i-th basic classifier on the first comprehensive feature vector.

[0103] The scene analysis model management library is configured to manage and maintain different scene analysis models, dynamically select a matched set of scene analysis models and parameters based on the agricultural scene recognition result, and adjust the set of scene analysis models in real time, and the process of dynamically selecting the matched set of scene analysis models and parameters is as follows: according to the agricultural scene category probability distribution, the scene matching degree score of each scene recognition model is calculated, and the calculation formula of the scene matching degree is as follows:

[0104]

[0105] wherein P(yj) represents the probability of the j-th scene category, C m (yj) is the coverage of the j-th scene category by the scene recognition model m; based on the calculation complexity of each scene recognition model and the current available computing resources of the system, the feasibility score of each scene recognition model is calculated; the weighted sum of the scene matching degree score and the feasibility score of each scene recognition model is calculated, the set of a first a scene analysis models with the highest scores and parameters are selected, and the set of scene analysis models is adjusted in real time, that is, when the scene recognition models in the set are continuously monitored and the evaluation indicators change, the model switching process is triggered, and the scene recognition models in the set are adjusted.

[0106] The scene analysis model set module is configured to perform multi-modal analysis on the standardized data set based on a set of scene analysis models and parameters thereof, to obtain a multi-modal data set of the agricultural scene data, and specifically includes feature recognition on the standardized data set, extraction of a second comprehensive feature vector of the agricultural scene data, extraction of agricultural features of the agricultural scene data based on the second comprehensive feature vector, and obtaining of the multi-modal data set of the agricultural scene data based on the agricultural features, wherein the multi-modal data set includes at least identifiers of data in the standardized data set, a list of applied enhancement operations and parameters thereof, a feature importance map, a list of agricultural features and confidence thereof, and a feature consistency check result.

[0107] The quality evaluation module is configured to perform quality evaluation on the multi-modal data set, and the quality evaluation includes at least rule verification, multi-dimensional quality evaluation, credibility scoring, and anomaly detection.

[0108] The above technical solutions make the system architecture clear, and the modules have clear division of labor and work cooperatively. The data preprocessing module ensures data quality, the recognition module realizes accurate scene recognition, the scene analysis model management library optimizes model selection, the scene analysis model set module performs deep multi-modal analysis, and the quality evaluation module guarantees data quality. The modules cooperate with each other to realize efficient and accurate execution of agricultural scene recognition and verification based on multi-modal data, and provide reliable support for agricultural production.

[0109] The agricultural scene recognition and verification system based on multi-modal data further includes a data acquisition module. The data acquisition module acquires agricultural scene data from multiple dimensions through multiple sensors, and the multiple dimensions include at least images, weather, soil, and crop growth conditions. The data acquisition module acquires data from multiple dimensions through multiple sensors, cooperates closely with other modules of the system, and provides a variety of raw data for the entire system. The system can analyze based on comprehensive information, and the adaptability and analysis capability of the system to complex and changeable agricultural scenes are enhanced, which helps to improve the accuracy and reliability of agricultural scene recognition and verification.

[0110] In the soft voting ensemble classifier in the agricultural scene recognition module, the base classifiers include at least a random forest classifier, a support vector machine classifier, and a naive Bayes classifier, and the weights w of different base classifiers satisfy w i The accuracy, recall rate, and F1 value of each base classifier on the training set are comprehensively determined. The base classifiers and the weight determination method of the soft voting ensemble classifier in the agricultural scene recognition module enable the module to integrate the advantages of different classifiers and dynamically adjust the weights according to the performance of different classifiers on the training set when identifying agricultural scenes, thereby effectively improving the recognition accuracy and stability and providing more reliable recognition results for subsequent modules.

[0111] In the process of calculating the feasibility score of the scene recognition model, the scene analysis model management library considers at least the following factors: the parameter quantity of the model, the number of calculation layers, and the memory occupation; and the system's current available computing resources include at least CPU usage, GPU memory remaining, and memory remaining. The scene analysis model management library considers multiple factors when calculating the feasibility score of the scene recognition model, which enables it to reasonably select and adjust the model according to the actual system conditions, ensuring that the model can run efficiently under different computing resource conditions, improving the model's applicability and the system's overall operation efficiency, and avoiding performance degradation caused by insufficient resources or improper model selection.

[0112] In the process of feature recognition on the standardized data set, the scene analysis model set module adopts a deep learning model, which at least includes a convolutional neural network and a recurrent neural network, and is pre-trained on a large-scale agricultural data set through transfer learning. The scene analysis model set module set adopts a specific deep learning model and transfer learning method, which can efficiently and accurately recognize the features of the standardized data set and extract representative agricultural scene features, providing high-quality feature data for subsequent generation of multi-modal data sets, and helping to improve the depth and accuracy of multi-modal analysis and providing more valuable information for agricultural production decision-making.

[0113] Working principle

[0114] 1. Data acquisition: Through various sensors, agricultural scene data is collected from multiple dimensions such as images, weather, soil, and crop growth conditions, to obtain raw data that comprehensively reflects the agricultural production environment and crop growth status.

[0115] 2. Data preprocessing: The collected multi-dimensional agricultural scene data is cleaned to remove noise and error data; multi-source data calibration is performed to unify the data format and unit of different data sources; time series synchronization and spatial registration are achieved to align the data in time and space dimensions, and a standardized data set is output to provide a high-quality data foundation for subsequent analysis.

[0116] 3. Agricultural scene recognition: Image features and time series features are extracted from the standardized data set, and the extracted image features and time series features are connected to form a first comprehensive feature vector. Then, the first comprehensive feature vector is input into a soft voting ensemble classifier composed of multiple basic classifiers (such as random forest classifier, support vector machine classifier, naive Bayes classifier, etc.), each basic classifier performs agricultural scene prediction, and finally the prediction results of each basic classifier are weighted and averaged to obtain the agricultural scene recognition result.

[0117] 4. Scene analysis model selection and adjustment: The scene analysis model management library calculates the scene matching score of each scene recognition model based on the agricultural scene recognition results, and considers the computational complexity of the model and the current available computing resources of the system to calculate the feasibility score. Select the top a scene analysis models and their parameters with the highest score. During system operation, continuously monitor model evaluation indicators, and when the indicators change, trigger the model switching process to adjust the scene analysis model set in real time to ensure the best match between the model and the actual scene.

[0118] 5. Multi-modal analysis: Based on the selected scene analysis model and its parameters, feature recognition is performed on the standardized data set to extract the second comprehensive feature vector of the agricultural scene data. Then, based on the vector, agricultural features are extracted, and finally a multi-modal data set of agricultural scene data containing rich information is obtained, such as identifiers of data in the standardized data set, lists of applied enhancement operations and their parameters, feature importance maps, lists of agricultural features and their confidence, feature consistency check results, etc.

[0119] 6. Quality evaluation: Quality evaluation is performed on the generated multi-modal data set, including rule verification, multi-dimensional quality evaluation, credibility scoring, and anomaly detection, etc. to ensure that the quality of the data set meets the requirements and provides reliable data support for subsequent agricultural production decisions.

[0120] Method of use

[0121] 1. Deploy the system: Deploy the multi-modal data-based agricultural scene recognition and verification system on suitable hardware devices to ensure stable operation of the system and connection with various sensors to ensure smooth data transmission.

[0122] 2. Data collection configuration: Configure multiple sensors according to the needs of actual agricultural production scenes to determine the frequency, range, and other parameters of data collection from image, weather, soil, and crop growth conditions, etc. Start the data collection process to allow sensors to continuously collect agricultural scene data.

[0123] 3. Run the system: Start the system, which automatically pre-processes the collected data into a standardized data set. Then perform agricultural scene recognition to obtain recognition results. The scene analysis model management library selects appropriate scene analysis models and their parameters based on the recognition results to perform multi-modal analysis on the standardized data set and generate a multi-modal data set. Finally, perform quality evaluation on the multi-modal data set and output the evaluation results.

[0124] 4. Result application and feedback: the agricultural producer or manager formulates agricultural production decisions such as irrigation, fertilization, pest control and the like according to the agricultural scene recognition result, the multi-modal data set and the quality evaluation result output by the system. Meanwhile, feedback information (such as the effect of decision implementation) in the actual production process is fed back to the system so as to optimize and improve the system.

[0125] The above merely describes preferred embodiments of the present application, but does not limit the embodiments and protection scope of the present application. It should be understood by those skilled in the art that any equivalent replacement and obvious change made according to the content of the present application should be included in the protection scope of the present application.

Claims

1. A method for agricultural scene recognition and verification based on multimodal data, characterized in that: The following steps are involved: Performing data preprocessing on multidimensional agricultural scene data to output a standardized dataset, wherein the data preprocessing includes at least data cleaning, multi-source data calibration, time series synchronization, and spatial registration. The standardized dataset is a multidimensional tensor containing agricultural data that is fully aligned in time, space, and spectral dimensions; Agricultural scene recognition is performed based on the standardized data set to obtain an agricultural scene recognition result. The specific operation is to extract image features and time series features from the standardized data set, and connect the image features and time series features to form a first comprehensive feature vector. Then, a soft voting integrated classifier is used to perform probability prediction on the first comprehensive feature vector to obtain the agricultural scene recognition result. The soft voting integrated classifier includes multiple basic classifiers. The first comprehensive feature vector is input into different basic classifiers respectively, and agricultural scene prediction results are obtained respectively. The agricultural scene prediction results are weighted averaged to obtain the agricultural scene recognition result. The calculation formula is: Where: P(y) represents the probability distribution of agricultural scene categories; w i represents the weight of the i-th basic classifier; P i (y) represents the agricultural scene prediction result of the i-th basic classifier for the first comprehensive feature vector; The scene analysis model management library manages and maintains different scene analysis models, dynamically selects a set of matching scene analysis models and their parameters based on the agricultural scene recognition results, and adjusts the set of scene analysis models in real time. The process of dynamically selecting a set of matching scene analysis models and their parameters is as follows: according to the agricultural scene category probability distribution, calculate the scene matching score of each scene recognition model. The scene matching calculation formula is as follows: Where: P(yj) represents the probability of the jth scene category, C m (yj) is the coverage of the j-th scene category by the scene recognition model m; based on the computational complexity of each scene recognition model and the currently available computing resources of the system, the feasibility score of each scene recognition model is calculated; the weighted sum of the scene matching score and the feasibility score of each scene recognition model is calculated, and the set of the top a scene analysis models with the highest scores and their parameters are selected. The set of scene analysis models is adjusted in real time when the evaluation indicators of the scene recognition models in the set change while being continuously monitored, which will trigger a model switching process and adjust the scene recognition models in the set; Based on the set of scene analysis models and their parameters, a multimodal analysis is performed on the standardized data set to obtain a multimodal data set of the agricultural scene data, specifically including performing feature recognition on the standardized data set to extract a second comprehensive feature vector of the agricultural scene data, extracting agricultural features of the agricultural scene data based on the second comprehensive feature vector, and obtaining a multimodal data set of the agricultural scene data based on the agricultural features, wherein the data information in the multimodal data set includes at least: an identifier of the data in the standardized data set, a list of applied enhancement operations and their parameters, a feature importance map, a list of the agricultural features and their confidence levels, and a feature consistency check result; A quality assessment is performed on the multimodal dataset, wherein the quality assessment at least includes rule verification, multidimensional quality assessment, credibility scoring, and anomaly detection.

2. The agricultural scene recognition and verification method based on multimodal data according to claim 1 is characterized in that: Before data preprocessing, it also includes: collecting agricultural scene data from multiple dimensions through multiple sensors, and the multiple dimensions include at least images, weather, soil, and crop growth conditions.

3. The agricultural scene recognition and verification method based on multimodal data according to claim 1 is characterized in that: The basic classifiers in the soft voting ensemble classifier include at least a random forest classifier, a support vector machine classifier, and a naive Bayes classifier, and the weights w of different basic classifiers are i It is determined comprehensively based on the accuracy, recall rate and F1 value of each basic classifier on the training set.

4. The agricultural scene recognition and verification method based on multimodal data according to claim 1 is characterized in that: When calculating the feasibility score of the scene recognition model, the computational complexity factors considered include at least the number of model parameters, the number of computing layers, and memory usage. The system's currently available computing resources include at least CPU usage, GPU memory remaining, and memory remaining.

5. The agricultural scene recognition and verification method based on multimodal data according to claim 1 is characterized in that: When performing feature recognition on the standardized data set, a deep learning model is used. The deep learning model includes at least a convolutional neural network and a recurrent neural network, and is pre-trained on a large-scale agricultural data set through transfer learning.

6. An agricultural scene recognition and verification system based on multimodal data, characterized in that: The system comprises: Agricultural scene data preprocessing module: used to preprocess multidimensional agricultural scene data and output a standardized dataset. The data preprocessing includes at least data cleaning, multi-source data calibration, time series synchronization and spatial registration. The standardized dataset is a multidimensional tensor containing agricultural data that is fully aligned in time, space and spectral dimensions. Agricultural scene recognition module: used to perform agricultural scene recognition on the standardized data set to obtain an agricultural scene recognition result. The specific operation is to extract image features and time series features from the standardized data set, and connect the image features and time series features to form a first comprehensive feature vector. Then, a soft voting integrated classifier is used to perform probability prediction on the first comprehensive feature vector to obtain the agricultural scene recognition result. The soft voting integrated classifier includes multiple basic classifiers. The first comprehensive feature vector is input into different basic classifiers respectively, and agricultural scene prediction results are obtained respectively. The agricultural scene prediction results are weighted averaged to obtain the agricultural scene recognition result. The calculation formula is: Where: P(y) represents the probability distribution of agricultural scene categories; w i represents the weight of the i-th basic classifier; P i (y) represents the agricultural scene prediction result of the i-th basic classifier for the first comprehensive feature vector; Scene analysis model management library: used to manage and maintain different scene analysis models, dynamically select a set of matching scene analysis models and their parameters based on the agricultural scene recognition results, and adjust the set of scene analysis models in real time. The process of dynamically selecting a set of matching scene analysis models and their parameters is as follows: based on the probability distribution of the agricultural scene category, calculate the scene matching score of each scene recognition model. The scene matching calculation formula is as follows: Where: P(yj) represents the probability of the jth scene category, C m (yj) is the coverage of the j-th scene category by the scene recognition model m; based on the computational complexity of each scene recognition model and the currently available computing resources of the system, the feasibility score of each scene recognition model is calculated; the weighted sum of the scene matching score and the feasibility score of each scene recognition model is calculated, and the set of the top a scene analysis models with the highest scores and their parameters are selected. The set of scene analysis models is adjusted in real time when the evaluation indicators of the scene recognition models in the set change while being continuously monitored, which will trigger a model switching process and adjust the scene recognition models in the set; A scene analysis model collection module is used to perform multimodal analysis on the standardized data set based on the set of scene analysis models and their parameters to obtain a multimodal data set of the agricultural scene data, specifically including performing feature recognition on the standardized data set to extract a second comprehensive feature vector of the agricultural scene data, extracting agricultural features of the agricultural scene data based on the second comprehensive feature vector, and obtaining a multimodal data set of the agricultural scene data based on the agricultural features. The data information in the multimodal data set includes at least: an identifier of the data in the standardized data set, a list of applied enhancement operations and their parameters, a feature importance map, a list of agricultural features and their confidence levels, and a feature consistency check result. Quality assessment module: used to perform quality assessment on the multimodal dataset, wherein the quality assessment includes at least rule verification, multidimensional quality assessment, credibility scoring and anomaly detection.

7. The agricultural scene recognition and verification system based on multimodal data according to claim 6 is characterized in that: It also includes a data acquisition module, which uses multiple sensors to collect agricultural scene data from multiple dimensions. The multiple dimensions include at least images, weather, soil, and crop growth conditions.

8. The agricultural scene recognition and verification system based on multimodal data according to claim 6 is characterized in that: The basic classifiers in the soft voting integrated classifier in the agricultural scene recognition module include at least a random forest classifier, a support vector machine classifier, and a naive Bayes classifier, and the weights w of different basic classifiers are i It is determined comprehensively based on the accuracy, recall rate and F1 value of each basic classifier on the training set.

9. The agricultural scene recognition and verification system based on multimodal data according to claim 6, characterized in that: When calculating the feasibility score of the scene recognition model, the scene analysis model management library considers computational complexity factors including at least the number of model parameters, the number of computational layers, and memory usage. The system's currently available computing resources include at least CPU usage, GPU memory remaining, and memory remaining.

10. The agricultural scene recognition and verification system based on multimodal data according to claim 6, characterized in that: When performing feature recognition on the standardized data set, the scene analysis model set module adopts a deep learning model, which includes at least a convolutional neural network and a recurrent neural network, and is pre-trained on a large-scale agricultural data set through transfer learning.

Images