Agricultural scene identification and verification method and system based on multi-modal data

Abstract

The invention provides an agricultural scene recognition and verification method and system based on multi-modal data, and the method comprises the steps: collecting multi-dimensional agricultural scene data, carrying out the data preprocessing of the multi-dimensional agricultural scene data, outputting a standardized data set, carrying out the agricultural scene recognition of the standardized data set, and obtaining an agricultural scene recognition result. The scene analysis model management library manages and maintains different scene recognition models, dynamically selects a set of matched scene analysis models and parameters thereof based on the agricultural scene recognition result, adjusts the set of scene analysis models in real time, and adjusts the set of scene analysis models in real time based on the set of scene analysis models and parameters thereof. And performing multi-modal analysis on the standardized data set to obtain a multi-modal data set, and finally performing quality evaluation on the multi-modal data set. Through multi-modal data acquisition, intelligent processing and verification, the accuracy and adaptability of agricultural scene recognition can be effectively improved, and reliable data support is provided for precision agriculture.

Description

Technical Field

[0001] The present invention belongs to the technical field of precision agriculture, and in particular relates to an agricultural scene recognition and verification method and system based on multimodal data. Background Art

[0002] In the agricultural field, it is often necessary to monitor crops in real time to provide workers with information on crop growth and ensure the normal growth of crops. However, existing monitoring solutions have the following problems:

[0003] (1) The preprocessing method of agricultural scene data is cumbersome and inefficient;

[0004] (2) The accuracy of agricultural scene recognition models is low, and the scene analysis models cannot be switched in real time or the switching efficiency is low;

[0005] (3) The accuracy of the monitored and extracted agricultural scenario data is low, and there is no corresponding data result verification and evaluation plan, which makes it of low reference value in agricultural scenarios.

[0006] Therefore, we need to develop an agricultural scene identification and verification method and system based on multimodal data, which can collect multimodal agricultural scene data and perform intelligent processing and verification, effectively improve the accuracy and adaptability of agricultural scene analysis, and provide reliable data support for precision agriculture. Summary of the invention

[0007] The purpose of the present invention is to provide an agricultural scene recognition and verification method and system based on multimodal data to solve the problems of low accuracy of agricultural scene data monitored and extracted by the existing monitoring system mentioned in the above background technology.

[0008] To achieve the above object, the present invention provides an agricultural scene recognition and verification method based on multimodal data, and the method is specifically as follows:

[0009] Preprocess multi-dimensional agricultural scene data and output standardized data sets;

[0010] Perform agricultural scene recognition based on the standardized data set to obtain an agricultural scene recognition result;

[0011] The scene analysis model management library manages and maintains different scene analysis models, dynamically selects a set of matching scene analysis models and their parameters based on the agricultural scene recognition results, and adjusts the set of scene analysis models in real time;

[0012] Based on the set of scene analysis models and their parameters, multimodal analysis is performed on the standardized data set to obtain a multimodal data set of the agricultural scene data;

[0013] A quality assessment is performed on the multimodal dataset.

[0014] Based on the aforementioned scheme, the data preprocessing includes but is not limited to data cleaning, multi-source data calibration, time series synchronization and spatial alignment, and the standardized data set is a multidimensional tensor, including but not limited to agricultural data that is fully aligned in time, space and spectral dimensions.

[0015] Based on the above scheme, the agricultural scene recognition is specifically performed as follows:

[0016] Extracting image features and time series features from the standardized data set, and connecting the image features and time series features to form a first comprehensive feature vector;

[0017] The agricultural scene recognition result is obtained by performing probability prediction on the first comprehensive feature vector through a soft voting integrated classifier.

[0018] Based on the above scheme, the soft voting integrated classifier includes multiple basic classifiers, the first comprehensive feature vector is input into different basic classifiers respectively, and agricultural scene prediction results are obtained respectively;

[0019] The agricultural scene prediction results are weighted averaged to obtain the agricultural scene recognition results: ,in represents the probability distribution of agricultural scene categories, represents the weight of the i-th base classifier, Represents the agricultural scene prediction result of the i-th basic classifier for the first comprehensive feature vector.

[0020] Based on the above solution, the set of scene analysis models and their parameters that are dynamically selected to match are specifically:

[0021] According to the agricultural scene category probability distribution, the scene matching score of each scene recognition model is calculated. The scene matching calculation formula is as follows: ,in It is The probability of scene categories, The scene recognition model is The coverage of scene categories;

[0022] Calculating a feasibility score of each of the scene recognition models based on the computational complexity of each of the scene recognition models and currently available computing resources of the system;

[0023] The weighted sum of the scene matching score and the feasibility score of each scene recognition model is calculated, and a set of the top A scene analysis models with the highest scores and their parameters are selected.

[0024] Based on the above-mentioned scheme, the real-time adjustment of the set of scene analysis models is that when the evaluation indicators of the scene recognition models in the set change while being continuously monitored, it will trigger the model switching process to adjust the scene recognition models in the set.

[0025] Based on the above scheme, the multimodal analysis of the standardized data set specifically includes:

[0026] Performing feature recognition on the standardized data set to extract a second comprehensive feature vector of the agricultural scene data;

[0027] Extracting agricultural features of the agricultural scene data based on the second comprehensive feature vector, and obtaining a multimodal dataset of the agricultural scene data based on the agricultural features;

[0028] The data information in the multimodal dataset includes but is not limited to: identifiers of the data in the standardized dataset, a list of applied enhancement operations and their parameters, a feature importance map, a list of agricultural features and their confidence levels, and feature consistency check results.

[0029] Based on the above scheme, the quality assessment of the multimodal data set includes but is not limited to: rule verification, multidimensional quality assessment, credibility scoring and anomaly detection.

[0030] In addition, the present invention also provides an agricultural scene recognition and verification system based on multimodal data, and the system specifically includes:

[0031] Agricultural scene data preprocessing module: used to preprocess multi-dimensional agricultural scene data and output standardized data sets;

[0032] Agricultural scene recognition module: used to perform agricultural scene recognition on the standardized data set to obtain agricultural scene recognition results;

[0033] Scenario analysis model management library: used to manage and maintain different scenario analysis models, dynamically select a set of matching scenario analysis models and their parameters based on the agricultural scenario recognition results, and adjust the set of scenario analysis models in real time;

[0034] A scene analysis model collection module: used to perform multimodal analysis on the standardized data set based on the scene analysis model collection and its parameters to obtain a multimodal data set of the agricultural scene data;

[0035] Quality assessment module: used to perform quality assessment on the multimodal dataset.

[0036] Based on the above solution, the scenario analysis model management library includes:

[0037] A scene analysis model selection unit: used to dynamically select a set of matching scene analysis models and their parameters based on the agricultural scene recognition result;

[0038] A scene analysis model dynamic adjustment unit is used to monitor the operating status of the set of scene analysis models in real time, and adjust the scene recognition models in the set when the evaluation indicators of the scene recognition models in the set change.

[0039] Compared with the prior art, the present invention has the following advantages and effects:

[0040] (1) Improve the efficiency of agricultural scene data preprocessing by performing data cleaning, multi-source data calibration, time series synchronization, and spatial registration on multi-dimensional agricultural scene data;

[0041] (2) By using a soft voting ensemble classifier, accurate agricultural scene recognition results are obtained, and the scene analysis model can be dynamically adjusted based on real-time evaluation;

[0042] (3) By extracting agricultural features from agricultural scene data, a multimodal dataset is obtained, which includes feature consistency check results to ensure the accuracy of the agricultural scene data extracted by the scene analysis model. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0044] Figure 1 It is a flow chart of an agricultural scene recognition and verification method based on multimodal data provided by an embodiment of the present invention;

[0045] Figure 2 is a flow chart of a method for dynamically selecting and adjusting a scene analysis model provided by an embodiment of the present invention;

[0046] Figure 3 It is a structural diagram of an agricultural scene recognition and verification system based on multimodal data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to more clearly explain the purpose, technical solutions and advantages of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. The example implementation methods can be implemented in various forms and should not be understood as being limited to the examples described herein. On the contrary, these implementation methods are provided to make the present invention more comprehensive and complete, and to fully convey the concepts of the example implementation methods to those skilled in the art.

[0048] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present invention. However, those skilled in the art will appreciate that the technical solution of the present invention can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring various aspects of the present invention.

[0049] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0050] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0051] The present invention is described in detail below in conjunction with specific embodiments:

[0052] Example 1

[0053] As attached Figure 1 As shown, Embodiment 1 of the present invention provides an agricultural scene recognition and verification method based on multimodal data, and the specific steps of the method are as follows:

[0054] Step S1, preprocessing the multidimensional agricultural scene data and outputting a standardized data set;

[0055] Step S2, performing agricultural scene recognition based on the standardized data set to obtain an agricultural scene recognition result;

[0056] Step S3: The scene analysis model management library manages and maintains different scene analysis models, dynamically selects a set of matching scene analysis models and their parameters based on the agricultural scene recognition results, and adjusts the set of scene analysis models in real time;

[0057] Step S4: Based on the set of scene analysis models and their parameters, perform multimodal analysis on the standardized data set to obtain a multimodal data set of the agricultural scene data;

[0058] Step S5: performing quality assessment on the multimodal dataset.

[0059] Preferably, before step S1, the method further includes: installing a variety of high-precision sensors on agricultural machinery to collect multi-dimensional agricultural scene data;

[0060] Specifically, the high-precision sensors include but are not limited to high-resolution cameras, thermal imagers, and multispectral cameras. The high-resolution cameras are used to capture visible light images of crops and provide information on crop growth, pests and diseases, etc. The thermal imagers are used to collect farmland temperature distribution data, which helps monitor crop moisture conditions and soil temperature. The multispectral cameras are used to obtain spectral information in different bands, which can be used to assess crop health and nutritional levels. The above sensors are connected to the system main control unit through a customized data acquisition interface, and a unified timestamp mechanism is used to ensure data synchronization. The sampling frequency of the sensors can be dynamically adjusted according to agricultural production needs, usually between 1Hz and 10Hz.

[0061] Preferably, in step S1, the data preprocessing includes but is not limited to data cleaning, multi-source data calibration, time series synchronization and spatial registration, and the standardized data set is a multidimensional tensor, including but not limited to agricultural data that is fully aligned in time, space and spectral dimensions, as follows:

[0062] Step S101, data cleaning: perform integrity check on the agricultural scene data. The system calculates the MD5 checksum of each data packet in the agricultural scene data and compares it with the checksum stored in the data packet header to ensure that the data has not been tampered with during transmission;

[0063] Specifically, for data packets that fail verification, the system will request the data acquisition module to retransmit; data packets that pass verification enter the outlier processing link, which uses the improved moving median method for outlier detection. The system applies a sliding window of length 15 to the time series data of each sensor, calculates the median and median absolute deviation within the window, and marks the data points that meet the conditions as outliers. The processing method varies depending on the data type: for continuous environmental parameter data such as temperature and humidity, linear interpolation is used to estimate outliers; for discrete spectral data, the nearest neighbor method is used for filling.

[0064] Step S102, multi-source data calibration: systematically correct the agricultural scene data according to the inherent errors of different sensors and the influence of environmental factors.

[0065] Specifically, for visible light and multispectral images, dark current correction and flat field correction are performed. Dark current correction is achieved by subtracting images acquired under completely dark conditions, while flat field correction uses images acquired with a uniform light source to eliminate spatial response variations caused by optical system inhomogeneity. The corrected image intensity The calculation is as follows:

[0066]

[0067] in, is the original image intensity, is the dark current image, is a flat-field image, is the normalization coefficient;

[0068] Specifically, for thermal infrared images, radiation correction based on the atmospheric transmission step is performed. Taking into account the influence of atmospheric absorption, scattering and the emissivity of the target object, the real target temperature is obtained through iterative calculation. During the correction process, the system will use the real-time meteorological data provided by the auxiliary sensing unit, including but not limited to temperature, humidity, and air pressure as correction input parameters.

[0069] Step S103, time series synchronization: resampling all data streams of the agricultural scene data to a frequency of 10 Hz;

[0070] Specifically, during the resampling process, for data streams above 10Hz, such as some environmental parameters, the sliding average method is used for downsampling; for data streams below 10Hz, the cubic spline interpolation method is used for upsampling. If there is still a slight time offset in the resampled data stream, the time offset is estimated by calculating the cross-correlation function, and accurate alignment is achieved through timestamp adjustment. The consistency error of the synchronized data in the time dimension is controlled within ±0.1ms.

[0071] Step S104, spatial registration: extracting feature points from the visible light image of the agricultural scene data, and using an improved corner point detection algorithm, wherein the corner point detection algorithm combines the stability of the corner point detector and the efficiency of the corner point detector;

[0072] Specifically, the number of extracted feature points is preferably between 1000 and 2000, covering the main structure of the image; the system searches for corresponding feature points on thermal infrared and multispectral images, and uses a similarity measure based on mutual information for matching. The matching process adopts a pyramid strategy, which is gradually optimized from coarse to fine to improve the robustness and efficiency of the registration; finally, the system calculates the spatial transformation matrix based on the matching point pairs, and uses the weighted least squares method to give greater weights to matching points with high confidence. The transformation matrix contains translation, rotation and scale transformation parameters, which are used to map thermal infrared and multispectral images to the coordinate system of visible light images. The registration accuracy is evaluated by the root mean square error (RMSE), and the RMSE is preferably controlled within 0.5 pixels.

[0073] Preferably, the multidimensional tensor in step S1 contains agricultural data that are fully aligned in time, space and spectral dimensions; specifically, for each time point at a sampling rate of 10 Hz, a data block is output, which contains: a corrected visible light image, a corrected thermal infrared image, a corrected multispectral image and a synchronized environmental parameter vector.

[0074] Preferably, the agricultural scene recognition in step S2 is specifically performed as follows:

[0075] Step S201, extracting image features and time series features from the standardized data set, and connecting the image features and time series features to form a first comprehensive feature vector;

[0076] Specifically, the feature extraction process adopts a multi-scale and multi-modal approach. For image data such as visible light, thermal infrared and multi-spectral, a hierarchical convolutional structure can be used for feature extraction; for time series environmental parameter data, a long short-term memory network structure can be used for feature extraction; finally, the system connects the image features and time series features to form the first comprehensive feature vector, denoted as feature vector x.

[0077] Step S202: Probabilistically predicting the first comprehensive feature vector by using a soft voting integrated classifier to obtain the agricultural scene recognition result;

[0078] Specifically, the soft voting integrated classifier includes a plurality of basic classifiers, the first comprehensive feature vector is input into different basic classifiers respectively, and agricultural scene prediction results are obtained respectively; the agricultural scene prediction results are weighted averaged to obtain the agricultural scene recognition result: ,in represents the probability distribution of agricultural scene categories, represents the weight of the i-th base classifier, Represents the agricultural scene prediction result of the i-th basic classifier for the first comprehensive feature vector.

[0079] Exemplarily, the soft voting ensemble classifier includes three basic classifiers: decision tree, support vector machine and multi-layer perceptron. Each basic classifier independently classifies the first comprehensive feature vector, that is, the extracted feature vector Input to each base classifier:

[0080]

[0081] in, Indicates The decision function of the i-th base classifier, is the feature vector of the i-th base classifier The prediction result is usually a probability distribution vector, which represents the feature vector The probability of belonging to each agricultural scene category.

[0082] During the soft voting process, the system dynamically adjusts the weight of each classifier. The weight calculation formula is as follows:

[0083]

[0084] in, It is The weight of the classifier, is its recent average classification error rate, is an adjustable smoothing parameter, usually set to 0.1. represents the sum index, traversing all classifiers; finally, the system performs a weighted average on these prediction results to obtain the probability distribution of the scene category:

[0085]

[0086] Among them, Y represents the final probability distribution of scene categories. The dynamic weight adjustment mechanism of the present invention can adapt to the changes in the performance of each classifier in different scenes. The system defines 50 typical agricultural scene categories, including different crop types, growth stages, weather conditions and farming activities. The output of scene recognition is a 50-dimensional probability vector, which indicates the probability that the current data belongs to each scene category.

[0087] Preferably, the scenario analysis model management library in step S3 maintains a dynamically updated scenario analysis model library, which contains scenario analysis models for different agricultural tasks such as crop growth status estimation, pest and disease detection, yield prediction, etc. and different agricultural scenarios. Each model has a metadata structure that records information such as the scenario category applicable to the model, input data requirements, computational complexity, and historical performance indicators; the scenario analysis model library adopts a hierarchical storage structure, with the top layer being the task category, the middle layer being the scenario category, and the bottom layer being the specific model.

[0088] Preferably, the set of scene analysis models and their parameters that are matched are dynamically selected and adjusted in real time in step S3, as shown in the attached Figure 2 FIG. 1 is a flow chart of a method for dynamically selecting and adjusting a scene analysis model in this embodiment, which specifically includes:

[0089] Step S301: Calculate the scene matching score of each scene recognition model according to the agricultural scene category probability distribution. The scene matching calculation formula is as follows: ,in It is The probability of scene categories, The scene recognition model is The coverage of scene categories;

[0090] Step S302: Calculate a feasibility score of each scene recognition model based on the computational complexity of each scene recognition model and currently available computing resources of the system;

[0091] Step S303, calculating the weighted sum of the scene matching score and the feasibility score of each scene recognition model, selecting the set of the top A scene analysis models with the highest scores and their parameters, wherein A is preferably between 3 and 5;

[0092] Exemplarily, the dynamic selection of the scene analysis model is based on the agricultural scene recognition results and the current system state, and the most suitable scene analysis model set is selected from the scene analysis model library. First, based on the identified scene probability vector, that is, the agricultural scene category probability distribution, the system calculates the scene matching score of each scene analysis model, where the scene matching calculation formula is as follows:

[0093]

[0094] in, It is The probability of scene categories, It is the coverage of the model for the scenario, and the coverage value is between 0 and 1. Secondly, the system considers the computational complexity of the model and the currently available computing resources, and calculates the feasibility score of each model. The final model selection score is the weighted sum of the scenario matching degree and the feasibility score, and the weight is dynamically adjusted according to the urgency of the current task. The system selects the model set with the highest score, which usually contains 3 to 5 complementary models.

[0095] Step S304, the real-time adjustment of the set of scene analysis models is that when the evaluation indicators of the scene recognition models in the set change while being continuously monitored, a model switching process will be triggered to adjust the scene recognition models in the set, i.e., for dynamic switching of scene recognition models.

[0096] Exemplarily, the running status of the selected scenario analysis model is continuously monitored, and the evaluation indicators of the running status include accuracy, recall rate, F1 score, calculation time, etc. The system uses a sliding window mechanism to calculate the above evaluation indicators, and the window size is dynamically adjusted according to the characteristics of the task, between 1000 and 10000 samples. When it is detected that the performance of the scenario analysis model has dropped significantly, such as the F1 score drops by more than 5%, the system will trigger the scenario analysis model switching process. The model switching adopts a smooth transition strategy. The old and new models run in parallel for a period of time. The output results are fused by weighted averaging, and the weights are gradually transferred from the old model to the new model to ensure the continuity of the output.

[0097] Exemplarily, a conditional generative adversarial network (cGAN) can be used to implement model quantization and hot switching techniques, combined with a scene recognition algorithm, to dynamically select the most suitable pre-trained model, and to evaluate the model performance in real time to adjust the selection strategy.

[0098] It should be noted that the output after dynamically selecting the scenario analysis model includes: the selected scenario analysis model set, the current weight attached to each model, and the preprocessing parameters required for model operation, such as feature normalization coefficients, input data reshaping parameters, etc.

[0099] Preferably, the multimodal analysis of the standardized data set in step S4 specifically includes:

[0100] Step S401, performing feature recognition on the standardized data set to extract a second comprehensive feature vector of the agricultural scene data;

[0101] Exemplarily, according to the requirements of the selected scene analysis model set, the input standardized data set is preprocessed, and the preprocessing steps include data reshaping, normalization and standardization. Normalization adopts an adaptive method to dynamically adjust the normalization parameters according to the statistical characteristics within the sliding window, such as 1000 samples; the standardization process uses the following formula: ,in, is the original data, and are the mean and standard deviation in the sliding window, respectively. A multi-branch parallel structure is used for feature extraction. The multi-branch includes visible light, thermal infrared, multi-spectral and environmental parameters. Each branch corresponds to a data modality. The visible light and multi-spectral branches use an improved residual network structure. The residual network structure contains 5 residual blocks, each residual block contains two 3x3 convolutional layers and a jump connection; the thermal infrared branch uses a dense connection network, which contains 4 dense blocks, each dense block contains 6 convolutional layers, and the growth rate is 32; the environmental parameter branch uses a bidirectional gated recurrent unit (GRU) network with a hidden state dimension of 64. The outputs of each branch are fused through the attention mechanism to obtain a comprehensive feature vector, namely the second comprehensive feature vector, with a dimension of 2048.

[0102] Step S402: identifying agricultural features of the agricultural scene data based on the second comprehensive feature vector, and obtaining a multimodal data set of the agricultural scene data based on the agricultural features, specifically by generating a feature importance graph, performing data enhancement, and performing feature consistency check on the agricultural features;

[0103] Exemplarily, based on the second comprehensive feature vector to identify key agricultural features, an unsupervised clustering algorithm such as improved K-means++ can be used to divide the feature space into multiple subspaces, each of which represents a type of agricultural feature such as leaf morphology, fruit color, disease spots, etc., and supervised learning methods such as random forests are applied in each subspace for fine classification. The system maintains a dynamically updated agricultural feature knowledge base, which contains hundreds of predefined feature categories and their descriptors. The agricultural feature recognition results are expressed in the form of a probability vector, and each element corresponds to the probability of existence of a feature category.

[0104] For example, based on the agricultural characteristics, an improved Class Activation Mapping technique is used to identify important features, generate a feature importance map, and protect the important features. The calculation formula of the feature importance map is as follows: ,in, It is The weight of feature channels, is the corresponding feature map. Weight Obtained through back propagation optimization, it reflects the contribution of each feature channel to the final decision. Based on the feature importance graph, the system defines three levels of protection strength: high, medium, and low. The high importance area is the agricultural features with importance values ​​in the top 20%, which are processed losslessly. The medium importance area is the agricultural features with importance values ​​in the top 20% to 60%, which allow slight reversible transformations. The low importance area is the agricultural features with importance values ​​in the bottom 40%, which can be transformed to a greater extent.

[0105] Exemplarily, data enhancement is performed on the agricultural features, and specific steps include geometric transformations such as rotation, scaling, translation, lighting adjustment, noise addition, etc. For example, the rotation operation will take into account the direction of the crop row to ensure that the enhanced image still conforms to the actual planting pattern; the lighting adjustment simulates the changes in lighting under different time and weather conditions, and the range is determined based on historical meteorological data; the noise addition uses a complex noise model that simulates sensor characteristics, including a combination of Gaussian noise, salt and pepper noise, and stripe noise. The intensity of data enhancement is dynamically adjusted through an adaptive controller, which takes into account factors such as current model performance, data diversity, and feature protection requirements.

[0106] For example, feature consistency check: for the feature importance graph and the agricultural features after data enhancement, ensure that the data enhancement and feature protection process will not destroy the key agricultural features. Specifically, an improved structural similarity index (SSIM) is used to perform feature consistency check. The formula is as follows:

[0107] SSIM(x,y) = [l(x,y)] α · [c(x,y)] β · [s(x,y)] γ

[0108] in, , and represent brightness, contrast and structural similarity respectively, , and is an adjustable weight parameter, is the original agricultural feature image, It is an agricultural feature image that has been enhanced or protected; the system dynamically adjusts these weights according to different feature types. For example, for color-sensitive features such as fruit maturity, the weight of the brightness term will be increased. SSIM calculation is performed at multiple scales to capture global and local similarities. If the SSIM of a certain area is lower than the preset threshold, the system will trigger a reprocessing mechanism to adjust the enhancement parameters or protection strength. The preset threshold is usually set to 0.85.

[0109] It should be noted that it also includes: implementing a straight line detection algorithm based on Hough transform, identifying regular crop rows, designing a curve fitting method, processing irregular planting patterns, designing an adaptive threshold algorithm, and coping with different lighting and growth stages.

[0110] Preferably, the data information in the multimodal dataset in step 4 includes but is not limited to: identifiers of the data in the standardized dataset, a list of applied enhancement operations and their parameters, a feature importance map, a list of agricultural features and their confidences, and feature consistency check results;

[0111] It should be noted that after data enhancement and protection, the multimodal image analysis and feature protection output the multimodal data set, in which each data sample is accompanied by a metadata structure, and the data information is contained in the metadata structure.

[0112] Preferably, the quality assessment in step 5 includes but is not limited to: rule verification, multi-dimensional quality assessment, credibility scoring and anomaly detection, specifically:

[0113] Step S501, rule verification: checking the logic and rationality of the multimodal data set based on a predefined agricultural knowledge rule base;

[0114] For example, rule verification checks the logic and rationality of the processing results based on a predefined agricultural knowledge rule base. The rule base adopts a semantic network structure and contains multi-level agricultural knowledge representation. The top layer is the basic concept nodes such as crops, environmental factors, agronomic measures, etc. The middle layer is the relationship nodes such as growth stage conversion rules, pest and disease development conditions, etc. The bottom layer is the specific constraint rules. The rule expression adopts an improved predicate logic form, for example:

[0115] The rule verification process uses a forward chain reasoning mechanism to deduce possible conclusions from the observed features. Fuzzy logic is used to handle uncertainty in the reasoning process. A confidence value is calculated for each reasoning step. The final verification result includes the degree of compliance with the rule and the corresponding explanatory text. The value of the degree is between 0 and 1.

[0116] Step S502, multi-dimensional quality assessment: assessing the quality of the multimodal dataset from multiple perspectives, including image quality, feature fidelity, data diversity, and temporal consistency. The assessment results of each dimension are mapped to a score between 0 and 1, and a comprehensive assessment quality score is obtained by weighted summation.

[0117] Specifically, the weights of the evaluation dimensions are dynamically adjusted through an adaptive learning algorithm to meet the needs of different agricultural tasks.

[0118] Step S503, credibility scoring: based on the rule verification results, quality assessment scores and scenario analysis model performance indicators, a method combining hierarchical analysis method and fuzzy comprehensive evaluation is used to calculate the final credibility score of the multimodal data set;

[0119] For example, the credibility score is calculated based on the rule verification results, quality assessment scores and model performance indicators. The credibility score is a combination of the analytic hierarchy process (AHP) and fuzzy comprehensive evaluation. First, a judgment matrix is ​​constructed:

[0120]

[0121]

[0122] The matrix represents the relative importance of the three indicators of "rule verification", "quality assessment" and "model performance". The elements in the matrix represent the importance of row elements relative to column elements. For example, the importance of "rule verification" relative to "quality assessment" is 2 times, and the importance of "rule verification" relative to "model performance" is 3 times. Then the eigenvector is calculated as the weight vector, and finally the fuzzy comprehensive evaluation method is used to consider the fuzziness of each factor to obtain the final credibility score. The credibility score ranges from 0 to 100 and is divided into five levels: extremely high, the credibility score is between 90 and 100; high, the credibility score is between 80 and 90; medium, the credibility score is between 70 and 80; low, the credibility score is between 60 and 70; extremely low, the credibility score is less than 60.

[0123] Step S504, anomaly detection: responsible for identifying potential anomalies in the multimodal data set, independently applying the local anomaly factor method on each data modality to identify local anomalies.

[0124] Exemplarily, anomaly detection is responsible for identifying potential anomalies in the processing results. First, the Local Outlier Factor (LOF) method is applied independently on each data modality to identify local outliers. Then, the anomaly indications of each modality are comprehensively considered through a multimodal fusion strategy. The fusion strategy adopts a weighted voting mechanism, and the weights are dynamically adjusted based on the reliability of each modality. Contextual information such as historical data, data of adjacent spatial locations, etc. are integrated into the anomaly detection process through the attention mechanism to improve the accuracy of detection. The anomaly detection results are used to adjust the credibility score. For each anomaly detected, the system calculates its severity and scope of impact, and reduces the credibility score accordingly. The reduction is calculated by the following formula:

[0125]

[0126] in, Represents the degree of decrease in credibility, Represents the severity of the abnormality, with a value range of 0 to 1; Represents the scope of influence, with a value range of 0~1; is an adjustable factor, usually set to 10.

[0127] Preferably, after quality assessment of the multimodal dataset in step 5, the quality assessment results are output, specifically including: the credibility score and level of each processing result in the multimodal dataset, a detailed quality assessment report including scores of each dimension, rule verification results and related explanatory text, a list of detected anomalies including anomaly type, location and severity, and visualization results, such as a credibility heat map, anomaly distribution map, etc.

[0128] In this embodiment, by collecting agricultural machinery data, preprocessing the collected agricultural scene data, and performing agricultural scene recognition, the set of scene analysis models that are closest to the recognized agricultural scene is matched and dynamically adjusted, and data enhancement and feature protection are performed on the agricultural scene data based on the scene analysis model. A multimodal data set of agricultural scenes is output, and the quality of the multimodal data set is evaluated. Finally, the quality evaluation results are output. This embodiment provides a complete and highly automated agricultural scene recognition and verification solution, which can effectively improve the accuracy and adaptability of agricultural scene analysis, and provide reliable data support for precision agriculture.

[0129] Embodiment 2:

[0130] As attached Figure 3 As shown, Embodiment 1 of the present invention provides an agricultural scene recognition and verification system 300 based on multimodal data, which is specifically as follows:

[0131] 301. Agricultural scene data preprocessing module: used to preprocess multi-dimensional agricultural scene data and output standardized data sets;

[0132] 302. Agricultural scene recognition module: used to perform agricultural scene recognition on the standardized data set to obtain an agricultural scene recognition result;

[0133] 303. Scene analysis model management library: used to manage and maintain different scene analysis models, dynamically select a set of matching scene analysis models and their parameters based on the agricultural scene recognition results, and adjust the set of scene analysis models in real time;

[0134] 304. A scene analysis model set module: configured to perform multimodal analysis on the standardized data set based on the scene analysis model set and its parameters to obtain a multimodal data set of the agricultural scene data;

[0135] 305. Quality assessment module: used to perform quality assessment on the multimodal dataset.

[0136] Preferably, the scenario analysis model management library 303 includes:

[0137] 3031. A scene analysis model selection unit: configured to dynamically select a set of matching scene analysis models and their parameters based on the agricultural scene recognition result;

[0138] 3032. Scene analysis model dynamic adjustment unit: used to monitor the operating status of the set of scene analysis models in real time, and adjust the scene recognition model in the set when the evaluation index of the scene recognition model in the set changes.

[0139] In this embodiment, by setting up an agricultural scene data preprocessing module, an agricultural scene recognition module, a scene analysis model management library, a scene analysis model collection module, and a quality assessment module, multimodal data collection, intelligent processing, and verification of agricultural scenes are achieved, effectively improving the accuracy of agricultural scene monitoring and analysis, and providing reliable data support for precision agriculture.

[0140] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present invention, which follows the general principles of the present invention and includes common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present invention are indicated by the claims. It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for agricultural scene recognition and verification based on multimodal data, characterized in that: include: Preprocess multi-dimensional agricultural scene data and output standardized data sets; Perform agricultural scene recognition based on the standardized data set to obtain an agricultural scene recognition result; The scene analysis model management library manages and maintains different scene analysis models, dynamically selects a set of matching scene analysis models and their parameters based on the agricultural scene recognition results, and adjusts the set of scene analysis models in real time; Based on the set of scene analysis models and their parameters, multimodal analysis is performed on the standardized data set to obtain a multimodal data set of the agricultural scene data; A quality assessment is performed on the multimodal dataset.

2. The agricultural scene recognition and verification method based on multimodal data according to claim 1 is characterized in that: The data preprocessing includes but is not limited to data cleaning, multi-source data calibration, time series synchronization and spatial registration, and the standardized data set is a multidimensional tensor, including but not limited to agricultural data that is fully aligned in time, space and spectral dimensions.

3. The agricultural scene recognition and verification method based on multimodal data according to claim 1 is characterized in that: The agricultural scene recognition is specifically performed as follows: Extracting image features and time series features from the standardized data set, and connecting the image features and time series features to form a first comprehensive feature vector; The agricultural scene recognition result is obtained by performing probability prediction on the first comprehensive feature vector through a soft voting integrated classifier.

4. The agricultural scene recognition and verification method based on multimodal data according to claim 3 is characterized in that: The soft voting integrated classifier includes a plurality of basic classifiers, and the first comprehensive feature vector is respectively input into different basic classifiers to obtain agricultural scene prediction results respectively; The agricultural scene prediction results are weighted averaged to obtain the agricultural scene recognition results: ,in represents the probability distribution of agricultural scene categories, represents the weight of the i-th base classifier, Represents the agricultural scene prediction result of the i-th basic classifier for the first comprehensive feature vector.

5. The agricultural scene recognition and verification method based on multimodal data according to claim 4 is characterized in that: The dynamically selected matching set of scene analysis models and their parameters are specifically: According to the agricultural scene category probability distribution, the scene matching score of each scene recognition model is calculated. The scene matching calculation formula is as follows: ,in It is The probability of scene categories, The scene recognition model is The coverage of scene categories; Calculating a feasibility score of each of the scene recognition models based on the computational complexity of each of the scene recognition models and currently available computing resources of the system; The weighted sum of the scene matching score and the feasibility score of each scene recognition model is calculated, and a set of the top A scene analysis models with the highest scores and their parameters are selected.

6. The agricultural scene recognition and verification method based on multimodal data according to claim 5 is characterized in that: The real-time adjustment of the set of scene analysis models is that when the evaluation index of the scene recognition model in the set changes while being continuously monitored, a model switching process will be triggered to adjust the scene recognition model in the set.

7. The agricultural scene recognition and verification method based on multimodal data according to claim 1 is characterized in that: The performing multimodal analysis on the standardized data set specifically includes: Performing feature recognition on the standardized data set to extract a second comprehensive feature vector of the agricultural scene data; Identifying agricultural features of the agricultural scene data based on the second comprehensive feature vector, and obtaining a multimodal dataset of the agricultural scene data based on the agricultural features; The data information in the multimodal dataset includes but is not limited to: identifiers of the data in the standardized dataset, a list of applied enhancement operations and their parameters, a feature importance map, a list of agricultural features and their confidence levels, and feature consistency check results.

8. The agricultural scene recognition and verification method based on multimodal data according to claim 1, characterized in that: The quality assessment of the multimodal data set includes but is not limited to: rule verification, multi-dimensional quality assessment, credibility scoring, and anomaly detection.

9. An agricultural scene recognition and verification system based on multimodal data, characterized in that: The system comprises: Agricultural scene data preprocessing module: used to preprocess multi-dimensional agricultural scene data and output standardized data sets; Agricultural scene recognition module: used to perform agricultural scene recognition on the standardized data set to obtain agricultural scene recognition results; Scenario analysis model management library: used to manage and maintain different scenario analysis models, dynamically select a set of matching scenario analysis models and their parameters based on the agricultural scenario recognition results, and adjust the set of scenario analysis models in real time; A scene analysis model collection module: used to perform multimodal analysis on the standardized data set based on the scene analysis model collection and its parameters to obtain a multimodal data set of the agricultural scene data; Quality assessment module: used to perform quality assessment on the multimodal dataset.

10. The agricultural scene recognition and verification system based on multimodal data according to claim 9, characterized in that: The scenario analysis model management library includes: A scene analysis model selection unit: used to dynamically select a set of matching scene analysis models and their parameters based on the agricultural scene recognition result; A scene analysis model dynamic adjustment unit is used to monitor the operating status of the set of scene analysis models in real time, and adjust the scene recognition models in the set when the evaluation indicators of the scene recognition models in the set change.

Images