Multimodal deep neural network model, system and method based on continuous learning

Abstract

The invention discloses a multi-modal deep neural network model, system and method based on continuous learning. The multi-modal deep neural network model comprises a data acquisition and preprocessing module used for acquiring multi-modal data of a crop growth environment; the feature extraction module is used for extracting key agricultural features; the multi-modal information fusion module is used for effectively fusing the extracted key agricultural features; the knowledge continuous learning module is used for memorizing and storing the crop growth mode to a crop growth mode library and applying a model parameter self-adaptive updating strategy; the intelligent decision-making module is used for performing crop management decision-making based on the fusion features; and the effect evaluation and feedback module is used for evaluating the decision effect. According to the invention, the problem of knowledge forgetting of the existing AI system is solved, and the adaptability and decision accuracy of the intelligent agricultural system are improved.

Description

Technical Field

[0001] This invention belongs to the field of smart agriculture and artificial intelligence technology, specifically relating to a multimodal deep neural network model based on continuous learning. Background Technology

[0002] As modern agriculture transforms into smart agriculture, deep learning-based agricultural AI systems have made significant progress in applications such as crop monitoring, pest and disease identification, and yield prediction. However, traditional agricultural AI models are prone to "knowledge forgetting" when faced with new crop varieties, new types of pests and diseases, or new planting environments—that is, they quickly lose their memory of previously learned crop management knowledge when learning new agricultural tasks. This limits the practicality of agricultural AI systems in scenarios such as long-term agricultural production, multi-seasonal crop rotation, and cross-regional applications.

[0003] Agricultural production has the following characteristics: Multimodal data includes crop images, soil sensor data, meteorological data, yield data, etc. Seasonal variation: Crop growth patterns differ significantly across seasons; Regional differences: Soil and climate conditions in different regions affect crop growth; Continuous evolution: New varieties, new diseases and pests, and new cultivation techniques are constantly emerging; Current agricultural AI technologies commonly employ solutions such as machine learning models based on historical data, rule-based expert systems, and single-task deep learning models. However, most of these are only applicable to specific crops or tasks, making it difficult to simultaneously address the needs of multimodal agricultural information fusion and continuous learning. Therefore, there is an urgent need for an agricultural AI system capable of fusing multiple agricultural modalities and updating online. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal deep neural network model, system, and method based on continuous learning, which aims to solve the problem that existing agricultural AI systems are limited to single crops or single tasks and lack the ability to cope with multimodal agricultural integration scenarios.

[0005] To achieve the above objectives, one of the technical solutions of this invention is as follows: A multimodal deep neural network model based on continuous learning includes, in sequence, a data acquisition and preprocessing module, a feature extraction module, a multimodal information fusion module, a knowledge continuous learning module, an intelligent decision-making module, and an effect evaluation and feedback module, wherein the effect evaluation and feedback module is connected to the knowledge continuous learning module. The data acquisition and preprocessing module is used to collect multimodal raw data of crop growth environment, preprocess it, and store it in the crop growth pattern library. The feature extraction module is used to extract key agricultural features from crop vision, soil sensor, and meteorological modes from the data acquisition and preprocessing module. The multimodal information fusion module is used to fuse the key agricultural features through a cross-modal attention mechanism, output agricultural decision fusion features, and construct a self-attention model. The knowledge continuous learning module is used to update the fusion features of the crop growth pattern library according to the agricultural decision fusion features, implement parameter protection / expansion dynamic updates based on parameter importance, and achieve agricultural knowledge alignment through positive and negative alignment training. The intelligent decision-making module is used to execute different tasks and call corresponding deep neural network decision-making reasoning based on the learning results of the knowledge continuous learning module to carry out agricultural management. The effect evaluation and feedback module is used to calculate various agricultural effect indicators based on the prediction results, evaluate the performance of the self-attention model on new and old agricultural tasks, and feed the evaluation results back to the knowledge continuous learning module.

[0006] Furthermore, it also includes a crop growth pattern library update module, which is connected to the effect evaluation and feedback module. This module is used to dynamically add or delete crop growth patterns based on the feedback results from the effect evaluation and feedback module. Specifically, it includes: It includes a crop pattern selection unit, a knowledge base capacity management unit, and a knowledge storage unit connected in sequence. The crop model selection unit screens representative crop growth patterns and management experiences, and selects key prototypes based on category distribution, cluster scores, or representative indicators. The knowledge base capacity management unit is used to manage the capacity of the crop growth pattern database, eliminate outdated information, monitor the memory database capacity, and maintain a preset threshold range through elimination and replacement strategies. The knowledge storage unit is used to store and retrieve crop growth patterns, management experience, common diseases and pests and their control methods, perform prototype serialization and compressed storage, and support retrieval and playback.

[0007] Furthermore, the data acquisition and preprocessing module includes a sensor interface unit, a timing synchronization unit, a data standardization unit, and a data enhancement unit connected in sequence. The sensor interface unit is used to communicate with agricultural hardware devices such as soil sensors, weather stations, and crop monitoring cameras, and to collect raw multimodal agricultural data of soil, weather, and crop images. The time synchronization unit is used to synchronize data on crop growth cycle, weather changes, and soil conditions at different time scales to ensure the time consistency of cross-modal agricultural data. The data standardization unit is used to achieve crop image standardization, soil parameter normalization, and meteorological data processing; The data augmentation unit is used to perform crop image rotation, illumination adjustment, soil data noise processing, meteorological data interpolation, time series expansion, and agricultural text knowledge synonym replacement.

[0008] Furthermore, the feature extraction module includes a crop visual feature extraction unit, a text information encoding unit, an environmental audio analysis unit, and a multi-source data preprocessing unit. The crop visual feature extraction unit, the text information encoding unit, and the environmental audio analysis unit are respectively connected to the multi-source data preprocessing unit. The crop visual feature extraction unit uses an agricultural-optimized Vision Transformer model or a Transformer ResNet encoder to extract visual feature vectors of crop leaves, fruits, and pests from the multimodal raw data and outputs a fixed-dimensional representation. The text information encoding unit is based on the Vision Transformer model to process agricultural expert knowledge, planting manuals, and descriptions of pests and diseases to generate embedded text information. The environmental audio analysis unit uses a one-dimensional convolutional neural network (1D-CNN) or a temporal convolutional network to analyze the multimodal raw data to obtain environmental audio features of pest sounds and agricultural machinery operation sounds. The multi-source data preprocessing unit is used to reduce or project multi-source data such as soil, meteorology, crop images, and audio to improve the efficiency of subsequent fusion.

[0009] Furthermore, the multimodal information fusion module includes a single-modal correlation analysis unit, a cross-modal correlation analysis unit, an information fusion projection unit, and a decision feature integration unit connected in sequence. The single-modal correlation analysis unit is used to perform self-attention modeling based on the key agricultural features, analyzing the internal correlation of soil parameters and the internal context of the correlation of meteorological elements within the single-modal features. The cross-modal association analysis unit is used to analyze cross-modal associations of soil-crop, weather-yield, and pests-environment based on the key agricultural features, and to use crop vision, soil sensor, and weather features as query, key, and value for cross-modal interaction. The information fusion projection unit is used to map multimodal agricultural information after cross-modal correlation analysis to a unified decision space, perform linear transformation on the attention output, and ensure numerical stability through layer normalization. The decision feature integration unit is used to integrate various types of agricultural information obtained by the information fusion projection unit to form decision features, and to perform weighted accumulation or splicing of multi-level attention results.

[0010] Furthermore, the knowledge continuous learning module includes a crop growth pattern memory unit, a knowledge protection and updating unit, and a knowledge consistency maintenance unit connected in sequence. The crop growth pattern memory unit is used to store typical crop growth patterns, pest and disease characteristics, and best management practices. After the agricultural task is completed, the representative prototype of the integrated features is stored in the crop growth pattern library. The knowledge protection and updating unit is used to protect learned agricultural knowledge according to the importance of parameters, and to expand the network capacity when the demand for new crops and new pests and diseases changes significantly. The knowledge consistency maintenance unit is used to ensure the consistency between newly learned knowledge and existing agricultural knowledge, and performs positive and negative alignment training based on new and old agricultural tasks and multimodal agricultural samples.

[0011] Furthermore, the intelligent decision-making module includes a task routing unit, a decision generation unit, and a multi-task decision-making unit connected in sequence. The task routing unit is used to call the corresponding decision branch according to different agricultural tasks, and to dynamically call different incremental network branches according to task labels or input features. The decision generation unit generates agricultural management decisions such as fertilization suggestions, irrigation plans, and pest and disease control schemes based on fusion features. The multi-task decision unit is used to simultaneously process multiple agricultural tasks such as crop monitoring, pest and disease diagnosis, and yield prediction, and maintains an independent output stream for each task so that they do not interfere with each other.

[0012] Furthermore, the effect evaluation and feedback module includes an effect indicator calculation unit, a decision threshold judgment unit, and an effect feedback unit connected in sequence. The performance index calculation unit is used to calculate agricultural performance indicators such as crop health, yield prediction accuracy, and pest and disease identification accuracy. The decision threshold judgment unit is used to determine whether the management strategy needs to be adjusted or the model needs to be updated based on whether the effect indicator is optimal. It compares the current indicator with the baseline or the historical highest value to determine whether to trigger continuous learning and updating. The effect feedback unit is used to generate feedback signals based on actual agricultural effects, transform the evaluation results into reward or punishment signals, and feed them back to the knowledge continuous learning module.

[0013] The second objective of this invention is to adopt the following technical solution: A system based on the aforementioned multimodal deep neural network model based on continuous learning, wherein the crop visual feature extraction unit in the feature extraction module adopts a Vision Transformer structure with N layers.

[0014] The third objective of this invention is to adopt the following technical solution: A learning method for a multimodal deep neural network model based on continuous learning includes the following steps: S1: Collect multimodal raw data on crop growth environment and perform data preprocessing; S2: Extract key agricultural features from crop vision, soil sensor, and meteorological multimodal data from S1; S3: Utilize a cross-modal attention mechanism to perform cross-modal fusion of the key agricultural features to obtain agricultural decision fusion features; S4: Update the fusion features of the crop growth pattern library based on the agricultural decision fusion features, implement parameter protection / expansion dynamic updates based on parameter importance, and achieve agricultural knowledge alignment through positive and negative alignment training; S5: Based on the fusion features, different tasks are executed by calling the corresponding deep neural network for decision-making and reasoning, and agricultural management prediction results are obtained; S6; Calculate various agricultural performance indicators based on the prediction results, evaluate the performance of the self-attention model on new and old agricultural tasks, and return the evaluation results to continuous learning.

[0015] The beneficial effects of this invention are: This invention employs a data acquisition and preprocessing module to perform time-series synchronization, standardization, and optional data augmentation on soil, meteorological, and crop image data from soil sensors, weather stations, crop monitoring cameras, and agricultural textual knowledge. A feature extraction module extracts deep feature vectors from crop images, soil sensor data, and meteorological data respectively. A multimodal information fusion module utilizes self-attention and cross-modal attention mechanisms to map the features of each agricultural modality to a unified decision space and perform fusion projection, generating agricultural decision fusion features. A knowledge continuous learning module saves representative agricultural prototypes of the fusion features at the end of each agricultural task, controls the stability of parameters from old agricultural tasks through parameter protection and dynamic expansion strategies, and adds network capacity for new agricultural tasks. Simultaneously, it incorporates alignment loss to maintain the consistency between new and old agricultural tasks and cross-modal information. The semantics of the modal agricultural samples are consistent; the intelligent decision-making module performs agricultural management task reasoning based on fused features, dynamically calls different decision branches through a task routing mechanism, and generates precision agricultural management decisions including fertilization suggestions, irrigation plans, and pest and disease control schemes; the effect evaluation and feedback module monitors the model's performance on new and old agricultural tasks in real time, calculates key agricultural effect indicators such as crop health, yield prediction accuracy, and pest and disease identification accuracy, and converts the evaluation results into feedback signals to be transmitted to the knowledge continuous learning module to form a closed-loop optimization; the crop growth pattern library update module intelligently selects representative crop growth patterns and management experiences based on the feedback results, and dynamically maintains the size of the knowledge base through a capacity management strategy to ensure that the system maintains efficient knowledge storage and retrieval capabilities during long-term operation.

[0016] This invention is a multimodal deep neural network model that integrates crop growth pattern memory with parameter protection / expansion dynamic updates, enabling incremental learning of new agricultural tasks while preventing the forgetting of old agricultural knowledge.

[0017] This invention is a deep neural network model system that supports multimodal agricultural data input and has continuous learning capabilities, particularly for intelligent systems applied to crop growth monitoring, pest and disease identification, yield prediction, and precision agricultural management. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a block diagram of a specific embodiment 1 of the present invention.

[0020] Figure 2 This is a schematic diagram of the data acquisition and preprocessing module in Specific Embodiment 1.

[0021] Figure 3 This is a schematic diagram of the feature extraction module in specific embodiment 1.

[0022] Figure 4 This is a schematic diagram of the multimodal information fusion module in specific embodiment 1.

[0023] Figure 5 This is a schematic diagram of the knowledge continuous learning module in Specific Implementation 1.

[0024] Figure 6 This is a schematic diagram of the intelligent decision-making module in specific embodiment 1.

[0025] Figure 7 This is a schematic diagram of the effect evaluation and feedback module in specific embodiment 1.

[0026] Figure 8 This is a schematic diagram of the crop growth pattern library update module in specific embodiment 1; Figure 9 This is a flowchart of specific embodiment 3.

[0027] 1-Data Acquisition and Preprocessing Module, 2-Feature Extraction Module, 3-Multimodal Information Fusion Module, 4-Continuous Knowledge Learning Module, 5-Intelligent Decision-Making Module, 6-Effect Evaluation and Feedback Module, 7-Crop Growth Pattern Library Update Module, 11-Sensor Interface Unit, 12-Time Sequence Synchronization Unit, 13-Data Standardization Unit, 14-Data Augmentation Unit, 21-Crop Visual Feature Extraction Unit, 22-Text Information Encoding Unit, 23-Environmental Audio Analysis Unit, 24-Multi-Source Data Preprocessing Unit, 31-Single-Modal Association Analysis unit, 32-Cross-modal association analysis unit, 33-Information fusion projection unit, 34-Decision feature integration unit, 41-Crop growth pattern memory unit, 42-Knowledge protection and updating unit, 43-Knowledge consistency maintenance unit, 51-Task routing unit, 52-Decision generation unit, 53-Multi-task decision unit, 61-Effect index calculation unit, 62-Decision threshold judgment unit, 63-Effect feedback unit, 71-Crop pattern selection unit, 72-Knowledge base capacity management unit, 73-Knowledge storage unit. Detailed Implementation

[0028] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention. Specific Implementation Example 1

[0029] like Figures 1 to 8 As shown, the present invention provides a multimodal deep neural network model based on continuous learning, including a data acquisition and preprocessing module 1, a feature extraction module 2, a multimodal information fusion module 3, a knowledge continuous learning module 4, an intelligent decision-making module 5, an effect evaluation and feedback module 6, and a crop growth pattern library update module 7 connected in sequence. The effect evaluation and feedback module 6 is connected to the knowledge continuous learning module 4.

[0030] The data acquisition and preprocessing module 1 is used to collect multimodal raw data of crop growth environment and perform time alignment and standardization preprocessing; specifically including: It includes a sensor interface unit 11, a timing synchronization unit 12, a data standardization unit 13, and a data enhancement unit 14, which are connected in sequence.

[0031] The sensor interface unit 11 is used to communicate with agricultural hardware devices such as soil pH sensor, soil moisture sensor, weather station, crop monitoring camera, drone aerial photography equipment, and environmental audio, and to collect raw multimodal agricultural data including soil composition data, meteorological environment data, and crop growth image data. The time synchronization unit 12 is used to synchronize crop growth cycle data, such as sowing period, growth period, and maturity period; meteorological change data, such as daily temperature difference, rainfall, and light intensity; and soil condition data, such as changes in nutrient content and water status. It provides a unified agricultural production timestamp for each modal agricultural data, ensuring the consistency of cross-modal agricultural data in crop growth time.

[0032] The data standardization unit 13 performs resolution adjustment to 1024×1024 pixels, illumination condition standardization, color space conversion, and soil parameter normalization, such as unifying the sensor ranges of pH value, nitrogen, phosphorus and potassium content, humidity, etc., and performing 0-1 range numerical standardization and meteorological data processing, such as standardizing temperature and humidity to degrees Celsius and percentages, normalizing rainfall, and standardizing wind speed and light intensity.

[0033] The data augmentation unit 14 is used to perform crop image data augmentation, including image rotation, illumination adjustment, noise injection to simulate different illumination and weather conditions, soil data noise processing, adding sensor error noise to improve model robustness, meteorological data interpolation and temporal extension, interpolation completion of missing meteorological records, and agricultural text knowledge augmentation, including synonym replacement and sentence reconstruction of texts such as descriptions of pests and diseases and cultivation manuals.

[0034] The feature extraction module 2 is used to extract key agricultural features from crop vision, soil sensor, and meteorological modalities, respectively; specifically including: The feature extraction module 2 includes a crop visual feature extraction unit 21, a text information encoding unit 22, an environmental audio analysis unit 23, and a multi-source data preprocessing unit 24. The crop visual feature extraction unit, the text information encoding unit, and the environmental audio analysis unit are respectively connected to the multi-source data preprocessing unit. The crop visual feature extraction unit 21 adopts a Vision Transformer or ResNet variant network structure optimized for agricultural scenarios to specifically extract visual feature vectors such as crop leaf health status, fruit maturity, disease and pest symptoms, and weed distribution, and outputs a 768-dimensional fixed-dimensional crop visual feature representation. The text information encoding unit 22, based on a Transformer model pre-trained in the agricultural field, such as AgBERT, processes text information such as agricultural expert knowledge bases, crop planting manuals, pest and disease control guidelines, and agricultural meteorological reports to generate 512-dimensional agricultural text semantic embeddings. The environmental audio analysis unit 23 uses a one-dimensional convolutional neural network 1D-CNN or a temporal convolutional network to analyze the audio features in the agricultural environment, including the sounds of pest activities, such as locusts and aphids, the sounds of agricultural machinery operations, such as tractors and harvesters, and natural environmental sounds, such as wind and rain, and extracts a 256-dimensional environmental audio feature vector. The multi-source data preprocessing unit 24 is used to unify the feature dimensions and perform projection transformation on multi-source heterogeneous agricultural data such as soil sensor data, meteorological station data, crop growth images, and audio, mapping the features of different modalities to the same 512-dimensional feature space, thereby improving the computational efficiency and fusion quality of subsequent cross-modal fusion.

[0035] The multimodal information fusion module 3 fuses the features of each agricultural modality through a cross-modal attention mechanism, and outputs agricultural decision fusion features; specifically including: The multimodal information fusion module 3 includes a single-modal correlation analysis unit 31, a cross-modal correlation analysis unit 32, an information fusion projection unit 33, and a decision feature integration unit 34, which are connected in sequence. The single-modal correlation analysis unit 31 is used to analyze the internal correlation of soil parameters, such as the correlation between pH value and nitrogen, phosphorus and potassium content, the correlation of meteorological elements, such as the relationship between temperature and humidity, light and rainfall, the internal relationship of crop visual features, such as the correlation between leaf color and disease severity, and other single-modal feature internal contexts. It models complex correlation patterns within a single agricultural modality through a self-attention mechanism. The cross-modal association analysis unit 32 is used to deeply analyze cross-modal agricultural associations such as the impact of soil conditions on crop growth, the mechanism of meteorological changes on yield, and the relationship between pests and diseases and environmental factors. It uses crop visual features, soil sensor features, and meteorological environmental features as queries, keys, and values ​​respectively to perform cross-modal attention interaction and explore cross-modal causal relationships in agricultural production. The information fusion projection unit 33 is used to map multimodal agricultural information after cross-modal correlation analysis to a unified agricultural decision space, perform linear transformation and feature integration on cross-modal attention output, and ensure numerical stability in the process of key agricultural feature fusion through layer normalization technology, thereby ensuring the effective fusion of agricultural data at different scales. The decision feature integration unit 34 is used to integrate various types of agricultural information obtained by the information fusion projection unit to form the final agricultural decision features, and to perform weighted accumulation or feature splicing on the multi-layer attention results to generate a 1536-dimensional agricultural decision fusion feature vector containing comprehensive information such as crop growth status, environmental adaptability, and management needs.

[0036] The knowledge continuous learning module 4 is used to update the fusion features of the crop growth pattern library based on the agricultural decision fusion features. It implements parameter protection / expansion dynamic updates based on parameter importance and achieves agricultural knowledge alignment through positive and negative alignment training. The parameter protection / expansion dynamic updates include freezing important agricultural knowledge parameters and automatically expanding the corresponding incremental network blocks for dynamic updates when the parameter importance is below a threshold. It also includes extracting old agricultural task samples from the crop growth pattern library according to a predefined strategy and mixing them with new agricultural task samples for training.

[0037] Specifically, it includes: The knowledge continuous learning module 4 includes a crop growth pattern memory unit 41, a knowledge protection and updating unit 42, and a knowledge consistency maintenance unit 43, which are connected in sequence. The crop growth pattern memory unit 41 is used to store core agricultural knowledge such as typical crop growth patterns, characteristics of common pests and diseases, and best management practices. After each agricultural task, representative prototypes of agricultural decision-making integration features are stored in the crop growth pattern library, including healthy crop patterns, diseased crop patterns, and patterns at different growth stages, as prototype memories. An agricultural knowledge graph covering multiple crops, multiple growth stages, and multiple environmental conditions is established.

[0038] The knowledge protection and updating unit 42 protects the important agricultural knowledge parameters that have been learned from being destroyed by the new task learning process by maintaining the parameter importance evaluation results calculated by the Elastic Weight Consolidation (EWC) algorithm based on the elastic weight. At the same time, when encountering significant changes in demand such as new crop varieties, new types of pests and diseases, or new cultivation techniques, it increases the dedicated network capacity for new agricultural tasks through a dynamic network expansion strategy to ensure that the learning of new knowledge does not affect the maintenance of old knowledge. The knowledge consistency maintenance unit 43 ensures that the newly learned agricultural knowledge maintains semantic consistency and logical coherence with the knowledge stored in the existing agricultural knowledge base. Based on the historical agricultural task samples in the crop growth pattern library and the multimodal agricultural samples of the current new agricultural tasks, it performs positive and negative sample alignment training and maintains the knowledge correlation between different agricultural tasks through a contrastive learning mechanism.

[0039] The intelligent decision-making module 5 executes different tasks and calls corresponding deep neural network decision reasoning based on the learning results of the knowledge continuous learning module to carry out agricultural management; specifically, the intelligent decision-making module 5 includes a task routing unit 51, a decision generation unit 52, and a multi-task decision-making unit 53, which are connected in sequence. The task routing unit 51 is used to invoke corresponding specialized decision branches according to different types of agricultural tasks, such as crop health monitoring, pest and disease diagnosis, yield prediction, and precision fertilization. Based on the feature patterns of the input agricultural task labels or agricultural decision fusion features, the corresponding incremental network branches are dynamically activated to achieve specialized task processing. The decision generation unit 52 generates specific agricultural management decisions based on agricultural decision fusion features through a fully connected neural network and decision tree integration method. These decisions include fertilization recommendations, such as nitrogen, phosphorus and potassium ratios and fertilization amounts; irrigation plans, such as irrigation time and water volume control; pest and disease control programs, such as pesticide selection and spraying programs; and practical agricultural management guidance, such as harvest timing prediction. The multi-task decision unit 53 is used to simultaneously process multiple related agricultural tasks such as crop growth monitoring, early diagnosis of pests and diseases, accurate yield prediction, and soil fertility assessment. It maintains independent output branches and loss functions for each agricultural task to ensure that the decisions of different agricultural tasks are independent and do not interfere with each other.

[0040] The effect evaluation and feedback module 6 is used to calculate various agricultural effect indicators based on the prediction results, evaluate the performance of the self-attention model on both new and old agricultural tasks, and feed the evaluation results back to the knowledge continuous learning module 4; specifically, it includes: The effect evaluation and feedback module 6 includes an effect index calculation unit 61, a decision threshold judgment unit 62, and an effect feedback unit 63, which are connected in sequence. The performance index calculation unit 61 is used to calculate multiple agricultural system performance indicators in real time, including crop health identification accuracy, yield prediction accuracy (MAPE) index, pest and disease identification accuracy and recall rate, actual yield increase effect of fertilization recommendations, and water resource utilization efficiency of irrigation plans, etc. The decision threshold judgment unit 62 is used to determine whether the current agricultural AI system needs to adjust the management strategy or update the model parameters. It compares the current agricultural performance indicators with historical baseline values ​​or industry standard benchmarks. When the performance indicators are lower than the preset threshold, the parameter update mechanism of the knowledge continuous learning module is triggered, such as when the accuracy rate is lower than 90% or the prediction error is greater than 15%. The effect feedback unit 63 is used to generate quantitative feedback signals based on the actual effects of agricultural production and farmer feedback, transforming the agricultural effect evaluation results into reward signals indicating good results or penalty signals indicating poor results, and transmitting these feedback signals to the knowledge continuous learning module to guide the continuous optimization and knowledge updating of the model.

[0041] The crop growth pattern library update module 7 dynamically adds or deletes crop growth patterns based on feedback results.

[0042] Specifically, the crop growth pattern library update module 7 includes a crop pattern selection unit 71, a knowledge base capacity management unit 72, and a knowledge storage unit 73, which are connected in sequence. The crop pattern selection unit 71 is responsible for selecting the most representative crop growth patterns and management experiences from a large amount of agricultural data. Based on multiple dimensions such as the balanced distribution of crop categories, the representativeness score of cluster centers, and the importance indicators marked by agricultural experts, it uses K-means clustering and representative sampling algorithms to select key crop growth prototypes, ensuring that the pattern library covers typical cases of different crops, different growth stages, and different environmental conditions. The knowledge base capacity management unit 72 is used to intelligently manage the storage capacity and knowledge quality of the crop growth pattern library, monitor the storage usage of the crop growth pattern library in real time, and automatically eliminate outdated or duplicate agricultural knowledge entries through strategies such as importance assessment, time decay, and access frequency when the storage capacity is close to the upper limit, while retaining the core knowledge that is most valuable to agricultural decision-making, and maintaining the knowledge base within the preset optimal capacity range. The knowledge storage unit 73 is used to store and retrieve crop growth patterns, management experience, common diseases and pests and their control methods, perform prototype serialization and compressed storage, and support efficient retrieval and playback. It adopts a hierarchical storage architecture and compression coding technology to perform serialization storage of agricultural prototypes, establishes an indexing mechanism based on semantic similarity, and supports efficient retrieval and intelligent playback of agricultural knowledge based on multiple dimensions such as crop type, growth stage, and environmental conditions.

[0043] Application scenarios of this invention: The present invention discloses a multimodal deep neural network model based on continuous learning. In practical use, the data acquisition and preprocessing module 1 collects multimodal raw agricultural data, including soil pH, nitrogen, phosphorus and potassium content, temperature and humidity, light intensity, rainfall, and crop growth images, from a soil sensor network deployed in farmland, meteorological monitoring stations, UAV aerial photography systems, and ground crop monitoring cameras. The time synchronization unit 12 ensures the consistency of data from different sensors over the crop growth timeline. The data standardization unit 13 performs 1024×1024 resolution standardization on the images, 0-1 normalization on the soil parameters, and standard unit conversion on the meteorological data. Optionally, the data enhancement unit 14 performs data enhancement operations such as image rotation, illumination simulation, and sensor noise injection.

[0044] Subsequently, the feature extraction module 2 uses the crop visual feature extraction unit 21 to extract 768-dimensional crop visual features based on the agricultural-optimized VisionTransformer, including information such as leaf health, fruit maturity, and pest and disease symptoms. The text information encoding unit 22 generates 512-dimensional agricultural text embeddings based on AgBERT (covering professional knowledge such as cultivation manuals and pest and disease guidelines). The environmental audio analysis unit 23 extracts 256-dimensional environmental audio features, such as sound information like pest activity and agricultural machinery operation. The multi-source data preprocessing unit 24 then maps all features to a unified 512-dimensional feature space.

[0045] The multimodal information fusion module 3 first models the internal correlation of soil parameters and the correlation of meteorological elements through the single-modal correlation analysis unit 31, and then deeply analyzes cross-modal agricultural correlations such as soil-crop, meteorology-yield, and pests-environment through the cross-modal correlation analysis unit 32. Finally, the cross-modal attention output is linearly transformed, layer-normalized and weighted through the information fusion projection unit 33 and the decision feature integration unit 34 to generate 1536-dimensional agricultural decision fusion features.

[0046] Within the knowledge continuous learning module 4, the system first uses the crop growth pattern memory unit 41 to calculate and store representative prototypes of agricultural decision fusion features after each agricultural task, such as healthy wheat patterns, corn disease patterns, and different growth stages of rice, to establish an agricultural knowledge base covering multiple crops. Simultaneously, the knowledge protection and updating unit 42 identifies and protects network parameters crucial to the learned agricultural tasks based on the EWC algorithm, and dynamically expands the corresponding network branches when new crop varieties or new pest and disease types are detected. Next, the knowledge consistency maintenance unit 43 uses historical agricultural samples from the crop growth pattern library and current new agricultural task samples to perform comparative learning and positive / negative alignment training, maintaining semantic consistency of knowledge between different agricultural tasks. All triggering conditions and weight adjustments for knowledge protection, network expansion, and alignment training are determined by real-time agricultural performance indicators provided by the effect evaluation and feedback module 6, such as crop health identification accuracy, yield prediction accuracy, pest and disease diagnosis accuracy, and agricultural system resource utilization efficiency. This ensures that the agricultural knowledge continuous learning strategy can flexibly respond to actual agricultural production effects and continuously optimize decision-making capabilities while maintaining the stability of learned agricultural knowledge.

[0047] The intelligent decision-making module 5, through the task routing unit 51, schedules corresponding specialized network branches according to the type of agricultural task, such as crop monitoring, disease diagnosis, yield prediction, fertilization planning, etc. The decision generation unit 52 generates specific agricultural management suggestions based on fusion features, including fertilizer formula, irrigation plan, pesticide spraying scheme, harvesting timing, etc. The multi-task decision-making unit 53 simultaneously outputs the decision results of multiple agricultural tasks.

[0048] In the effect evaluation and feedback module 6, the effect index calculation unit 61 calculates key agricultural indicators in real time, such as crop health, yield prediction accuracy, and pest and disease identification accuracy; the decision threshold judgment unit 62 compares the current agricultural performance with historical baselines and industry standards; and the effect feedback unit 63 transforms the actual agricultural production effect into quantitative feedback signals to guide the parameter updates of the knowledge continuous learning module 4.

[0049] Finally, the crop growth pattern library update module 7 uses the crop pattern selection unit 71 to select key agricultural prototypes based on crop category balance and representativeness scores, the knowledge base capacity management unit 72 manages the knowledge base capacity based on importance and timeliness, and the knowledge storage unit 73 performs efficient storage and intelligent retrieval of agricultural knowledge.

[0050] Through the collaborative work of the above modules in the field of smart agriculture, this system can continuously learn new crop management tasks under the limited computing power and storage resources of the agricultural production environment, efficiently accumulate agricultural professional knowledge, and effectively prevent the forgetting of agricultural knowledge already acquired while learning new crop varieties, new pest and disease control, and new cultivation techniques, thus providing continuously optimized intelligent decision support for modern precision agriculture. Specific Implementation Example 2

[0051] This specific embodiment is a system based on the continuous learning multimodal deep neural network model of specific embodiment 1. The crop visual feature extraction unit in the feature extraction module adopts a Vision Transformer structure with N layers. Specific Implementation Example 3

[0052] See Figure 9 As shown, a learning method for a multimodal deep neural network model based on continuous learning includes the following steps: S1: Collect multimodal raw data on crop growth environment and perform data preprocessing; S2: Extract key agricultural features from crop vision, soil sensor, and meteorological multimodal data from S1, and possibly also extract audio information; S3: Utilize a cross-modal attention mechanism to perform cross-modal fusion of the key agricultural features to obtain agricultural decision fusion features; S4: Update the fusion features of the crop growth pattern library based on the agricultural decision fusion features, implement parameter protection / expansion dynamic updates based on parameter importance, and achieve agricultural knowledge alignment through positive and negative alignment training; specifically, this includes: performing incremental training on new agricultural tasks to update the fusion features of the crop growth pattern library, implementing dynamic updates based on parameter importance, and achieving agricultural knowledge alignment through positive and negative training; the parameter protection / expansion dynamic updates include freezing important agricultural knowledge parameters, and automatically expanding the corresponding incremental network block for dynamic updates when the parameter importance is below a threshold, and also includes the step of extracting old agricultural task samples from the crop growth pattern library according to a predefined strategy and mixing them with new agricultural task samples for training; S5: Based on the fusion features, different tasks are executed by calling the corresponding deep neural network for decision-making and reasoning, and agricultural management prediction results are obtained; S6; Calculate various agricultural performance indicators based on the prediction results, and evaluate the performance of the self-attention model on new and old agricultural tasks. Specifically, this includes evaluating the current indicators based on various agricultural performance indicators and returning the evaluation results to continuous learning.

[0053] It may also include S7: updating the crop growth pattern library and adjusting the weights of the self-attention model based on the evaluation results; S8: Repeat steps S2 to S7 until all agricultural tasks are completed.

[0054] This specific embodiment is based on the multimodal deep neural network model based on continuous learning described in Specific Embodiment 1. The specific steps and methods are the same as the functions of each module in Specific Embodiment 1, so this specific embodiment is omitted here.

[0055] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A multimodal deep neural network model based on continuous learning, characterized in that, It includes, in sequence, a data acquisition and preprocessing module, a feature extraction module, a multimodal information fusion module, a knowledge continuous learning module, an intelligent decision-making module, and an effect evaluation and feedback module, wherein the effect evaluation and feedback module is connected to the knowledge continuous learning module. The data acquisition and preprocessing module is used to collect multimodal raw data of crop growth environment, preprocess it, and store it in the crop growth pattern library. The feature extraction module is used to extract key agricultural features from crop vision, soil sensor, and meteorological modes from the data acquisition and preprocessing module. The multimodal information fusion module is used to fuse the key agricultural features through a cross-modal attention mechanism, output agricultural decision fusion features, and construct a self-attention model. The knowledge continuous learning module is used to update the fusion features of the crop growth pattern library according to the agricultural decision fusion features, implement parameter protection / expansion dynamic updates based on parameter importance, and achieve agricultural knowledge alignment through positive and negative alignment training. The intelligent decision-making module is used to execute different tasks and call corresponding deep neural network decision-making reasoning based on the learning results of the knowledge continuous learning module to carry out agricultural management. The effect evaluation and feedback module is used to calculate various agricultural effect indicators based on the prediction results, evaluate the performance of the self-attention model on new and old agricultural tasks, and feed the evaluation results back to the knowledge continuous learning module.

2. The multimodal deep neural network model based on continuous learning according to claim 1, characterized in that: It also includes a crop growth pattern library update module, which is connected to the effect evaluation and feedback module. This module is used to dynamically add or delete crop growth patterns based on the feedback results from the effect evaluation and feedback module. Specifically, it includes: It includes a crop pattern selection unit, a knowledge base capacity management unit, and a knowledge storage unit connected in sequence. The crop model selection unit screens representative crop growth patterns and management experiences, and selects key prototypes based on category distribution, cluster scores, or representative indicators. The knowledge base capacity management unit is used to manage the capacity of the crop growth pattern database, eliminate outdated information, monitor the memory database capacity, and maintain a preset threshold range through elimination and replacement strategies. The knowledge storage unit is used to store and retrieve crop growth patterns, management experience, common diseases and pests and their control methods, perform prototype serialization and compressed storage, and support retrieval and playback.

3. The multimodal deep neural network model based on continuous learning according to claim 1, characterized in that: The data acquisition and preprocessing module includes a sensor interface unit, a timing synchronization unit, a data standardization unit, and a data enhancement unit connected in sequence. The sensor interface unit is used to communicate with agricultural hardware devices such as soil sensors, weather stations, and crop monitoring cameras, and to collect raw multimodal agricultural data of soil, weather, and crop images. The time synchronization unit is used to synchronize data on crop growth cycle, weather changes, and soil conditions at different time scales to ensure the time consistency of cross-modal agricultural data. The data standardization unit is used to achieve crop image standardization, soil parameter normalization, and meteorological data processing; The data augmentation unit is used to perform crop image rotation, illumination adjustment, soil data noise processing, meteorological data interpolation, time series expansion, and agricultural text knowledge synonym replacement.

4. The multimodal deep neural network model based on continuous learning according to claim 3, characterized in that: The feature extraction module includes a crop visual feature extraction unit, a text information encoding unit, an environmental audio analysis unit, and a multi-source data preprocessing unit. The crop visual feature extraction unit, the text information encoding unit, and the environmental audio analysis unit are respectively connected to the multi-source data preprocessing unit. The crop visual feature extraction unit uses an agricultural-optimized Vision Transformer model or a Transformer ResNet encoder to extract visual feature vectors of crop leaves, fruits, and pests from the multimodal raw data and outputs a fixed-dimensional representation. The text information encoding unit is based on the Vision Transformer model to process agricultural expert knowledge, planting manuals, and descriptions of pests and diseases to generate embedded text information. The environmental audio analysis unit uses a one-dimensional convolutional neural network (1D-CNN) or a temporal convolutional network to analyze the multimodal raw data to obtain environmental audio features of pest sounds and agricultural machinery operation sounds. The multi-source data preprocessing unit is used to reduce or project multi-source data such as soil, meteorology, crop images, and audio to improve the efficiency of subsequent fusion.

5. A multimodal deep neural network model based on continuous learning according to claim 1, characterized in that: The multimodal information fusion module includes a single-modal correlation analysis unit, a cross-modal correlation analysis unit, an information fusion projection unit, and a decision feature integration unit connected in sequence. The single-modal correlation analysis unit is used to perform self-attention modeling based on the key agricultural features, analyzing the internal correlation of soil parameters and the internal context of the correlation of meteorological elements within the single-modal features. The cross-modal association analysis unit is used to analyze cross-modal associations of soil-crop, weather-yield, and pests-environment based on the key agricultural features, and to use crop vision, soil sensor, and weather features as query, key, and value for cross-modal interaction. The information fusion projection unit is used to map multimodal agricultural information after cross-modal correlation analysis to a unified decision space, perform linear transformation on the attention output, and ensure numerical stability through layer normalization. The decision feature integration unit is used to integrate various types of agricultural information obtained by the information fusion projection unit to form decision features, and to perform weighted accumulation or splicing of multi-level attention results.

6. The multimodal deep neural network model based on continuous learning according to claim 1, characterized in that: The knowledge continuous learning module includes a crop growth pattern memory unit, a knowledge protection and updating unit, and a knowledge consistency maintenance unit connected in sequence. The crop growth pattern memory unit is used to store typical crop growth patterns, pest and disease characteristics, and best management practices. After the agricultural task is completed, the representative prototype of the integrated features is stored in the crop growth pattern library. The knowledge protection and updating unit is used to protect learned agricultural knowledge according to the importance of parameters, and to expand the network capacity when the demand for new crops and new pests and diseases changes significantly. The knowledge consistency maintenance unit is used to ensure the consistency between newly learned knowledge and existing agricultural knowledge, and performs positive and negative alignment training based on new and old agricultural tasks and multimodal agricultural samples.

7. A multimodal deep neural network model based on continuous learning according to claim 1, characterized in that: The intelligent decision-making module includes a task routing unit, a decision generation unit, and a multi-task decision-making unit connected in sequence. The task routing unit is used to call the corresponding decision branch according to different agricultural tasks, and to dynamically call different incremental network branches according to task labels or input features. The decision generation unit generates agricultural management decisions such as fertilization suggestions, irrigation plans, and pest and disease control schemes based on fusion features. The multi-task decision unit is used to simultaneously process multiple agricultural tasks such as crop monitoring, pest and disease diagnosis, and yield prediction, and maintains an independent output stream for each task so that they do not interfere with each other.

8. A multimodal deep neural network model based on continuous learning according to claim 1, characterized in that: The effect evaluation and feedback module includes an effect indicator calculation unit, a decision threshold judgment unit, and an effect feedback unit connected in sequence. The performance index calculation unit is used to calculate agricultural performance indicators such as crop health, yield prediction accuracy, and pest and disease identification accuracy. The decision threshold judgment unit is used to determine whether the management strategy needs to be adjusted or the model needs to be updated based on whether the performance indicator is optimal. It compares the current indicator with the baseline or the historical highest value to determine whether to trigger continuous learning and updating. The effect feedback unit is used to generate feedback signals based on actual agricultural effects, transform the evaluation results into reward or punishment signals, and feed them back to the knowledge continuous learning module.

9. The system based on a continuous learning multimodal deep neural network model according to any one of claims 1 to 8, characterized in that, The crop visual feature extraction unit in the feature extraction module adopts a VisionTransformer structure with N layers.

10. A learning method for a multimodal deep neural network model based on continuous learning, characterized in that, Includes the following steps: S1: Collect multimodal raw data on crop growth environment and perform data preprocessing; S2: Extract key agricultural features from crop vision, soil sensor, and meteorological multimodal data from S1; S3: Utilize a cross-modal attention mechanism to perform cross-modal fusion of the key agricultural features to obtain agricultural decision fusion features; S4: Update the fusion features of the crop growth pattern library based on the agricultural decision fusion features, implement parameter protection / expansion dynamic updates based on parameter importance, and achieve agricultural knowledge alignment through positive and negative alignment training; S5: Based on the fusion features, different tasks are executed by calling the corresponding deep neural network for decision-making and reasoning, and agricultural management prediction results are obtained; S6; Calculate various agricultural performance indicators based on the prediction results, evaluate the performance of the self-attention model on new and old agricultural tasks, and return the evaluation results to continuous learning.

Images