Method and system for filling multi-modal heterogeneous time series data of tobacco field based on diffusion model

Abstract

The application discloses a tobacco field multi-modal heterogeneous time sequence data completion method and system based on a diffusion model, and relates to the technical field of tobacco field disease and pest monitoring. The application discloses a tobacco field multi-modal heterogeneous time sequence data completion method and system based on a diffusion model, and relates to the technical field of tobacco field disease and pest monitoring. The application discloses a tobacco field multi-modal heterogeneous time sequence data completion method and system based on a diffusion model, and relates to the technical field of tobacco field disease and pest monitoring. The application discloses a tobacco field multi-modal heterogeneous time sequence data completion method and system based on a diffusion model, and relates to the technical field of tobacco field disease and pest monitoring. The application discloses a tobacco field multi-modal heterogeneous time sequence data completion method and system based on a diffusion model, and relates to the technical field of tobacco field disease and pest monitoring. The application discloses a tobacco field multi-modal heterogeneous time sequence data completion method and system based on a diffusion model, and relates to the technical field of tobacco field disease and pest monitoring.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for completing multimodal heterogeneous time-series data of tobacco fields based on a diffusion model. Background Technology

[0002] In recent years, the situation regarding the control of pests, diseases, and weeds in tobacco fields has become severe and complex due to factors such as agricultural industrial restructuring and regional climate change. Current pest and disease control recommendations largely rely on predictions generated from single historical data sources (such as remote sensing images and meteorological data), neglecting the comprehensive impact of climate, region, soil conditions, and modal data such as video, images, and production process records on pest and disease occurrence. This leads to an inability to accurately characterize the impact of pests and diseases when significant changes occur in the external environment. By utilizing multimodal heterogeneous time-series data, it is possible to effectively explore the growth and reproduction of pests and diseases in tobacco fields, as well as their potential correlations with factors such as meteorological conditions, tobacco varieties, planting environment, and pesticide use. This provides a scientific basis for the subsequent detection and prediction of abnormal pest and disease occurrences.

[0003] Due to the frequent equipment data gaps, large fluctuations in data modality quality, and difficulties in fine-grained classification of pests, diseases, and weeds in tobacco field scenarios, completing missing values ​​in multimodal time series is particularly important. Although existing multimodal heterogeneous time series data completion strategies, such as structural variational approximation and recurrent neural networks, have achieved some success, the constantly changing types, quantities, densities, distribution ranges, and new pest and disease species in tobacco fields mean that existing methods still face significant heterogeneity, poor scalability, and complexity when completing multimodal tobacco pest and disease data, making them difficult to adapt to open and complex tobacco field scenarios. Therefore, designing a multimodal, multi-view data completion algorithm tailored to the characteristics of tobacco pest and disease data to effectively complete multimodal time series data including pest and disease categories, remote sensing data, and meteorological data has become a critical problem that urgently needs to be solved. Summary of the Invention

[0004] This invention overcomes the shortcomings of the prior art and provides a method and system for completing multimodal heterogeneous time-series data in tobacco fields based on a diffusion model. Its important purpose is to provide data support for monitoring and predicting diseases, pests and weeds in tobacco fields.

[0005] To achieve the above objectives, the first aspect of this invention provides a method for completing multimodal heterogeneous time-series data of tobacco fields based on a diffusion model, comprising: Obtain multi-source raw data from representative tobacco fields throughout the entire tobacco growth period, classify, identify, and record the characteristics of diseases, pests, and weeds in the multi-source raw data, and construct a raw multimodal dataset with species labels. Based on the original multimodal dataset, deep learning technology is used to perform fine-grained annotation and aggregation on the three types of modal data in the original multimodal dataset. After time stamp alignment, the data are organized into serialized samples to construct a multimodal heterogeneous time series dataset. A multimodal heterogeneous time series data completion network is constructed based on a diffusion model. Time series data with completeness higher than a preset threshold are selected from the multimodal heterogeneous time series dataset to train the completion network and learn the joint distribution law of the multimodal data. Temporal representations of each modality are obtained through the Transformer model. A contrastive learning mechanism is introduced to construct a cross-modal contrastive loss function. The training process of the diffusion model is optimized by utilizing the correlation constraints between modalities, and optimized multimodal heterogeneous temporal data is generated to complete the network. The missing tobacco field multi-source time series data is obtained, and the missing data segments are marked as regions to be completed. The data is then input into the optimized multimodal heterogeneous time series data completion network. The missing data is generated through a reverse diffusion process to obtain the completed tobacco field multi-source time series dataset.

[0006] In this scheme, the acquisition of multi-source raw data from representative tobacco fields throughout the entire tobacco growth period, the classification, identification, and feature recording of pests, diseases, and weeds in the multi-source raw data, and the construction of a raw multimodal dataset with species labels specifically include: Representative tobacco fields were selected as fixed monitoring points. During the entire growth period of tobacco, air temperature, soil moisture content and light intensity were continuously collected by environmental sensors. High-resolution imaging equipment was used to obtain leaf image sequences of tobacco plants. Plant protection technicians recorded the characteristics of diseases, pests and weeds on site, forming multi-source raw data covering three types of data sources. The structured data collected by environmental sensors are archived according to date and monitoring point number. For tobacco leaf image sequences, a semi-automatic annotation tool is used to delineate the boundaries of disease lesions, areas of insect feeding traces, and areas covered by weeds, and disease status category labels are assigned to complete the preliminary classification and identification of visual data. The text data recorded on site was extracted using entity recognition methods in natural language processing to extract structured fields of disease type, pest type, weed type, infected part and degree of damage. After cross-validation with the preliminary classification and identification results of visual data, a preliminary classification and identification dataset was formed. For plants with similar pest and disease characteristics that are difficult to distinguish with the naked eye in the preliminary identification, as well as plants with two or more diseases, the collectors will mark the plants and collect samples on site. At the same time, they will use high-resolution cameras to obtain images of the plants with the overall morphology of the plant, close-up of the lesions, close-up of the insects, and the morphological characteristics of the weeds. Based on the acquired samples and images of problematic plants, morphological observation, culture and separation, and physiological and biochemical tests were used to identify diseases, pests and weeds in the laboratory. The laboratory identification results were then associated and integrated with the preliminary classification and identification dataset to construct an original multimodal dataset with clear species labels that includes structured data, visual data and text data.

[0007] In this scheme, based on the original multimodal dataset, deep learning technology is used to perform fine-grained annotation and aggregation on the three modalities of the original multimodal dataset, and after timestamp alignment, the data is organized into serialized samples to construct a multimodal heterogeneous time-series dataset. Specifically, this includes: Visual data from the original multimodal dataset is imported into the Faster R-CNN object detection network with ResNet as the backbone network, and the bounding box coordinates, confidence scores and category prediction results of the candidate regions are output. For candidate regions with confidence scores higher than the first threshold, the coordinates of the lesion center are extracted and the relative area of ​​the lesion is calculated. For regions with confidence scores lower than the first threshold, they are pushed to the manual review interface for confirmation or correction, and visual fine-grained annotation results are generated. Text data from the original multimodal dataset is input into a BERT-based joint model for text classification and entity recognition to extract entities. After extracting contextual semantic features through a multi-layer Transformer encoder, the categories of harmful organisms are predicted. The BIO annotation model was used to extract the entities of infection site, degree of harm and number of occurrences. Text records with predictions below the second threshold were manually reviewed and corrected. The extraction results were matched with the visual fine-grained annotation results according to the monitoring sample point, collection date and plant number to form a multimodal heterogeneous feature that integrates visual features and text semantic features. The air temperature, soil moisture content and light intensity data collected by environmental sensors are grouped according to the monitoring sample point number. The arithmetic mean, maximum value, minimum value and standard deviation of the daily collected values ​​of each sample point are calculated to form the environmental summary features. These features are then matched with the multimodal heterogeneous features according to the monitoring sample point number and date to generate comprehensive multimodal heterogeneous features. The comprehensive multimodal heterogeneous features of each tobacco plant at each monitoring point throughout the entire growth period of the tobacco were arranged in ascending order of date to form a serialized sample. After standardization, a multimodal heterogeneous time-series dataset containing three channels—structured environmental data, visual features, and textual semantic features—was constructed.

[0008] In this scheme, the construction of a multimodal heterogeneous time-series data completion network based on a diffusion model, and the selection of time-series data with completeness exceeding a preset threshold from the multimodal heterogeneous time-series dataset to train the completion network and learn the joint distribution pattern of the multimodal data, specifically includes: The missing rate of each sample in the structured environment data channel, visual feature channel and text semantic feature channel is statistically analyzed from the multimodal heterogeneous time series dataset. Samples with missing rates in all three channels below the preset threshold are marked as complete samples and selected as training sets. Continuous time series segments are extracted with preset step size and window length to form standard input samples. A diffusion model is constructed for structured environment data channels. The forward diffusion process is defined to gradually add Gaussian noise to the complete time segment. The total number of diffusion steps is set and the noise variance of each step is calculated. The reverse process constructs a noise prediction network with the goal of noise prediction. The input is the noise sample of the current step, the time code, and the multimodal conditional embedding. The standard input sample is input into the diffusion model to randomly sample the diffusion steps and calculate the noisy sample. Based on the noise prediction network, the network parameters are optimized by minimizing the mean square error between the predicted noise and the real noise. Cross-modal consistency constraints are introduced during the training process, and the hidden features of the visual feature channel and the text feature channel are fused as conditional embeddings through the attention mechanism. Independent diffusion models are constructed for the visual feature channel and the text feature channel respectively. The diffusion model for the visual feature channel uses structured environmental data and text features as conditional embeddings, while the diffusion model for the text feature channel uses structured environmental data and visual features as conditional embeddings, forming a joint training framework with mutual conditional constraints among the three channels. Each channel is trained independently using the same network architecture and loss function. By using an alternating training method, the diffusion models of the three channels converge synchronously. After training, each channel model learns its own data distribution patterns and cross-modal condition generation capabilities. The diffusion models of the three channels are then integrated to construct a multimodal heterogeneous time series data completion network.

[0009] In this scheme, the temporal representations of each modality are obtained through the Transformer model, a contrastive learning mechanism is introduced to construct a cross-modal contrastive loss function, and the training process of the diffusion model is optimized by utilizing inter-modal correlation constraints to generate an optimized multimodal heterogeneous temporal data completion network. Specifically, this includes: All complete samples in the training set are input into the three-channel diffusion model that has been trained. The complete completion sequence of each channel is generated through the reverse sampling process, resulting in a set of multimodal completion sequences aligned in the time dimension for the structured environment data channel, visual feature channel and text feature channel. The multimodal completion sequence of each sample is input into the Transformer encoder model. The Transformer encoder includes a multi-head self-attention layer and a feedforward neural network. The sequence of each modality is position-encoded and then the temporal dependency is extracted through multi-head self-attention. The sequence is then aggregated by a global average pooling layer to obtain a structured environment data representation vector, a visual feature representation vector, and a text feature representation vector. A cross-modal contrastive loss function is constructed based on the representation vectors of each modality. The cosine similarity is calculated by treating different modal representations of the same sample as positive sample pairs and the cosine similarity is calculated by treating the same modal representations of different samples as negative sample pairs. A normalized contrastive loss term is formed by narrowing the distance between positive sample pairs and widening the distance between negative sample pairs. The contrastive loss function is added as an auxiliary constraint to the training objective of the diffusion model. In each iteration, the diffusion loss and contrastive loss of each channel are calculated simultaneously. The diffusion loss and contrastive loss are weighted and summed to obtain the total loss. The parameters of the diffusion model and the Transformer model are updated simultaneously through backpropagation. Through multiple rounds of iterative training, the parameters of the diffusion model are continuously optimized under the constraint of contrastive loss. The optimized three-channel diffusion model is then integrated with the Transformer model to generate an optimized multimodal heterogeneous time-series data completion network.

[0010] In this scheme, the acquisition of missing multi-source time-series data of tobacco fields, marking the missing data segments as regions to be completed, and inputting them into the optimized multimodal heterogeneous time-series data completion network, generates missing data through a reverse diffusion process to obtain the completed multi-source time-series dataset of tobacco fields, specifically including: Acquire multi-source time series data of tobacco fields with missing values, scan the time series of each sample to be completed, identify the time points where the values ​​are empty in the structured environment data channel, visual feature channel and text semantic feature channel and mark them as regions to be completed, and record the complete observation values ​​in each channel as conditional embedding to form an input data structure containing observation masks and missing masks. The input data structure is imported into the optimized multimodal heterogeneous temporal data completion network. Random samples are taken from the standard Gaussian distribution as initial noise samples. The initial noise samples, the time encoding of the current diffusion step, and the conditional embedding are used to complete the data in the common input noise prediction network. During the reverse diffusion process, the total number of diffusion steps decreases to 1. The noise prediction network outputs the predicted noise components based on the current noise sample, time encoding, and conditional embedding. A classifier-independent guidance mechanism is introduced to calculate the noise sample of the previous time step and combine the noise prediction values ​​under both conditional and unconditional settings to generate the degree of dependence of the result on the observation conditions. Repeat the reverse diffusion process until the diffusion step count reaches zero to obtain the completion sequence of the current channel. Use the completion sequence as a conditional embedding to assist in the generation of other channels. Iterate through the three channels in turn until the missing positions of all channels are filled. During the iterative completion process, the cross-modal semantic representation extracted by the Transformer model is used to constrain the generation process, ensuring that the completion results maintain semantic consistency across modalities. All the completed channel data are reorganized according to the original time order, and the completed tobacco field multi-source time series dataset is output.

[0011] The second aspect of this invention provides a multimodal heterogeneous time-series data completion system for tobacco fields based on a diffusion model. This system includes: a data acquisition and classification module, a data processing and construction module, a completion network construction and training module, a contrastive learning optimization module, and a data completion and application module. The data acquisition and classification module is used to acquire multimodal raw data from the tobacco field, collect production environment and biological characteristic information using sensors and imaging equipment, and classify and identify pests, diseases and weeds. The data processing and construction module is connected to the data acquisition and classification module, and is used to construct a multimodal heterogeneous time-series dataset covering structured data, visual data and text data; The completion network construction and training module is connected to the data processing and construction module, and is used to select time series segments with completeness higher than a preset threshold from the multimodal heterogeneous time series dataset as training samples to construct a multimodal heterogeneous time series data completion network; The contrastive learning optimization module is connected to the completion network construction and training module and is used to optimize the completion network for multimodal heterogeneous temporal data. The data completion and application module is connected to the comparative learning and optimization module, and is used to complete the missing multi-source time series data of tobacco fields to obtain the completed multi-source time series dataset of tobacco fields.

[0012] Compared with existing technologies, this invention has the following advantages and beneficial effects: It employs heterogeneous multimodal temporal joint modeling and utilizes cross-modal consistency constraints and alignment mechanisms to ensure that the completed results are not only numerically reasonable but also consistent at the semantic level of pests, diseases, and weeds. This makes it more suitable for the real-world data characteristics of tobacco fields, such as frequent equipment measurement gaps, large fluctuations in modal quality, and difficulty in distinguishing fine-grained categories. It constructs a multimodal heterogeneous temporal series dataset of tobacco pests, diseases, and weeds with temporal continuity and integrity, providing data support for monitoring and predicting pests, diseases, and weeds in tobacco fields. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments or examples of the present invention, the drawings used in the embodiments or examples will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained according to these drawings without creative effort.

[0014] Figure 1 A flowchart of a method for completing multimodal heterogeneous time-series tobacco field data based on a diffusion model, provided in an embodiment of the present invention; Figure 2 A block diagram of a tobacco field multimodal heterogeneous time-series data completion system based on a diffusion model is provided in an embodiment of the present invention. The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0015] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0016] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0017] Figure 1 A flowchart of a method for completing multimodal heterogeneous time-series tobacco field data based on a diffusion model, provided in an embodiment of the present invention; like Figure 1 As shown, this invention provides a flowchart of a method for completing multimodal heterogeneous time-series data in tobacco fields based on a diffusion model, including: S102, Obtain multi-source raw data of representative tobacco fields throughout the entire tobacco growth period, classify, identify and record the pests, diseases and weeds in the multi-source raw data, and construct a raw multimodal dataset with species labels. S104. Based on the original multimodal dataset, deep learning technology is used to perform fine-grained annotation and aggregation on the three types of modal data of the original multimodal dataset, and after time stamp alignment, they are organized into serialized samples to construct a multimodal heterogeneous time series dataset. S106, Construct a multimodal heterogeneous time series data completion network based on the diffusion model, and select time series data with completeness higher than a preset threshold from the multimodal heterogeneous time series dataset to train the completion network and learn the joint distribution law of multimodal data; S108 obtains temporal representations of each modality through the Transformer model, introduces a contrastive learning mechanism to construct a cross-modal contrastive loss function, utilizes intermodal correlation constraints and optimizes the training process of the diffusion model, and generates optimized multimodal heterogeneous temporal data to complete the network. S110: Obtain the missing multi-source time series data of the tobacco field, mark the missing data segments as regions to be completed, and input them into the optimized multimodal heterogeneous time series data completion network. The missing data is generated through a reverse diffusion process to obtain the completed multi-source time series dataset of the tobacco field.

[0018] Furthermore, in a preferred embodiment of the present invention, the step of acquiring multi-source raw data from representative tobacco fields throughout the entire tobacco growth period, classifying and identifying pests, diseases, and weeds in the multi-source raw data, and constructing a raw multimodal dataset with species labels specifically includes: Representative tobacco fields were selected as fixed monitoring points. During the entire growth period of tobacco, air temperature, soil moisture content and light intensity were continuously collected by environmental sensors. High-resolution imaging equipment was used to obtain leaf image sequences of tobacco plants. Plant protection technicians recorded the characteristics of diseases, pests and weeds on site, forming multi-source raw data covering three types of data sources. The structured data collected by environmental sensors are archived according to date and monitoring point number. For tobacco leaf image sequences, a semi-automatic annotation tool is used to delineate the boundaries of disease lesions, areas of insect feeding traces, and areas covered by weeds, and disease status category labels are assigned to complete the preliminary classification and identification of visual data. The text data recorded on site was extracted using entity recognition methods in natural language processing to extract structured fields of disease type, pest type, weed type, infected part and degree of damage. After cross-validation with the preliminary classification and identification results of visual data, a preliminary classification and identification dataset was formed. For plants with similar pest and disease characteristics that are difficult to distinguish with the naked eye in the preliminary identification, as well as plants with two or more diseases, the collectors will mark the plants and collect samples on site. At the same time, they will use high-resolution cameras to obtain images of the plants with the overall morphology of the plant, close-up of the lesions, close-up of the insects, and the morphological characteristics of the weeds. Based on the acquired samples and images of problematic plants, morphological observation, culture and separation, and physiological and biochemical tests were used to identify diseases, pests and weeds in the laboratory. The laboratory identification results were then associated and integrated with the preliminary classification and identification dataset to construct an original multimodal dataset with clear species labels that includes structured data, visual data and text data.

[0019] It should be noted that, firstly, representative tobacco fields in typical tobacco-growing areas need to be selected as fixed monitoring points. Within each selected monitoring point, three types of data collection operations are carried out simultaneously throughout the tobacco's entire growth cycle, focusing on key growth stages: mid-to-late seedling stage, mid-to-late vigorous growth stage, budding stage, and harvest stage. The first type of operation utilizes environmental sensors deployed in the tobacco fields to continuously collect air temperature, soil moisture content, and light intensity at an hourly sampling frequency, forming structured time-series data with precise timestamps. The second type of operation involves personnel using high-resolution imaging equipment to photograph tobacco plants along a fixed inspection route during periods of stable light conditions, acquiring visual image sequences that include the overall morphology of the tobacco plants, leaf lesions, and insect characteristics. The third type of operation is conducted simultaneously with image capture, with technicians possessing plant protection expertise recording preliminary characteristic descriptions of diseases, pests, and weeds on a plant-by-plant basis in standardized forms. After data collection is completed, the structured data collected by the environmental sensors is archived according to date and monitoring point number, forming a daily environmental feature vector for each sampling point. For visual image sequences, a semi-automatic annotation tool was used by annotators to delineate the boundaries of disease lesions, areas marked by insect feeding, and areas covered by weeds. The annotation tool automatically calculated the pixel area and location coordinates of the delineated areas. Annotators then assigned labels based on their professional judgment, indicating the disease type, insect type, or weed type, and the corresponding level of infection, completing the initial classification and annotation of the visual data. For text data, a BERT-based entity recognition method was used to extract structured fields for disease type, insect type, weed type, infected location, and degree of damage. The extraction results were cross-validated with the visual annotation results. In cases where there were discrepancies between the text and image records, the image annotation results took precedence and were reviewed and confirmed by the technical lead, forming a preliminary classification and identification dataset.

[0020] For plants exhibiting similar pest and disease characteristics that are difficult to distinguish visually during preliminary identification, as well as plants infected with two or more diseases simultaneously, the collectors additionally marked the plants on-site. Leaf tissue samples, including those from the diseased / healthy zone, insect specimens, and whole weed specimens were collected and placed in numbered sampling containers. Simultaneously, a high-resolution camera was used to photograph the plants from multiple angles and at multiple scales, obtaining images of the suspected plants, including overall plant morphology, close-ups of lesions, close-ups of insects, and weed morphological characteristics. The collected specimens were then transferred to the laboratory. For diseased specimens, tissue separation methods were used for microscopic structural observation to determine the pathogen species; for example, observing conidial morphology to identify brown spot disease. For insect specimens, stereomicroscopic observation was used to observe the morphological characteristics of antennal segments, wing vein direction, and leg structure, identifying the pest species according to insect taxonomy keys; for example, identifying tobacco budworms by wing vein characteristics. For weed specimens, morphological observation methods were used to determine the weed species based on leaf shape, phyllotaxis, and floral structure characteristics. The laboratory identification results were linked and integrated with the preliminary classification and identification dataset to generate a unique record number for each sample, thus constructing an original multimodal dataset in which each sample has a clear category label and covers structured data, visual data, and text data.

[0021] Furthermore, in a preferred embodiment of the present invention, the step of using deep learning technology to perform fine-grained annotation and aggregation processing on the three types of modal data of the original multimodal dataset based on the original multimodal dataset, organizing them into serialized samples after timestamp alignment, and constructing a multimodal heterogeneous time-series dataset specifically includes: Visual data from the original multimodal dataset is imported into the Faster R-CNN object detection network with ResNet as the backbone network, and the bounding box coordinates, confidence scores and category prediction results of the candidate regions are output. For candidate regions with confidence scores higher than the first threshold, the coordinates of the lesion center are extracted and the relative area of ​​the lesion is calculated. For regions with confidence scores lower than the first threshold, they are pushed to the manual review interface for confirmation or correction, and visual fine-grained annotation results are generated. Text data from the original multimodal dataset is input into a BERT-based joint model for text classification and entity recognition to extract entities. After extracting contextual semantic features through a multi-layer Transformer encoder, the categories of harmful organisms are predicted. The BIO annotation model was used to extract the entities of infection site, degree of harm and number of occurrences. Text records with predictions below the second threshold were manually reviewed and corrected. The extraction results were matched with the visual fine-grained annotation results according to the monitoring sample point, collection date and plant number to form a multimodal heterogeneous feature that integrates visual features and text semantic features. The air temperature, soil moisture content and light intensity data collected by environmental sensors are grouped according to the monitoring sample point number. The arithmetic mean, maximum value, minimum value and standard deviation of the daily collected values ​​of each sample point are calculated to form the environmental summary features. These features are then matched with the multimodal heterogeneous features according to the monitoring sample point number and date to generate comprehensive multimodal heterogeneous features. The comprehensive multimodal heterogeneous features of each tobacco plant at each monitoring point throughout the entire growth period of the tobacco were arranged in ascending order of date to form a serialized sample. After standardization, a multimodal heterogeneous time-series dataset containing three channels—structured environmental data, visual features, and textual semantic features—was constructed.

[0022] It should be noted that, firstly, fine-grained quantitative annotation is performed on the visual data. The sequence of tobacco leaf images is input into a Faster R-CNN object detection network with ResNet as the backbone. This network generates candidate regions through a region proposal network, and then outputs the bounding box coordinates, confidence scores, and corresponding category predictions for each candidate region via classification and regression branches. For example, when the network detects a lesion area with a confidence score of 0.92, the coordinates of the center pixel of that area are automatically extracted as the lesion location, and the ratio of the total number of pixels within the bounding box to the total number of pixels in the image is calculated as the relative area of ​​the lesion. For candidate regions with confidence scores between a preset threshold, they are pushed to a manual review interface for confirmation or correction by the annotators to ensure the accuracy of the annotation results. Deep cleaning and structured extraction are performed on the text data. The text descriptions recorded by technicians on-site are input into a BERT-based joint text classification and entity recognition model. This model extracts contextual semantic features through a multi-layer Transformer encoder, and then uses a classification head at the output layer to predict the category of harmful organisms. The BIO annotation mode is used to extract entities such as the infected site, the degree of harm, and the number of occurrences. For example, for the description "irregular brown lesions appear on the top leaves of tobacco plants, with an infection area of ​​approximately 30%", the model can extract structured fields indicating the infected location as "top leaves" and the degree of damage as "30%". For text records with low prediction confidence, after manual review and correction, the extracted results are matched with the visual fine-grained annotation results according to the monitoring points, collection dates, and plant numbers, forming a multimodal feature record that integrates visual quantitative features and textual semantic features.

[0023] In terms of environmental data processing, air temperature, soil moisture content, and light intensity data collected by environmental sensors at an hourly frequency are grouped according to the monitoring sample point number. For each sample point, the daily collected values ​​are calculated for their arithmetic mean, maximum, minimum, and standard deviation, forming a daily-level environmental summary feature. This environmental summary feature is then matched with the aforementioned multimodal feature records according to the monitoring sample point number and date, generating a comprehensive multimodal feature record containing environmental data, visual features, and textual features. Finally, the comprehensive multimodal feature records for each tobacco plant at each monitoring sample point for all collected dates throughout the tobacco's entire growth period are arranged in ascending order by date, forming a time series sample. For cases where data is missing on certain dates due to equipment malfunction, null values ​​are retained at those time steps. All samples are standardized: continuous features are standardized using Z-scores, categorical features are one-hot encoded, and list features are padded with zero vectors to a fixed length. This results in a multimodal heterogeneous time series dataset where each sample contains three channels: structured environmental data, visual quantitative features, and textual semantic features, providing standardized input for subsequent diffusion model training.

[0024] Furthermore, in a preferred embodiment of the present invention, the step of constructing a multimodal heterogeneous time-series data completion network based on a diffusion model, and training the completion network by selecting time-series data with completeness higher than a preset threshold from the multimodal heterogeneous time-series dataset to learn the joint distribution law of the multimodal data, specifically includes: The missing rate of each sample in the structured environment data channel, visual feature channel and text semantic feature channel is statistically analyzed from the multimodal heterogeneous time series dataset. Samples with missing rates in all three channels below the preset threshold are marked as complete samples and selected as training sets. Continuous time series segments are extracted with preset step size and window length to form standard input samples. A diffusion model is constructed for structured environment data channels. The forward diffusion process is defined to gradually add Gaussian noise to the complete time segment. The total number of diffusion steps is set and the noise variance of each step is calculated. The reverse process constructs a noise prediction network with the goal of noise prediction. The input is the noise sample of the current step, the time code, and the multimodal conditional embedding. The standard input sample is input into the diffusion model to randomly sample the diffusion steps and calculate the noisy sample. Based on the noise prediction network, the network parameters are optimized by minimizing the mean square error between the predicted noise and the real noise. Cross-modal consistency constraints are introduced during the training process, and the hidden features of the visual feature channel and the text feature channel are fused as conditional embeddings through the attention mechanism. Independent diffusion models are constructed for the visual feature channel and the text feature channel respectively. The diffusion model for the visual feature channel uses structured environmental data and text features as conditional embeddings, while the diffusion model for the text feature channel uses structured environmental data and visual features as conditional embeddings, forming a joint training framework with mutual conditional constraints among the three channels. Each channel is trained independently using the same network architecture and loss function. By using an alternating training method, the diffusion models of the three channels converge synchronously. After training, each channel model learns its own data distribution patterns and cross-modal condition generation capabilities. The diffusion models of the three channels are then integrated to construct a multimodal heterogeneous time series data completion network.

[0025] It should be noted that, firstly, the training samples of the constructed multimodal heterogeneous time-series dataset were screened. The missing rate of each sample in the structured environment data channel, visual feature channel, and text semantic feature channel was calculated. Samples with a missing rate of less than 10% in all three channels were marked as complete samples and selected as the training set. A sliding window strategy was used to extract continuous time-series segments from the training set with a step size of 5 days and a window length of 30 days, forming standard input samples to ensure that the model could learn the short-term dependencies and long-term trends of the time-series data. A diffusion model was constructed for the structured environment data channel, defining a forward diffusion process that progressively adds Gaussian noise to the complete time-series segments. The total number of diffusion steps was set to 1000, and a cosine scheduling strategy was used to calculate the noise variance at each step, progressively transforming the original data into a standard Gaussian distribution. The reverse process was used to construct a noise prediction network with the goal of noise prediction. This network adopted a temporal U-Net architecture, with the input being the noise sample of the current step, the sine position encoding of the current diffusion step, and the multimodal conditional embedding, and the output being the predicted noise components. The selected standard input samples are fed into the diffusion model for training. After randomly sampling the diffusion steps and noise, noisy samples are calculated. The network parameters are optimized by minimizing the mean square error between the predicted noise and the actual noise. A cross-modal consistency constraint is introduced during training. The hidden features of the visual and textual feature channels are fused using an attention mechanism and used as conditional embeddings. This allows the model to learn the temporal dependencies of environmental data while utilizing visual and textual information to assist in the generation of environmental data. Independent diffusion models are constructed for the visual and textual feature channels, and each channel is trained independently using the same network architecture and loss function. The diffusion model for the visual feature channel uses structured environmental data and textual features as conditional embeddings, while the diffusion model for the textual feature channel uses structured environmental data and visual features as conditional embeddings, forming a joint training framework with mutual conditional constraints among the three channels. For example, when training the visual channel model, the complete observations of the environmental channel and the semantic descriptions of the textual channel serve as conditions, guiding the visual model to generate lesion image features that conform to actual phenological changes. Alternating training methods are used to ensure synchronous convergence of the three channel diffusion models, with the network parameters of the three channels updated sequentially in each iteration. After training, each channel model learns its own data distribution patterns and cross-modal condition generation capabilities, and the diffusion models of the three channels are integrated into a multimodal heterogeneous time series data completion network.

[0026] It is worth mentioning that complete time-series data was extracted from dataset D. As training samples, the multimodal heterogeneous time-series data X is represented as follows: = ( , , ... , ), in, Given a time series covering T time steps related to pests, diseases, and weeds, after obtaining the training samples, for each modality m... Construct and train diffusion models for {d, v, u} respectively. Minimize the loss function... This allows the model to learn the mean of the posterior distribution and its ability to recover the original data from noise. Specifically, the loss function for each mode m with respect to the diffusion model is expressed as: ( )= || ||, in, This provides complete data for mode m, where T is the time step. This represents the conditional probability distribution for the forward diffusion process. Representing the posterior The mean.

[0027] Furthermore, in a preferred embodiment of the present invention, the step of obtaining temporal representations of each modality through a Transformer model, introducing a contrastive learning mechanism to construct a cross-modal contrastive loss function, utilizing inter-modal correlation constraints and optimizing the training process of the diffusion model to generate an optimized multimodal heterogeneous temporal data completion network specifically includes: All complete samples in the training set are input into the three-channel diffusion model that has been trained. The complete completion sequence of each channel is generated through the reverse sampling process, resulting in a set of multimodal completion sequences aligned in the time dimension for the structured environment data channel, visual feature channel and text feature channel. The multimodal completion sequence of each sample is input into the Transformer encoder model. The Transformer encoder includes a multi-head self-attention layer and a feedforward neural network. The sequence of each modality is position-encoded and then the temporal dependency is extracted through multi-head self-attention. The sequence is then aggregated by a global average pooling layer to obtain a structured environment data representation vector, a visual feature representation vector, and a text feature representation vector. A cross-modal contrastive loss function is constructed based on the representation vectors of each modality. The cosine similarity is calculated by treating different modal representations of the same sample as positive sample pairs and the cosine similarity is calculated by treating the same modal representations of different samples as negative sample pairs. A normalized contrastive loss term is formed by narrowing the distance between positive sample pairs and widening the distance between negative sample pairs. The contrastive loss function is added as an auxiliary constraint to the training objective of the diffusion model. In each iteration, the diffusion loss and contrastive loss of each channel are calculated simultaneously. The diffusion loss and contrastive loss are weighted and summed to obtain the total loss. The parameters of the diffusion model and the Transformer model are updated simultaneously through backpropagation. Through multiple rounds of iterative training, the parameters of the diffusion model are continuously optimized under the constraint of contrastive loss. The optimized three-channel diffusion model is then integrated with the Transformer model to generate an optimized multimodal heterogeneous time-series data completion network.

[0028] It should be noted that after independently training the three channel diffusion models, a contrastive learning mechanism was introduced to optimize the models to further enhance cross-modal semantic consistency. First, all complete samples from the training set were input into the three trained channel diffusion models. Through a backsampling process, complete reconstruction sequences for each channel were generated, resulting in a set of multimodal reconstruction sequences that were strictly aligned temporally for the structured environment data channel, visual feature channel, and text feature channel. Next, the multimodal reconstruction sequence of each sample was input into a Transformer encoder model. This model adopts a standard Transformer encoder architecture, containing six multi-head self-attention layers and a feedforward neural network. The sequence of each modality first undergoes positional encoding to embed temporal order information, and then the temporal dependencies within the sequence are extracted through a multi-head self-attention mechanism. After multi-layer encoding, the corresponding hidden layer representation is output at each time step. Then, a global average pooling layer aggregates the information of the entire time series into a fixed-dimensional representation vector, thus obtaining the structured environment data representation vector, visual feature representation vector, and text feature representation vector. These three vectors describe the semantic information of the same pest and weed sample from different perspectives.

[0029] A cross-modal contrastive loss function is constructed based on the representation vectors of each modality. The core objective of this loss function is to bring different modal representations of the same object closer together and push the same modal representations of different objects further apart. Specifically, the visual and textual representations of the same sample are used as positive sample pairs to calculate cosine similarity, while the visual representation of the current sample is used as a negative sample pair to calculate cosine similarity. A temperature coefficient is used to adjust the sharpness of the similarity distribution, forming a normalized contrastive loss term. This makes the model tend to bring different modal representations of the same object closer together in the semantic space during optimization. This contrastive loss function is added as an auxiliary constraint to the training objective of the diffusion model, forming a joint optimization framework. In each iteration, the diffusion loss and cross-modal contrastive loss of each channel are calculated simultaneously. The diffusion loss and contrastive loss are weighted and summed according to preset weights to obtain the total loss. Backpropagation is used to update the parameters of both the diffusion model and the Transformer model. After multiple rounds of iterative training, the parameters of the diffusion model are continuously optimized under the constraint of the contrastive loss, making the generated modal reconstruction sequences more semantically aligned. Finally, the optimized three-channel diffusion model and the Transformer model are integrated into a unified multimodal heterogeneous time series data completion network. This network can generate data for missing modes using the complete information of the observed modes in subsequent completion tasks, and ensure that the completion results between different modes remain semantically consistent.

[0030] It is worth mentioning that multimodal data on the same pests, diseases, and weeds can be regarded as describing the characteristics of the project from multiple perspectives, and there is a certain correlation between data from different modalities. By calculating the cross-modal contrastive loss function, we can bring the different modal characteristics of the same pest, disease, and weed object at the same time point closer together, while pushing away irrelevant features.

[0031] The Transformer model g is used to obtain the representation of each mode of pests, diseases, and weeds i. , means as follows: = g( ; ), in, This represents the parameters of the converter model.

[0032] Next, a contrastive learning mechanism is introduced to construct a contrastive loss function. To maximize the similarity between different modalities of the same pest and weed infestation object, such as “yellow spot image” and “yellow spot text”, the diffusion model is constrained and optimized by utilizing the strong correlation between modalities.

[0033] The cross-modal contrastive loss function is specifically expressed as follows: , Where M represents the total number of modalities, m represents the modal index, i represents the index of disease, pest, and weed samples, and j represents the index of other samples within the batch. Let represent the representation vector of sample i in mode m. Let represent the representation vector of sample i in the next modality m+1. This represents the temperature coefficient.

[0034] Furthermore, in a preferred embodiment of the present invention, the step of acquiring missing multi-source time-series data of tobacco fields, marking the missing data segments as regions to be completed, and inputting them into the optimized multimodal heterogeneous time-series data completion network, generating missing data through a reverse diffusion process, and obtaining the completed multi-source time-series dataset of tobacco fields, specifically includes: Acquire multi-source time series data of tobacco fields with missing values, scan the time series of each sample to be completed, identify the time points where the values ​​are empty in the structured environment data channel, visual feature channel and text semantic feature channel and mark them as regions to be completed, and record the complete observation values ​​in each channel as conditional embedding to form an input data structure containing observation masks and missing masks. The input data structure is imported into the optimized multimodal heterogeneous temporal data completion network. Random samples are taken from the standard Gaussian distribution as initial noise samples. The initial noise samples, the time encoding of the current diffusion step, and the conditional embedding are used to complete the data in the common input noise prediction network. During the reverse diffusion process, the total number of diffusion steps decreases to 1. The noise prediction network outputs the predicted noise components based on the current noise sample, time encoding, and conditional embedding. A classifier-independent guidance mechanism is introduced to calculate the noise sample of the previous time step and combine the noise prediction values ​​under both conditional and unconditional settings to generate the degree of dependence of the result on the observation conditions. Repeat the reverse diffusion process until the diffusion step count reaches zero to obtain the completion sequence of the current channel. Use the completion sequence as a conditional embedding to assist in the generation of other channels. Iterate through the three channels in turn until the missing positions of all channels are filled. During the iterative completion process, the cross-modal semantic representation extracted by the Transformer model is used to constrain the generation process, ensuring that the completion results maintain semantic consistency across modalities. All the completed channel data are reorganized according to the original time order, and the completed tobacco field multi-source time series dataset is output.

[0035] It should be noted that, firstly, multi-source time-series data of tobacco fields with missing values ​​are acquired. The time series of each sample to be completed is scanned to identify time points where values ​​are empty in the structured environment data channel, visual feature channel, and text semantic feature channel. These locations are uniformly marked as regions to be completed. Simultaneously, the complete observations in each channel are encoded and used as conditional embeddings, forming an input data structure containing observation masks and missing value masks. The marked samples are then input into the optimized multimodal heterogeneous time-series data completion network. For the channel currently needing completion, a random sample is taken from a standard Gaussian distribution as an initial noise sample. This noise sample, the time encoding of the current diffusion step, and the conditional embedding are jointly input into the noise prediction network. The construction of the conditional embedding depends on the observation data; for example, when completing the visual feature channel, the structured environment data and text features are encoded and used as conditional embeddings to assist in the generation of visual features. During the back-diffusion process, the total diffusion step count decreases to 1, and the noise prediction network outputs the predicted noise component based on the current noise sample, time encoding, and conditional embedding. A classifier-independent guidance mechanism is introduced to jointly calculate noise prediction values ​​under both conditional and unconditional settings. Guiding weights adjust the dependence of the generated results on the observation conditions; a larger guiding weight results in a more faithful completion of the observed data. The noise sample from the previous time step is calculated. The reverse diffusion process is repeated until the diffusion step count reaches zero, obtaining the complete completion sequence for the current channel. Missing values ​​are filled with generated values, while the observed values ​​remain unchanged. This completed sequence is encoded and used as a conditional embedding to assist in the generation of other channels. Iterative completion is performed on all three channels sequentially. During the iterative completion process, cross-modal semantic representations extracted by the Transformer model are used to constrain the noise prediction network, ensuring semantic consistency across modalities. All completed channel data are reorganized in their original chronological order, outputting a completed tobacco multi-source time-series dataset that is continuous in time series and semantically consistent across modalities. It is worth mentioning that the classifier-independent guidance mechanism is a control technique in the sampling process of diffusion models. Its core idea is to adjust the dependence of the generated results on conditional information by jointly combining conditional and unconditional predictions without relying on additional classifier training. In the backsampling process of the standard diffusion model, the noise prediction network can output conditional prediction noise based on the input conditional embedding, or it can output unconditional prediction noise without inputting a conditional embedding. The classifier-independent guidance mechanism linearly combines these two predictions, that is, it adds the unconditional prediction to the guidance weight multiplied by the difference between the conditional and unconditional predictions to obtain the corrected noise prediction value. When the guidance weight is greater than 1, the generated results are more biased towards satisfying the conditional constraints, making the completed data more faithful to the observation information; when the guidance weight is equal to 1, it degenerates into standard conditional generation; when the guidance weight is less than 1, the generated results are more diverse. In this invention, this mechanism is used to adjust the dependence on observation conditions during multimodal time-series data completion. The larger the guidance weight, the closer the completion result is to the constraints of the existing observation data, ensuring that the generated missing data is semantically consistent with the true observations.

[0036] Figure 2 A diffusion-based multimodal heterogeneous time-series data completion system 2 for tobacco fields is provided as an embodiment of the present invention. The system includes: a data acquisition and classification module, a data processing and construction module, a completion network construction and training module, a contrastive learning optimization module, and a data completion and application module. The data acquisition and classification module is used to acquire multimodal raw data from the tobacco field, collect production environment and biological characteristic information using sensors and imaging equipment, and classify and identify pests, diseases and weeds. The data processing and construction module is connected to the data acquisition and classification module, and is used to construct a multimodal heterogeneous time-series dataset covering structured data, visual data and text data; The completion network construction and training module is connected to the data processing and construction module, and is used to select time series segments with completeness higher than a preset threshold from the multimodal heterogeneous time series dataset as training samples to construct a multimodal heterogeneous time series data completion network; The contrastive learning optimization module is connected to the completion network construction and training module and is used to optimize the completion network for multimodal heterogeneous temporal data. The data completion and application module is connected to the comparative learning and optimization module, and is used to complete the missing multi-source time series data of tobacco fields to obtain the completed multi-source time series dataset of tobacco fields.

[0037] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0038] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0039] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0040] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0041] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0042] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for completing multimodal heterogeneous time-series data of tobacco fields based on a diffusion model, characterized in that, include: Obtain multi-source raw data from representative tobacco fields throughout the entire tobacco growth period, classify, identify, and record the characteristics of diseases, pests, and weeds in the multi-source raw data, and construct a raw multimodal dataset with species labels. Based on the original multimodal dataset, deep learning technology is used to perform fine-grained annotation and aggregation on the three types of modal data in the original multimodal dataset. After time stamp alignment, the data are organized into serialized samples to construct a multimodal heterogeneous time series dataset. A multimodal heterogeneous time series data completion network is constructed based on a diffusion model. Time series data with completeness higher than a preset threshold are selected from the multimodal heterogeneous time series dataset to train the completion network and learn the joint distribution law of the multimodal data. Temporal representations of each modality are obtained through the Transformer model. A contrastive learning mechanism is introduced to construct a cross-modal contrastive loss function. The training process of the diffusion model is optimized by utilizing the correlation constraints between modalities, and optimized multimodal heterogeneous temporal data is generated to complete the network. The missing tobacco field multi-source time series data is obtained, and the missing data segments are marked as regions to be completed. The data is then input into the optimized multimodal heterogeneous time series data completion network. The missing data is generated through a reverse diffusion process to obtain the completed tobacco field multi-source time series dataset.

2. The method for completing multimodal heterogeneous time-series data of tobacco fields based on a diffusion model according to claim 1, characterized in that, The process of obtaining multi-source raw data from representative tobacco fields throughout the entire tobacco growth period, classifying and identifying pests, diseases, and weeds in the multi-source raw data, and recording their characteristics to construct a raw multimodal dataset with species labels specifically includes: Representative tobacco fields were selected as fixed monitoring points. During the entire growth period of tobacco, air temperature, soil moisture content and light intensity were continuously collected by environmental sensors. High-resolution imaging equipment was used to obtain leaf image sequences of tobacco plants. Plant protection technicians recorded the characteristics of diseases, pests and weeds on site, forming multi-source raw data covering three types of data sources. The structured data collected by environmental sensors are archived according to date and monitoring point number. For tobacco leaf image sequences, a semi-automatic annotation tool is used to delineate the boundaries of disease lesions, areas of insect feeding traces, and areas covered by weeds, and disease status category labels are assigned to complete the preliminary classification and identification of visual data. The text data recorded on site was extracted using entity recognition methods in natural language processing to extract structured fields of disease type, pest type, weed type, infected part and degree of damage. After cross-validation with the preliminary classification and identification results of visual data, a preliminary classification and identification dataset was formed. For plants with similar pest and disease characteristics that are difficult to distinguish with the naked eye in the preliminary identification, as well as plants with two or more diseases, the collectors will mark the plants and collect samples on site. At the same time, they will use high-resolution cameras to obtain images of the plants with the overall morphology of the plant, close-up of the lesions, close-up of the insects, and the morphological characteristics of the weeds. Based on the acquired samples and images of problematic plants, morphological observation, culture and separation, and physiological and biochemical tests were used to identify diseases, pests and weeds in the laboratory. The laboratory identification results were then associated and integrated with the preliminary classification and identification dataset to construct an original multimodal dataset with clear species labels that includes structured data, visual data and text data.

3. The method for completing multimodal heterogeneous time-series data of tobacco fields based on a diffusion model according to claim 1, characterized in that, Based on the original multimodal dataset, deep learning techniques are used to perform fine-grained annotation and aggregation on the three modalities of the original multimodal dataset. After timestamp alignment, the data is organized into serialized samples to construct a multimodal heterogeneous time-series dataset, specifically including: Visual data from the original multimodal dataset is imported into the Faster R-CNN object detection network with ResNet as the backbone network, and the bounding box coordinates, confidence scores and category prediction results of the candidate regions are output. For candidate regions with confidence scores higher than the first threshold, the coordinates of the lesion center are extracted and the relative area of ​​the lesion is calculated. For regions with confidence scores lower than the first threshold, they are pushed to the manual review interface for confirmation or correction, and visual fine-grained annotation results are generated. Text data from the original multimodal dataset is input into a BERT-based joint model for text classification and entity recognition to extract entities. After extracting contextual semantic features through a multi-layer Transformer encoder, the categories of harmful organisms are predicted. The BIO annotation model was used to extract the entities of infection site, degree of harm and number of occurrences. Text records with predictions below the second threshold were manually reviewed and corrected. The extraction results were matched with the visual fine-grained annotation results according to the monitoring sample point, collection date and plant number to form a multimodal heterogeneous feature that integrates visual features and text semantic features. The air temperature, soil moisture content and light intensity data collected by environmental sensors are grouped according to the monitoring sample point number. The arithmetic mean, maximum value, minimum value and standard deviation of the daily collected values ​​of each sample point are calculated to form the environmental summary features. These features are then matched with the multimodal heterogeneous features according to the monitoring sample point number and date to generate comprehensive multimodal heterogeneous features. The comprehensive multimodal heterogeneous features of each tobacco plant at each monitoring point throughout the entire growth period of the tobacco were arranged in ascending order of date to form a serialized sample. After standardization, a multimodal heterogeneous time-series dataset containing three channels—structured environmental data, visual features, and textual semantic features—was constructed.

4. The method for completing multimodal heterogeneous time-series data of tobacco fields based on a diffusion model according to claim 1, characterized in that, The multimodal heterogeneous time-series data completion network constructed based on the diffusion model is trained by selecting time-series data with completeness higher than a preset threshold from the multimodal heterogeneous time-series dataset to learn the joint distribution law of the multimodal data. Specifically, this includes: The missing rate of each sample in the structured environment data channel, visual feature channel and text semantic feature channel is statistically analyzed from the multimodal heterogeneous time series dataset. Samples with missing rates in all three channels below the preset threshold are marked as complete samples and selected as training sets. Continuous time series segments are extracted with preset step size and window length to form standard input samples. A diffusion model is constructed for structured environment data channels. The forward diffusion process is defined to gradually add Gaussian noise to the complete time segment. The total number of diffusion steps is set and the noise variance of each step is calculated. The reverse process constructs a noise prediction network with the goal of noise prediction. The input is the noise sample of the current step, the time code, and the multimodal conditional embedding. The standard input sample is input into the diffusion model to randomly sample the diffusion steps and calculate the noisy sample. Based on the noise prediction network, the network parameters are optimized by minimizing the mean square error between the predicted noise and the real noise. Cross-modal consistency constraints are introduced during the training process, and the hidden features of the visual feature channel and the text feature channel are fused as conditional embeddings through the attention mechanism. Independent diffusion models are constructed for the visual feature channel and the text feature channel respectively. The diffusion model for the visual feature channel uses structured environmental data and text features as conditional embeddings, while the diffusion model for the text feature channel uses structured environmental data and visual features as conditional embeddings, forming a joint training framework with mutual conditional constraints among the three channels. Each channel is trained independently using the same network architecture and loss function. By using an alternating training method, the diffusion models of the three channels converge synchronously. After training, each channel model learns its own data distribution patterns and cross-modal condition generation capabilities. The diffusion models of the three channels are then integrated to construct a multimodal heterogeneous time series data completion network.

5. The method for completing multimodal heterogeneous time-series data of tobacco fields based on a diffusion model according to claim 1, characterized in that, The process involves obtaining temporal representations of each modality using a Transformer model, introducing a contrastive learning mechanism to construct a cross-modal contrastive loss function, utilizing inter-modal correlation constraints and optimizing the training process of the diffusion model, and generating an optimized multimodal heterogeneous temporal data completion network. Specifically, this includes: All complete samples in the training set are input into the three-channel diffusion model that has been trained. The complete completion sequence of each channel is generated through the reverse sampling process, resulting in a set of multimodal completion sequences aligned in the time dimension for the structured environment data channel, visual feature channel and text feature channel. The multimodal completion sequence of each sample is input into the Transformer encoder model. The Transformer encoder includes a multi-head self-attention layer and a feedforward neural network. The sequence of each modality is position-encoded and then the temporal dependency is extracted through multi-head self-attention. The sequence is then aggregated by a global average pooling layer to obtain a structured environment data representation vector, a visual feature representation vector, and a text feature representation vector. A cross-modal contrastive loss function is constructed based on the representation vectors of each modality. The cosine similarity is calculated by treating different modal representations of the same sample as positive sample pairs and the cosine similarity is calculated by treating the same modal representations of different samples as negative sample pairs. A normalized contrastive loss term is formed by narrowing the distance between positive sample pairs and widening the distance between negative sample pairs. The contrastive loss function is added as an auxiliary constraint to the training objective of the diffusion model. In each iteration, the diffusion loss and contrastive loss of each channel are calculated simultaneously. The diffusion loss and contrastive loss are weighted and summed to obtain the total loss. The parameters of the diffusion model and the Transformer model are updated simultaneously through backpropagation. Through multiple rounds of iterative training, the parameters of the diffusion model are continuously optimized under the constraint of contrastive loss. The optimized three-channel diffusion model is then integrated with the Transformer model to generate an optimized multimodal heterogeneous time-series data completion network.

6. The method for completing multimodal heterogeneous time-series data of tobacco fields based on a diffusion model according to claim 1, characterized in that, The process of acquiring missing multi-source time-series data of tobacco fields, marking missing data segments as regions to be completed, and inputting them into the optimized multimodal heterogeneous time-series data completion network, generates missing data through a reverse diffusion process to obtain the completed multi-source time-series dataset of tobacco fields, specifically includes: Acquire multi-source time series data of tobacco fields with missing values, scan the time series of each sample to be completed, identify the time points where the values ​​are empty in the structured environment data channel, visual feature channel and text semantic feature channel and mark them as regions to be completed, and record the complete observation values ​​in each channel as conditional embedding to form an input data structure containing observation masks and missing masks. The input data structure is imported into the optimized multimodal heterogeneous temporal data completion network. Random samples are taken from the standard Gaussian distribution as initial noise samples. The initial noise samples, the time encoding of the current diffusion step, and the conditional embedding are used to complete the data in the common input noise prediction network. During the reverse diffusion process, the total number of diffusion steps decreases to 1. The noise prediction network outputs the predicted noise components based on the current noise sample, time encoding, and conditional embedding. A classifier-independent guidance mechanism is introduced to calculate the noise sample of the previous time step and combine the noise prediction values ​​under both conditional and unconditional settings to generate the degree of dependence of the result on the observation conditions. Repeat the reverse diffusion process until the diffusion step count reaches zero to obtain the completion sequence of the current channel. Use the completion sequence as a conditional embedding to assist in the generation of other channels. Iterate through the three channels in turn until the missing positions of all channels are filled. During the iterative completion process, the cross-modal semantic representation extracted by the Transformer model is used to constrain the generation process, ensuring that the completion results maintain semantic consistency across modalities. All the completed channel data are reorganized according to the original time order, and the completed tobacco field multi-source time series dataset is output.

7. A system for completing multimodal heterogeneous time-series data in tobacco fields based on a diffusion model, characterized in that, The system includes: a data acquisition and classification module, a data processing and construction module, a network completion and training module, a contrastive learning optimization module, and a data completion and application module. The data acquisition and classification module is used to acquire multimodal raw data from the tobacco field, collect production environment and biological characteristic information using sensors and imaging equipment, and classify and identify pests, diseases and weeds. The data processing and construction module is connected to the data acquisition and classification module, and is used to construct a multimodal heterogeneous time-series dataset covering structured data, visual data and text data; The completion network construction and training module is connected to the data processing and construction module, and is used to select time series segments with completeness higher than a preset threshold from the multimodal heterogeneous time series dataset as training samples to construct a multimodal heterogeneous time series data completion network; The contrastive learning optimization module is connected to the completion network construction and training module and is used to optimize the completion network for multimodal heterogeneous temporal data. The data completion and application module is connected to the comparative learning and optimization module, and is used to complete the missing multi-source time series data of tobacco fields to obtain the completed multi-source time series dataset of tobacco fields.

Images