A multi-modal data fusion method of a tea garden intelligent monitoring system

Abstract

The present application relates to the technical field of ecological environment monitoring and information intelligent processing, and particularly relates to a multi-modal data fusion method of a tea garden intelligent monitoring system, comprising: obtaining environment monitoring data, image data and text data in a tea garden area, cleaning, standardizing and processing the multi-modal data based on a unified time base and space position for time correspondence and space mapping; respectively modeling the time sequence of the environment monitoring data, the spatial features of the image data and the semantic features of the text data to obtain respective modal feature representations; aligning and jointly modeling the feature representations based on the time correlation, spatial correlation and semantic correlation between the respective modal features to obtain multi-modal fusion features; introducing tea garden related field knowledge on the basis of the fusion features for constraint analysis to obtain analysis result information, and outputting monitoring, prediction or early warning results accordingly, while updating the single modal modeling and multi-modal fusion process.

Description

Technical Field

[0001] This invention relates to the field of ecological environment monitoring and intelligent information processing technology, and in particular to a multimodal data fusion method for an intelligent monitoring system for tea gardens. Background Technology

[0002] Climate change has significantly impacted agricultural production. With the rapid development of agricultural informatization and intelligent technologies, real-time monitoring and intelligent early warning of natural disasters help improve agricultural disaster prevention and mitigation efficiency and the level of agricultural modernization. Therefore, environmental monitoring equipment, image acquisition equipment, and management information systems have been gradually introduced into traditional tea garden production processes to acquire multi-source data such as tea garden environmental parameters, tea tree growth images, and agricultural management records. Analysis of this data assists in tea garden production management and status monitoring. In existing technologies, environmental monitoring data is typically statistically analyzed, image data is processed for identification, and management text is recorded and queried separately. Different types of data are often collected and processed by independent systems, resulting in multi-source, multi-modal data. In existing technologies, different modal data exhibit significant differences in acquisition frequency, time scale, and spatial granularity, lacking a unified time benchmark and spatial correlation mechanism. This makes it difficult to establish an effective correspondence between environmental monitoring data, image data, and text data within the same computational framework, thereby limiting the joint modeling and comprehensive analysis capabilities of multimodal data. Summary of the Invention

[0003] To overcome the above shortcomings, this invention provides a multimodal data fusion method for a smart tea garden monitoring system, aiming to improve the problems in the existing technology where different modal data lack a unified time reference and spatial correlation mechanism, and it is difficult to establish an effective correspondence under the same computing framework.

[0004] This invention provides the following technical solution: a multimodal data fusion method for a tea garden intelligent monitoring system, comprising the following steps: S1. Acquire multimodal data in the tea garden area, including environmental monitoring data, image data, and text data; S2. The multimodal data are cleaned and standardized respectively, and based on a unified time reference and spatial location, the data of different modes are processed for time correspondence and spatial mapping. S3. For data of different modalities, perform time series modeling processing on environmental monitoring data, spatial feature modeling processing on image data, and semantic feature modeling processing on text data to obtain feature representations corresponding to each modality. S4. Based on the correlation between the feature representations of each modality, the feature representations are aligned and jointly modeled to obtain multimodal fusion features; S5. Based on the multimodal fusion features, introduce knowledge from the tea garden-related field to perform constraint analysis on the multimodal fusion features to obtain analysis results. S6. Output the corresponding monitoring, prediction or early warning results based on the analysis results, and update the single-modal modeling and multi-modal fusion processing based on the results.

[0005] By adopting the above technical solutions, the alignment and correlation of environmental monitoring data, image data and text data under a unified time reference and spatial location are realized, enabling different modal data to establish effective correspondence within the same computing framework, thereby supporting the joint modeling and comprehensive analysis of multimodal data.

[0006] Preferably, in S1, acquiring multimodal data in the tea garden area includes: Environmental monitoring equipment is deployed within the tea garden area to continuously monitor environmental elements and obtain environmental monitoring data; The growth status of tea trees is periodically collected using aerial or ground-based image acquisition equipment to obtain image data; Text data reflecting agricultural operations and management behaviors in tea gardens is collected through manual input or system recording. During the data acquisition process, the acquisition time and location information are recorded uniformly for different modal data.

[0007] Preferably, in S2, the cleaning and standardization processing of the multimodal data includes: The collected multimodal data is processed to identify and remove or mark data that does not conform to preset rules. Missing values ​​are processed for data with missing information to obtain complete multimodal data; Data from different modalities are normalized or standardized according to a unified data scale to obtain cleaned and standardized multimodal data.

[0008] Preferably, in S2, the time-correlation and spatial mapping processing of data from different modalities includes: According to the preset time granularity, the data of different modalities are subjected to time resampling processing, and the environmental monitoring data acquired at low frequency are completed by linear interpolation or Kalman filtering to align the timestamps with the image data acquired at high frequency, so that the data of different modalities correspond to the same time series. Data from different modalities are mapped to preset spatial units according to their acquisition locations, so that data corresponding to the same spatial unit can be associated. This generates multimodal structured data indexed by time series and spatial units.

[0009] Preferably, in S3, the time-series modeling processing of the environmental monitoring data includes: The environmental monitoring data are sorted according to a preset time order to construct a time series of environmental monitoring data; Based on time series, environmental monitoring data are segmented and processed according to preset time windows; We then perform feature modeling on the segmented time series data to obtain a time series feature representation.

[0010] Preferably, in S3, the spatial feature modeling process for the image data includes: The acquired image data is spatially divided into multiple spatial sub-regions. Based on spatial sub-regions, local feature extraction is performed on image data to obtain local feature information corresponding to each spatial sub-region; By performing convergence modeling on local feature information, an image feature representation is obtained; When the image data is hyperspectral image data, the local feature extraction includes using a three-dimensional convolutional network or residual network to perform spatial-spectral joint feature extraction, and introducing an attention mechanism to weight key bands.

[0011] Preferably, in S3, the semantic feature modeling process for the text data includes: The acquired text data is divided into text units, which are then divided into multiple semantic units. Based on semantic units, semantic representation modeling is performed on text data to obtain semantic feature information corresponding to each semantic unit; The semantic feature information is integrated and modeled to obtain the text feature representation.

[0012] Preferably, in S4, the alignment and joint modeling processing of the feature representation includes: The feature representations from different modalities are mapped to a unified feature representation space. Based on the temporal, spatial, and semantic relationships between different modal features in the feature representation space, an association structure is constructed. Under the constraints of the association structure, the feature representations of different modalities are jointly modeled to obtain multimodal fusion features.

[0013] Preferably, in S5, the process of incorporating tea garden-related domain knowledge to perform constraint analysis on multimodal fusion features includes: Construct a domain knowledge structure to describe the relationships between ecological elements in a tea garden. The domain knowledge structure includes the relationships between environmental elements, soil elements, tea tree growth elements, and management behavior elements in a tea garden. Based on the domain knowledge structure, knowledge constraints are generated to constrain multimodal fusion features and the fusion weights of different modal features; Under the constraints of knowledge, consistency verification and relational reasoning are performed on the multimodal fusion features to obtain the analysis results.

[0014] Preferably, in S6, updating the single-modal modeling and multi-modal fusion processes includes: Based on the analysis results, update parameters related to environmental monitoring data modeling, image data modeling, text data modeling, and multimodal joint modeling are determined. Based on the updated parameters, adjust the model parameters or processing rules in the single-modal modeling process; It also synchronously updates the feature alignment method, association structure, or joint modeling parameters during the multimodal fusion process.

[0015] The present invention has the following beneficial effects: 1. In this invention, by aligning environmental monitoring data, image data, and text data under a unified time reference and spatial location, and constructing temporal, spatial, and semantic relationships within a unified feature representation space, joint modeling of multimodal features under spatiotemporal consistency constraints is achieved. This overcomes the problem that different modal data are difficult to effectively integrate within the same computational framework, enabling the fused features to fully characterize the multidimensional state of the complex tea garden ecosystem.

[0016] 2. In this invention, a domain knowledge structure describing the relationship between tea garden environmental elements, soil elements, tea tree growth elements and management behavior elements is constructed based on multimodal fusion features. The domain knowledge is transformed into executable knowledge constraints, which guide the fusion features to perform consistency verification and relational reasoning under the constraints of the knowledge constraints. This achieves a deep integration of data-driven modeling results and tea garden domain knowledge, which is significantly different from multimodal analysis methods that rely solely on statistical models.

[0017] 3. In this invention, the analysis results obtained based on domain knowledge constraints are used in reverse to process single-modal modeling and multi-modal fusion, updating the model parameters, feature alignment and correlation structure, and constructing a closed-loop operation mechanism from multi-modal data acquisition, fusion analysis to model update. This enables the model to continuously adjust with changes in the tea garden environment and management behavior, forming an intelligent analysis method system suitable for long-term continuous monitoring. Attached Figure Description

[0018] Figure 1This is a flowchart of a multimodal data fusion method for a smart tea garden monitoring system proposed in this invention; Figure 2 This is a technical roadmap for a multimodal data fusion method in a tea garden intelligent monitoring system proposed in this invention. Figure 3 This is a flowchart illustrating the development of a single-modal model for a multimodal data fusion method in a tea garden intelligent monitoring system proposed in this invention. Figure 4 This is a flowchart illustrating the multimodal fusion and knowledge graph integration of a multimodal data fusion method for a tea garden intelligent monitoring system proposed in this invention. Detailed Implementation

[0019] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] In a first embodiment of the present invention, the present invention provides a multimodal data fusion method for a smart monitoring system for tea gardens, such as... Figures 1-4 As shown, it includes the following steps: S1. Acquire multimodal data in the tea garden area, including environmental monitoring data, image data, and text data; Furthermore, in S1, acquiring multimodal data in the tea garden area includes: Environmental monitoring equipment is deployed within the tea garden area to continuously monitor environmental elements and obtain environmental monitoring data; The growth status of tea trees is periodically collected using aerial or ground-based image acquisition equipment to obtain image data; Text data reflecting agricultural operations and management behaviors in tea gardens is collected through manual input or system recording. During the data acquisition process, the acquisition time and location information are recorded uniformly for different modal data.

[0021] Specifically, taking tea garden areas as the target, a multimodal raw data set of tea gardens is constructed through multi-source data collection methods, providing basic input for subsequent data preprocessing, multimodal modeling and fusion analysis; Environmental monitoring equipment is deployed within the tea garden area to continuously monitor environmental elements. Various types of sensors can be selected based on the specific conditions of the tea garden, such as sensors for collecting environmental parameters like air temperature, humidity, light intensity, rainfall, and soil moisture. During operation, each monitoring device automatically samples according to a preset collection cycle and outputs the sampling results as environmental monitoring data. When generating data, each piece of environmental monitoring data is appended with corresponding collection time and location information. The collection time information can be represented by a system timestamp, and the collection location information can be determined through device deployment location identifiers or geographic coordinates. During image data acquisition, a combination of aerial and ground-based image acquisition equipment can be used to obtain images of tea tree growth. The aerial or ground-based equipment includes hyperspectral imaging devices to acquire hyperspectral image data containing information across multiple spectral bands. Aerial image acquisition equipment can also be, for example, drones or high-altitude fixed cameras, to acquire image information of the tea garden area at a holistic scale. Ground-based image acquisition equipment can also be, for example, fixed cameras or mobile acquisition terminals, to acquire image information of individual tea trees or local areas at a local scale. The image acquisition process can be carried out periodically according to preset time intervals or tea tree growth stages to cover different stages of tea tree growth. When generating image data, the corresponding acquisition time and location information are recorded for each frame or group of image data, thereby ensuring the traceability of the image data in both time and space dimensions. During the text data collection process, text data reflecting agricultural operations and management behaviors in tea gardens are acquired through manual input or system recording. Manual input allows managers to enter agricultural operations such as fertilization, irrigation, pruning, and pest and disease control through terminal devices. System recording allows the tea garden management system to automatically generate operation logs or management records. When the text data is generated, the corresponding time information is also recorded, and the corresponding location information is associated with the area or object where the management behavior occurred, so that the text data can be linked with environmental monitoring data and image data in the time and space dimensions. To establish a unified time and space identification system during the data acquisition phase, this embodiment represents multimodal data as data units with time and space identifiers; for any modality of data sample, it can be represented as: ; in, Indicates the first A multimodal data sample, This represents the data content corresponding to the modality in environmental monitoring data. For environmental parameter vectors, in image data For image data matrices, in text data Representing text content, This indicates the time of data collection for this data sample. This indicates the location information of the data sample collection; through the above unified representation method, data of different modalities have consistent temporal and spatial dimension descriptions during the generation stage; During operation, environmental monitoring data, image data, and text data are continuously generated according to their respective acquisition methods and transmitted to the data management platform for centralized storage via a unified data interface; the data management platform then uses the recorded time stamps... and spatial signage This involves archiving and organizing multimodal data, enabling different modal data within the same time period and spatial region to establish corresponding relationships at the data level. Through the above-mentioned multimodal data acquisition and identification process, a multimodal data set containing environmental monitoring data, image data, and text data is formed. This data set has clear temporal order and spatial location information in its structure, and logically reflects the objective relationship between tea garden environmental changes, tea tree growth status, and management behavior. This provides basic data input for subsequent data cleaning, spatiotemporal alignment, multimodal feature modeling, and domain knowledge-based analysis and processing.

[0022] S2. Clean and standardize the multimodal data separately, and perform time correspondence and spatial mapping processing on the data of different modes based on a unified time reference and spatial location; Furthermore, in S2, the cleaning and standardization of multimodal data includes: The collected multimodal data is processed to identify and remove or mark data that does not conform to preset rules. Missing values ​​are processed for data with missing information to obtain complete multimodal data; Data from different modalities are normalized or standardized according to a unified data scale to obtain cleaned and standardized multimodal data.

[0023] Furthermore, in S2, the time-correlation and spatial mapping processing of data from different modalities includes: According to the preset time granularity, the data of different modalities are subjected to time resampling processing, and the environmental monitoring data acquired at low frequency are completed by linear interpolation or Kalman filtering to align the timestamps with the image data acquired at high frequency, so that the data of different modalities correspond to the same time series. Data from different modalities are mapped to preset spatial units according to their acquisition locations, so that data corresponding to the same spatial unit can be associated. This generates multimodal structured data indexed by time series and spatial units.

[0024] Specifically, after the multimodal data is collected, it enters the data cleaning, standardization and spatiotemporal alignment processing stage. This stage is used to perform unified data quality control and structured organization on environmental monitoring data, image data and text data, so that the data of different modalities have a consistent organization in the time and space dimensions, thereby providing a unified data foundation for subsequent single-modal feature modeling and multimodal fusion processing. During the data cleaning process, anomaly identification processing is performed on data of different modalities. Anomaly identification can be based on preset data value ranges, thresholds for changes in adjacent time points, and data integrity rules. When a data sample does not meet the preset rules, it can be marked as an anomaly or removed from subsequent processing to avoid anomaly data affecting the subsequent modeling process. After anomaly identification, missing value identification processing is performed on multimodal data. When data missing is detected at a certain time point or spatial location, missing data is filled or missing markers are retained to ensure the continuity of data at the time series organization level. In missing value completion processing, for numerical time series data in environmental monitoring data, interpolation can be used for completion; for example, when a certain environmental element is at time... and time There are valid sampled values ​​at each location. and And in The moment When a value is missing, linear interpolation can be used to estimate the missing value, and the completed value is expressed as follows: ; in, Indicates time The completion value, and These represent the valid sampled values ​​at adjacent time points, and Indicates the corresponding sampling time. This indicates the missing sampling time. When the missing time span is large or the continuous missing time period is long, the corresponding time period can be marked as an unusable interval and masked in the subsequent processing stage. For image data and text data, the consistency of the time series structure can be maintained by recording missing markers or setting placeholder information. In another alternative implementation, to improve the completion stability under extreme weather or sensor noise conditions, Kalman filtering can be used to filter and complete the environmental monitoring data. The environmental monitoring data is modeled as a state-space model, and a prediction-update mechanism is used to estimate the environmental monitoring values ​​at missing or abnormally fluctuating times, thereby obtaining environmental monitoring time series data aligned with high-frequency image data under a unified time reference. Through the above method, low-frequency acquired environmental monitoring data can establish a synchronous correspondence with instantaneously acquired or high-frequency acquired image data, eliminating the phase difference between multimodal data. After anomaly and missing data handling, standardization is performed on the multimodal data to eliminate differences in units and ranges between data from different sources. For numerical variables in environmental monitoring data, min-max normalization can be used, and the sample value of a certain environmental element is denoted as... The minimum and maximum values ​​in the sample set are respectively and The normalization result is then expressed as: ; in, This represents the normalized sample values; for image data, the numerical representation of the image input or before image feature extraction can be scaled; for text data, quantifiable fields such as text length and event count can be uniformly scaled, or the text data can be converted into a uniform input format required for subsequent semantic modeling; through the above processing, multimodal data after cleaning and standardization is obtained. After data cleaning and standardization, time-correlation and spatial mapping were performed on data from different modalities; in the time dimension, a unified time benchmark was constructed and a preset time granularity was set. Based on the start time Constructing a unified time series ,in, Represents a unified time series set. Indicates the first A time reference point, The value is a non-negative integer; different modal data are mapped to the corresponding time reference point according to their respective acquisition time information. When the original acquisition time is inconsistent with the time reference point, the time correspondence can be completed by methods such as preservation, interpolation or interval merging. In terms of spatial dimension, the tea garden area is spatially divided based on the collected location information, and a pre-defined set of spatial units is constructed. ,in, Represents a set of spatial units. Indicates the first Each spatial unit This indicates the number of spatial units. Different modal data are mapped to corresponding spatial units based on their acquisition location information. Environmental monitoring data can have its spatial unit number determined based on the equipment deployment location. Image data can have its spatial unit number determined based on the correspondence between the captured coverage area and the spatial unit. Text data can have its spatial unit number determined based on the plot number or work area associated in the record. When multiple pieces of data of the same modality exist at the same time reference point and in the same spatial unit, they can be merged or retained according to preset rules. Through the aforementioned time correspondence and spatial mapping processing, multimodal data is organized into a structured data form indexed by unified time series and spatial units. This establishes a clear correspondence between environmental monitoring data, image data, and text data in the time and spatial dimensions, thereby providing standardized and unified data input for subsequent single-modal feature modeling, cross-modal joint modeling, and domain knowledge-based analysis and processing.

[0025] S3. For data of different modalities, perform time series modeling processing on environmental monitoring data, spatial feature modeling processing on image data, and semantic feature modeling processing on text data to obtain feature representations corresponding to each modality. Furthermore, in S3, time series modeling of environmental monitoring data includes: The environmental monitoring data are sorted according to a preset time order to construct a time series of environmental monitoring data; Based on time series, environmental monitoring data are segmented and processed according to preset time windows; We then perform feature modeling on the segmented time series data to obtain a time series feature representation.

[0026] Furthermore, in S3, spatial feature modeling of image data includes: The acquired image data is spatially divided into multiple spatial sub-regions. Based on spatial sub-regions, local feature extraction is performed on image data to obtain local feature information corresponding to each spatial sub-region; By performing convergence modeling on local feature information, an image feature representation is obtained; When the image data is hyperspectral image data, local feature extraction includes using a three-dimensional convolutional network or residual network for joint spatial-spectral feature extraction, and introducing an attention mechanism to weight key bands.

[0027] Furthermore, in S3, semantic feature modeling of text data includes: The acquired text data is divided into text units, which are then divided into multiple semantic units. Based on semantic units, semantic representation modeling is performed on text data to obtain semantic feature information corresponding to each semantic unit; The semantic feature information is integrated and modeled to obtain the text feature representation.

[0028] Specifically, after the multimodal data has been cleaned, standardized and spatiotemporally aligned, it enters the single-modal feature modeling stage. In this stage, corresponding feature modeling processes are constructed for environmental monitoring data, image data and text data respectively, so as to convert the raw data of different modalities into feature representations that can characterize their inherent laws and structural information, and provide a unified feature input for subsequent multimodal joint modeling. In the time series modeling of environmental monitoring data, the data is first sorted according to a unified time benchmark, forming an ordered data sequence based on the collection time. This constructs a time series data reflecting the changes in the tea garden environment over time. After the time series is constructed, it is segmented according to a preset time window. The time window can be set according to the characteristics of environmental changes and modeling requirements, such as using a fixed-length window or a sliding window, so that each time series data corresponds to a continuous time segment. Through time window segmentation, the long time series is divided into multiple subsequences with local temporal continuity, which facilitates the subsequent extraction of environmental change trends and periodic features. When performing feature modeling on segmented time series data, the environmental monitoring data within each time window can be represented as a time series vector, and the time series within a certain time window can be denoted as... ,in, Indicates the first time within the time window Environmental monitoring data vectors corresponding to each time point This represents the length of the time window; based on this, the time series data is modeled to extract feature representations that characterize the patterns of time variation. The resulting time series features can be represented as: ; in, This represents the time series features corresponding to the time window. This represents a time series feature modeling function, which is used to model the changing trends, periodic information, and dependencies between adjacent time points in time series data; through the above processing, environmental monitoring data is converted into a time series feature representation to characterize the temporal changes in the environment; In the spatial feature modeling and processing of image data, the acquired image data is first divided into spatial regions, and a single image is divided into multiple spatial sub-regions. The division method of spatial sub-regions can be set according to the image resolution, tea tree distribution and modeling requirements, such as using regular grid division or division based on the target region. Through spatial region division, information from different spatial locations in the image is separated so as to facilitate the analysis of the growth state information contained in different regions of the image. After spatial region division, local feature extraction is performed on each sub-region to extract local feature information that reflects the growth status of tea trees within that region; let the set of spatial sub-regions obtained by dividing an image be denoted as . ,in, Indicates the first Each spatial sub-region Indicates the number of spatial sub-regions; for each spatial sub-region The corresponding local feature representation is extracted through the spatial feature modeling function, denoted as: ; in, Indicates the first Local feature representation of each spatial sub-region This represents the modeling function used to extract spatial features from an image; Furthermore, when the image data is hyperspectral image data, the hyperspectral image data contains spatial and spectral dimensions in its data structure, and can be represented as a spectral-spatial data cube. For this hyperspectral data cube, a three-dimensional convolutional neural network (3D-CNN) or a residual network (ResNet) can be used to perform joint spatial-spectral feature extraction to simultaneously capture the tea tree canopy texture information and the spectral response information of different bands. An attention mechanism can be introduced to weight key absorption bands, so that the model can highlight the subtle spectral differences that characterize the physiological stress, nutritional status, or occurrence of pests and diseases in tea trees, thereby obtaining a hyperspectral feature representation. After obtaining the local feature information of each spatial sub-region, the local feature information is processed by convergence modeling to integrate the local features of multiple spatial sub-regions into a unified image feature representation. The local feature convergence process can employ methods such as weighted summation, feature concatenation, or statistical convergence to... Mapped to a single image feature vector, the final image feature representation is denoted as... Through the above spatial feature modeling and convergence processing, the image data is transformed into an image feature representation that can reflect the spatial distribution and growth status of tea trees; In the semantic feature modeling of text data, the first step is to divide the acquired text data into text units, dividing a text record into multiple semantic units. Semantic units can be basic text components such as words, phrases, or sentences. The specific division method can be set according to the text content structure and semantic modeling requirements. Through text unit division, the text data has basic semantic units that can be processed in terms of structure. After dividing the text into units, semantic representation modeling is performed on each semantic unit to extract semantic features that reflect the meaning of the text and information about management behavior; let the set of semantic units obtained from dividing a certain text record be denoted as . ,in, Indicates the first A semantic unit, This represents the number of semantic units; each semantic unit is processed by a semantic feature modeling function to obtain the corresponding semantic feature representation, denoted as: ; in, Indicates the first Semantic feature representation of a semantic unit This represents a function used for semantic representation modeling; After obtaining the semantic feature information of each semantic unit, the semantic feature information is integrated and modeled to integrate the semantic features of multiple semantic units into a unified text feature representation. The semantic feature integration process can use weighted combination or serialization modeling based on the order or importance relationship between semantic units, and finally obtain the text feature representation used to represent the semantic information of the entire text record, denoted as . Through the above semantic feature modeling and integration processing, the text data is transformed into a text feature representation that can reflect the semantic information of tea garden management behavior and agricultural operations. By performing time series modeling, spatial feature modeling, and semantic feature modeling on environmental monitoring data, image data, and text data respectively, corresponding time series feature representations, image feature representations, and text feature representations are obtained. While maintaining their respective data characteristics, each modal feature forms a feature representation with a clear structure and controllable dimensions, thus providing a unified and standardized feature input foundation for subsequent multimodal feature alignment, joint modeling, and domain knowledge-based analysis and processing.

[0029] S4. Based on the correlation between the feature representations of each modality, the feature representations are aligned and jointly modeled to obtain multimodal fusion features; Furthermore, in S4, the alignment and joint modeling of feature representations includes: The feature representations from different modalities are mapped to a unified feature representation space. Based on the temporal, spatial, and semantic relationships between different modal features in the feature representation space, an association structure is constructed. Under the constraints of the association structure, the feature representations of different modalities are jointly modeled to obtain multimodal fusion features.

[0030] Specifically, after completing the single-modal feature modeling of environmental monitoring data, image data, and text data, the process moves into the multimodal feature alignment and joint modeling stage. This stage takes the feature representations of each modality as input, and through representation space mapping, association structure construction, and joint modeling processing under association constraints, it establishes a computable association relationship between the features of different modalities in a unified representation space and forms multimodal fusion features for subsequent constraint analysis and application output. In the representation space mapping process, corresponding mapping functions are set to address the inconsistencies in dimensionality and distribution differences of features across different modalities, mapping the feature representations of different modalities to a unified feature representation space. In one possible implementation, the time series feature representation is denoted as... Image feature representation is denoted as Text feature representation is denoted as The dimension of the unified feature representation space is denoted as ; construct linear mappings respectively to obtain a unified spatial feature vector, denoted as; ; in, These represent time-series features, image features, and text features, respectively, mapped to a unified feature representation space. , , This is the mapping matrix for the corresponding modes. This is the bias vector for the corresponding mode; in another possible implementation, the mapping function can also be a nonlinear mapping structure, used to obtain a more stable feature representation in a unified representation space; In constructing the association structure, based on the temporal, spatial, and semantic relationships between different modal features within a unified feature representation space, an association structure is formed to describe the interaction relationships of multimodal features. In one possible implementation, the association structure is represented as a graph structure. , where the set of nodes Used to hold feature vectors and edge sets within a unified representation space. Used to carry the association relationships between nodes; index at any given time. Spatial unit index Based on the following features, construct nodes respectively. , , and with , , As the node features of the corresponding nodes; when constructing edge relationships, cross-node connections can be generated through temporal proximity, spatial proximity and semantic consistency rules, and each edge can be assigned a weight to quantify the association strength; In edge weight calculation, to simultaneously reflect temporal, spatial, and semantic relationships, a weighted combination approach can be optionally used to define edge weights; for nodes... With nodes Edge weights between , can be represented as; ; in, The weight coefficients are non-negative and satisfy the following conditions: , Represents the time correlation function. Represents the spatial correlation function. This represents a semantic relevance function; in one possible implementation, It can be determined based on the time index difference of the data corresponding to the node. It can be determined based on the adjacency relationship of the spatial units corresponding to the nodes. The correlation function can be determined based on the degree of matching between text event labels and image or environmental state labels. It can be implemented using rule-based judgment or similarity calculation. Through the above methods, a correlation structure containing cross-modal correlation information is obtained, which is used to constrain the scope of information transmission and fusion in the subsequent joint modeling stage. In the joint modeling process under the constraints of the associated structure, different modal features are jointly modeled to generate multimodal fusion features based on the adjacency relationships and edge weights defined by the associated structure. In one possible implementation, iterative message passing modeling is performed on the graph structure, updating the node features layer by layer from the initial representation, denoted as the... Layer Time Node The representation of is The initial layer is set to Node updates can be represented as: ; in, Represents nodes in an association structure The set of neighboring nodes, Representing neighboring nodes To the node edge weights, and For the first The parameter matrix of the layer, It is a nonlinear activation function; through the above joint modeling and updating, cross-modal information can interact and aggregate within the scope of the association structure, thereby forming a fusion representation containing temporal, spatial and semantic association information in a unified feature representation space; When outputting multimodal fusion features, the same time index can be optionally applied. Spatial unit index The multi-node representations are converged to obtain the corresponding multimodal fusion feature vector; in one possible implementation, the updated three modal node representations are... , , By splicing or weighted fusion, fusion features are formed. ,in Indicates the number of iterations in the joint modeling. Represents the dimension of the fused features, fused features As input to the subsequent domain knowledge constraint analysis and application output module; through the above alignment and joint modeling processing, the correlation structure constraint fusion of different modal features under a unified representation space is realized, and multimodal fusion features that can characterize the overall state of the tea garden are obtained.

[0031] S5. Based on the multimodal fusion features, knowledge from the tea garden-related field is introduced to perform constraint analysis on the multimodal fusion features to obtain the analysis results. Furthermore, in S5, knowledge from the tea garden domain is introduced to perform constraint analysis on the multimodal fusion features, including: Construct a domain knowledge structure to describe the relationships between ecological elements in a tea garden. The domain knowledge structure includes the relationships between environmental elements, soil elements, tea tree growth elements, and management behavior elements in a tea garden. Based on the domain knowledge structure, knowledge constraints are generated to constrain multimodal fusion features and the fusion weights of different modal features; Under the constraints of knowledge, consistency verification and relational reasoning are performed on the multimodal fusion features to obtain the analysis results.

[0032] Specifically, after the multimodal fusion features are aligned and jointly modeled, they enter the domain knowledge-based constraint analysis stage. This stage takes the multimodal fusion features as input, constructs a domain knowledge structure and generates knowledge constraints, and performs consistency verification and relational reasoning on the multimodal fusion features under the constraints of the knowledge constraints, thereby outputting analysis results information used to characterize the state analysis of the tea garden. In the process of constructing the domain knowledge structure, a computable domain knowledge structure is built based on the objective relationships among environmental elements, soil elements, tea tree growth elements, and management behavior elements in the tea garden ecosystem. In one possible implementation, the domain knowledge structure is represented as a knowledge graph structure. ,in Represents a set of entities. Represents a set of relations; a set of entities. It can include four types of entities: environmental entities, soil entities, growth entities, and management entities. Environmental entities include temperature, humidity, light, and rainfall; soil entities include soil moisture content and soil nutrient indicators; growth entities include growth indicators, leaf color indicators, or pest and disease status indicators; and management entities include fertilization, irrigation, pruning, or pest control operations. (Relationship set) This is used to describe the relationships between entities. The types of relationships can include causal relationships, conditional relationships, temporal relationships, or spatial relationships. Entities and relationships can be organized and solidified into structured knowledge through domain experience rules, historical management records, or expert annotation information. During operation, the domain knowledge structure can be stored in a database or graph data storage structure and supports retrieval and updating by entity and relationship. When generating knowledge constraints based on domain knowledge structure, the relationships within the domain knowledge structure are transformed into an executable constraint expression for multimodal fusion features. In one possible implementation, the multimodal fusion features are indexed in time. Spatial unit index The fusion features are represented as follows And extract the state variables that need to be constrained as characteristic functions. ,in This represents a mapping function for extracting the target variable from the fused features. This represents the state variable vector derived from the fused features; based on this, the relational rules in the domain knowledge structure are mapped to a set of constraints. ,in Indicates the first Knowledge constraint functions are used to limit the logical consistency, numerical range, or association conditions between state variables. In order to make the constraints quantifiable during the calculation process, a constraint violation measure can be optionally defined for each constraint function, forming a constraint cost function. In one optional implementation, the knowledge constraints also include prior rules for constraining the confidence or fusion weights of different modal features. When extreme weather conditions or sensor anomalies are detected, the fusion weights of the corresponding modal features are adjusted according to the prior rules to avoid distortion of the fusion results due to abnormal data. For example, when rainfall exceeds a preset threshold, visual modal image data may be affected by rain and fog, which can reduce the weight or confidence of visual modal features in the multimodal fusion process, thereby improving the robustness of monitoring, prediction, or early warning output results. In the consistency verification process, the state variables derived from the fused features are judged for constraint satisfaction based on knowledge constraints; in one possible implementation, this is done for the constraint set. Construct the total constraint cost function The degree to which the current state variable violates the knowledge constraints is represented as: ; in, Indicates the number of knowledge constraints. Indicates the first The weighting coefficients of the knowledge constraints. Indicates the measurement of the first A non-negative function of the degree of constraint violation, and when the constraint is satisfied... ;when When the threshold is exceeded, the fusion features under the corresponding time index and spatial unit index can be marked as having consistency risks, and the triggered constraint entries can be recorded as constraint conflict information to facilitate subsequent reasoning and result interpretation. In relational reasoning, based on entity relationships and constraints within the domain knowledge structure, relational deduction is performed on state variables derived from fused features to obtain supplementary analytical information. In one possible implementation, the reasoning process is represented as starting from a set of known facts. To the set of inference conclusions The rule derivation process, in which the set of known facts... It can be determined by state variables. The discretized results or event labels constitute the inference rules, which are derived from the relation set. The corresponding rule expression; the reasoning process can adopt the forward chain reasoning method, that is, starting from known facts, iteratively matching the rule antecedents and generating new conclusion facts until no rules are triggered or the preset reasoning depth is reached; during the reasoning process, the generated reasoning conclusions can include deductive information related to the relationship between environmental conditions, soil conditions, tea tree growth status and management behavior, and together with the conflict entries of consistency verification, constitute the analysis result information; When outputting analysis results, the results can be organized into a structured output containing time indexes, spatial unit indexes, fused state variables, constraint verification results, and inference conclusions, for subsequent monitoring, prediction, or early warning modules to use. In one possible implementation, the analysis results can be represented as follows: ,in Indicates a time index. Indicates spatial cell index, Represents a vector of state variables. Represents the cost of consistency. This represents the set of reasoning conclusions. Through the above method, a constraint analysis and processing flow based on domain knowledge structure is realized, enabling multimodal fusion features to complete consistency verification and relational reasoning under knowledge constraints, and output analysis results information that can be used for subsequent application processing.

[0033] S6. Output the corresponding monitoring, prediction or early warning results based on the analysis results, and update the single-modal modeling and multi-modal fusion processing based on the results.

[0034] Furthermore, in S6, updates to the single-modal modeling and multi-modal fusion processes include: Based on the analysis results, update parameters related to environmental monitoring data modeling, image data modeling, text data modeling, and multimodal joint modeling are determined. Based on the updated parameters, adjust the model parameters or processing rules in the single-modal modeling process; It also synchronously updates the feature alignment method, association structure, or joint modeling parameters during the multimodal fusion process.

[0035] Specifically, after the domain knowledge constraint analysis is completed, the analysis results enter the application output and model update stage. In this stage, monitoring, prediction or early warning results are generated based on the analysis results and output to the application. At the same time, the results and feedback information generated during the application output process are used as the basis for updates to adjust the parameters and rules of the single-modal modeling process and the multi-modal fusion process, so that the modeling and fusion process can maintain consistency with the actual data distribution and business scenarios during continuous operation. In the process of generating and outputting monitoring, prediction, or early warning results, time indexing is used. Spatial unit index The analysis results are used to form a structured output for the application; in one possible implementation, the analysis results are denoted as... The application output module is based on The state variables, constraint costs, and inference conclusions in the process generate corresponding application results. ,in It can include monitoring result fields, prediction result fields, or early warning result fields; the monitoring result field reflects the current value of the state variable and its corresponding spatial location; the prediction result field outputs the estimated value of the state variable under the future time series index; and the early warning result field outputs the early warning identifier and triggering basis that meet the preset triggering conditions; the application output module can... Data is written to the data management platform, sent to the management terminal, or used to drive the interface display of the tea garden management system; In determining the update parameters, the update requirements for the single-modal modeling process and the multi-modal fusion process are quantitatively described based on the analysis results. In one possible implementation, the set of update parameters is denoted as... ,in This indicates the update parameters related to environmental monitoring data modeling. This represents the update parameters related to image data modeling. This represents the update parameters related to text data modeling. This represents the update parameters related to multimodal joint modeling. The update parameters may include the update amount of trainable parameters of the model, the threshold update amount of the processing rules, the selection parameters of the feature alignment method, and the update amount of the weight coefficients of the association structure. The update parameters can be determined based on the constraint costs, conflict item statistics, and the deviation between the inference conclusions and the actual observations in the analysis results. In the update process of single-modal modeling, the model parameters or processing rules corresponding to environmental monitoring data modeling, image data modeling, and text data modeling are adjusted respectively; in one possible implementation, the environmental monitoring data modeling model parameters are denoted as... The parameters of the image modeling model are The parameters of the text semantic modeling model are When the analysis results indicate that a certain type of input data has many constraint conflicts under a specific time period or spatial unit, the corresponding single-modal model parameters can be updated. Parameter updates can be performed iteratively, denoted as the... The model parameters at the next update are: and update step size Adjusting the parameters is represented as follows: ; in, This represents the set of parameters for the single-modal model to be updated. Indicates the update step size. Represents the loss function Gradient of model parameters; loss function It can be formed by combining the deviation metric between the application output and the actual observation with the knowledge constraint violation cost term, so that the single-modal model can consider both data fitting and knowledge constraint requirements during the update process; when the processing rules are used for adjustment, the anomaly identification threshold, missing completion strategy parameters or data screening conditions can be optionally updated to match the statistical characteristics of the newly collected data. In the synchronous update of the multimodal fusion process, adjustments are made to the feature alignment method, association structure, or joint modeling parameters. In one possible implementation, the update of the feature alignment method may include the parameter update of the unified representation space mapping matrix, denoted as the mapping matrix parameter set. ,but The parameters can be included in the update process along with those of the unimodal model; the update of the associated structure can include the edge weight coefficients. Alternatively, the edge weight calculation rules can be adjusted to reflect changes in temporal, spatial, and semantic relationships across different seasons or different land parcels; updates to the joint modeling parameters may include the message passing model parameter matrix. and Adjustments to configuration parameters such as update or update iteration layer number; synchronized updates to feature alignment method, association structure, and joint modeling parameters to ensure that the parameter evolution trajectory of the multimodal fusion feature generation process is consistent with that of the single-modal modeling process; In terms of the organization of model updates, the update process can be executed according to a preset cycle or when a trigger condition is met. The trigger condition may include constraint cost exceeding a threshold, warning trigger frequency exceeding a threshold, or application feedback indicating that the output deviation has reached a preset range. After the update is completed, the new parameters are written into the model management module and used in the next round of data processing and inference, thus forming a closed-loop operation process between application output and model update.

[0036] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal data fusion method for a smart tea garden monitoring system, characterized in that, Includes the following steps: S1. Acquire multimodal data in the tea garden area, including environmental monitoring data, image data, and text data; S2. The multimodal data are cleaned and standardized respectively, and based on a unified time reference and spatial location, the data of different modes are processed for time correspondence and spatial mapping. S3. For data of different modalities, perform time series modeling processing on environmental monitoring data, spatial feature modeling processing on image data, and semantic feature modeling processing on text data to obtain feature representations corresponding to each modality. S4. Based on the correlation between the feature representations of each modality, the feature representations are aligned and jointly modeled to obtain multimodal fusion features; S5. Based on the multimodal fusion features, introduce knowledge from the tea garden-related field to perform constraint analysis on the multimodal fusion features to obtain analysis results. S6. Output the corresponding monitoring, prediction or early warning results based on the analysis results, and update the single-modal modeling and multi-modal fusion processing based on the results.

2. The multimodal data fusion method for a tea garden intelligent monitoring system according to claim 1, characterized in that, In S1, acquiring multimodal data in the tea garden area includes: Environmental monitoring equipment is deployed within the tea garden area to continuously monitor environmental elements and obtain environmental monitoring data; The growth status of tea trees is periodically collected using aerial or ground-based image acquisition equipment to obtain image data; Text data reflecting agricultural operations and management behaviors in tea gardens is collected through manual input or system recording. During the data acquisition process, the acquisition time and location information are recorded uniformly for different modal data.

3. The multimodal data fusion method for a tea garden intelligent monitoring system according to claim 1, characterized in that, In S2, the cleaning and standardization of the multimodal data includes: The collected multimodal data is processed to identify and remove or mark data that does not conform to preset rules. Missing values ​​are processed for data with missing information to obtain complete multimodal data; Data from different modalities are normalized or standardized according to a unified data scale to obtain cleaned and standardized multimodal data.

4. The multimodal data fusion method for a tea garden intelligent monitoring system according to claim 1, characterized in that, In S2, the time-correlation and spatial mapping processing of data from different modalities includes: According to the preset time granularity, the data of different modalities are resampled in time, and the environmental monitoring data acquired at low frequency are completed by linear interpolation or Kalman filtering to align the timestamps with the image data acquired at high frequency, so that the data of different modalities correspond to the same time series. Data from different modalities are mapped to preset spatial units according to their acquisition locations, so that data corresponding to the same spatial unit can be associated. This generates multimodal structured data indexed by time series and spatial units.

5. The multimodal data fusion method for a tea garden intelligent monitoring system according to claim 1, characterized in that, In S3, the time series modeling processing of environmental monitoring data includes: The environmental monitoring data are sorted according to a preset time order to construct a time series of environmental monitoring data; Based on time series, environmental monitoring data are segmented and processed according to preset time windows; We then perform feature modeling on the segmented time series data to obtain a time series feature representation.

6. The multimodal data fusion method for a tea garden intelligent monitoring system according to claim 1, characterized in that, In S3, the spatial feature modeling process for the image data includes: The acquired image data is spatially divided into multiple spatial sub-regions. Based on spatial sub-regions, local feature extraction is performed on image data to obtain local feature information corresponding to each spatial sub-region; By performing convergence modeling on local feature information, an image feature representation is obtained; When the image data is hyperspectral image data, the local feature extraction includes using a three-dimensional convolutional network or residual network to perform spatial-spectral joint feature extraction, and introducing an attention mechanism to weight key bands.

7. The multimodal data fusion method for a tea garden intelligent monitoring system according to claim 1, characterized in that, In S3, the semantic feature modeling process for the text data includes: The acquired text data is divided into text units, which are then divided into multiple semantic units. Based on semantic units, semantic representation modeling is performed on text data to obtain semantic feature information corresponding to each semantic unit; The semantic feature information is integrated and modeled to obtain the text feature representation.

8. The multimodal data fusion method for a tea garden intelligent monitoring system according to claim 1, characterized in that, In S4, the alignment and joint modeling of the feature representation includes: The feature representations from different modalities are mapped to a unified feature representation space. Based on the temporal, spatial, and semantic relationships between different modal features in the feature representation space, an association structure is constructed. Under the constraints of the association structure, the feature representations of different modalities are jointly modeled to obtain multimodal fusion features.

9. The multimodal data fusion method for a tea garden intelligent monitoring system according to claim 1, characterized in that, In S5, the process of incorporating tea garden-related domain knowledge to perform constraint analysis on multimodal fusion features includes: Construct a domain knowledge structure to describe the relationships between ecological elements in a tea garden. The domain knowledge structure includes the relationships between environmental elements, soil elements, tea tree growth elements, and management behavior elements in a tea garden. Based on the domain knowledge structure, knowledge constraints are generated to constrain multimodal fusion features and the fusion weights of different modal features; Under the constraints of knowledge, consistency verification and relational reasoning are performed on the multimodal fusion features to obtain the analysis results.

10. The multimodal data fusion method for a tea garden intelligent monitoring system according to claim 1, characterized in that, In S6, updating the single-modal modeling and multi-modal fusion processes includes: Based on the analysis results, update parameters related to environmental monitoring data modeling, image data modeling, text data modeling, and multimodal joint modeling are determined. Based on the updated parameters, adjust the model parameters or processing rules in the single-modal modeling process; It also synchronously updates the feature alignment method, association structure, or joint modeling parameters during the multimodal fusion process.

Images