Brassica oleracea knowledge dynamic expression and interaction method and system based on multi-modal fusion

Abstract

The invention provides a cabbage knowledge dynamic expression and interaction method and system based on multi-modal fusion, and relates to the technical field of agricultural planting management, and the method comprises the steps: obtaining multi-modal data of cabbage planting management, carrying out the feature extraction of the multi-modal data, obtaining multi-modal features, and carrying out the feature alignment to obtain final fusion features; constructing a dynamic knowledge graph of cabbage planting management based on the final fusion features; and for the to-be-processed cabbage planting interaction data, calling the dynamic knowledge graph to retrieve the cabbage planting interaction data to obtain a corresponding cabbage planting feedback result. According to the application, accurate dynamic management and decision support in the cabbage planting process are realized by realizing the processes of multi-modal data acquisition and fusion, dynamic knowledge graph construction, intelligent interaction function and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of agricultural planting management, and in particular to a cabbage knowledge dynamic expression and interaction method and system based on multimodal fusion. Background Art

[0002] As an important economic vegetable, cabbage cultivation management covers multiple key links, including growth monitoring, environmental regulation, pest and disease control, and fertilization and irrigation optimization. However, traditional agricultural knowledge management systems usually present knowledge in a single form of text or tables, lacking the ability to effectively integrate multimodal information in actual field scenarios, resulting in significant limitations in practical applications.

[0003] On the one hand, the construction method of static knowledge base is difficult to reflect the dynamic changes of field environment in real time, and it is impossible to intelligently adjust management strategies according to real-time sensor data or user feedback, resulting in a lack of pertinence in guidance opinions; on the other hand, these systems lack efficient problem reasoning and decision-making capabilities, and cannot solve cross-domain or unstructured problems, such as identification and prevention and control suggestions for new diseases, which affects the decision-making efficiency of farmers. In addition, the traditional interaction method is also relatively simple, mostly presented in a fixed format retrieval interface, which is difficult to meet the needs of users for flexible questions, and lacks intuitiveness and interactivity, especially in multimodal interaction functions such as image recognition and voice input. There is a significant gap. Summary of the invention

[0004] The present invention provides a method and system for dynamic expression and interaction of cabbage knowledge based on multimodal fusion, which realizes accurate dynamic management and decision support of the cabbage planting process through multimodal data collection and fusion, dynamic knowledge graph construction and intelligent interaction functions.

[0005] The present invention provides a cabbage knowledge dynamic expression and interaction method based on multimodal fusion, comprising: Acquire multimodal data of cabbage planting management, wherein the multimodal data includes image data of cabbage growth, environmental time series data, and annotated text data of cabbage planting management; Extracting features from the multimodal data to obtain multimodal features, and calling a cross-modal Transformer model to perform feature alignment on the multimodal features to obtain final fusion features; Constructing a dynamic knowledge graph for cabbage planting management based on the final fusion features; For the cabbage planting interaction data to be processed, the dynamic knowledge graph is called to search the cabbage planting interaction data to obtain corresponding cabbage planting feedback results.

[0006] In some embodiments, the multimodal features include image features of the image data, time series features of the environmental time series data, and text features of the annotated text data, and extracting features from the multimodal data to obtain the multimodal features includes: Calling the VisionTransformer model to perform a disease classification task and a leaf health state estimation task on the image data to obtain image features of the image data, wherein the disease classification task is used to classify cabbage diseases on the image data, and the leaf health state estimation task is used to estimate the cabbage leaf health state on the image data, and the VisionTransformer model is trained by disease classification loss and state estimation loss; Calling a causal attention-based temporal Transformer model to extract features from the environmental temporal data to obtain temporal features; Calling the Tokenizer model to encode the annotated text data into a Token ID sequence; Obtain prompt words of specific domain knowledge, and input the prompt words and the Token ID sequence into the RoBERTa model to extract contextual semantic features to obtain text features of the annotated text data.

[0007] In some embodiments, calling the cross-modal Transformer model to perform feature alignment on the multi-modal features to obtain the final fusion features includes: The cross-modal attention mechanism of the cross-modal Transformer model is used to perform inter-modal interaction calculation on the multi-modal features, thereby obtaining image-time series interaction features, time series-text interaction features, and image-text interaction features respectively; Fusing the image-time sequence interaction feature, the time sequence-text interaction feature and the image-text interaction feature to obtain a joint feature representation; The joint feature representation is subjected to feature dimension transformation processing to obtain a final fusion feature.

[0008] In some embodiments, the cross-modal Transformer model is trained by constructing an overall loss function, and the construction process of the overall loss function includes: Calculating feature similarity between modalities for the multimodal features, and constructing modality contrast loss based on the feature similarity, wherein the modality contrast loss includes image-text modality contrast loss, image-time series modality contrast loss, and time series-text modality contrast loss; Reconstructing the final fused features through a decoder of the cross-modal Transformer model to obtain feature representations of each modality; Based on the feature representation of each modality and the multimodal feature, construct a modality reconstruction loss; The modality contrast loss and the modality reconstruction loss are weighted respectively by preset contrast weights and reconstruction weights, and the weighted results are summed to obtain an overall loss function.

[0009] In some embodiments, constructing a dynamic knowledge graph for cabbage planting management based on the final fusion features includes: Performing data alignment on the multimodal data of cabbage planting management and the final fusion feature to obtain structured multimodal input data; Perform entity extraction on multimodal features to obtain corresponding entity nodes and form an entity set; Performing relationship extraction on the multimodal input data to obtain the relationship between the entity nodes and form a relationship set; Construct triples based on the entity set and the relationship set to obtain a primary knowledge graph; The primary knowledge graph is dynamically updated to obtain a dynamic knowledge graph for cabbage planting management.

[0010] In some embodiments, the dynamic update process of the primary knowledge graph includes: Acquire real-time environmental data during the cabbage planting process, and aggregate the real-time environmental data through a time window to obtain window environmental features; Determine a feature change rate of the window environment feature between time windows, and when the feature change rate is greater than a feature change threshold, trigger an update operation on the primary knowledge graph; The updating operation on the primary knowledge graph includes: According to the window environment characteristics, a relationship change matrix before and after the update is constructed; Within the time window, the relationship change matrix is ​​merged with the original relationship set in the primary knowledge graph to obtain a new relationship set.

[0011] In some embodiments, after constructing the relationship change matrix before and after the update, the method further includes: Determine the feature similarity between the relationship set before updating and the relationship set after updating; Respectively determining a mutual exclusion conflict score, an attribute conflict score, and an inference conflict score between the relationship set before the update and the relationship set after the update; Add the mutual exclusion conflict score, the attribute conflict score and the reasoning conflict score to obtain a conflict degree score; The difference between the feature similarity and the conflict degree score is used as an update conflict score, and when the update conflict score is greater than a preset conflict threshold, it is determined that there is a conflict between the relationship set before the update and the relationship set after the update; The relationship weights in the relationship set are adjusted according to the conflict degree scores to correct the conflicts between the relationship set before the update and the relationship set after the update.

[0012] In some embodiments, the method further comprises: Determine the feature similarity between any two entity nodes in the primary knowledge graph, and when the feature similarity is greater than a preset similarity threshold, merge the two entity nodes; Extract entity nodes and edges in the dynamic knowledge graph, call the graph convolutional neural network to predict the relationship between the entity nodes and the edges, and obtain predicted nodes and predicted relationships; The predicted nodes and predicted relationships are merged with the entity set and relationship set in the dynamic knowledge graph.

[0013] In some embodiments, for the cabbage planting interaction data to be processed, calling the dynamic knowledge graph to search the cabbage planting interaction data to obtain the corresponding cabbage planting feedback result includes: When the cabbage planting interactive data to be processed is a cabbage image that needs to be diagnosed for a disease, the VisionTransformer model is called to extract the disease features of the cabbage image, and the corresponding disease node is retrieved in the dynamic knowledge graph according to the disease features; Performing knowledge reasoning on the disease node through the dynamic knowledge graph to obtain a disease diagnosis result of the cabbage image as a cabbage planting feedback result; When the cabbage planting interaction data to be processed is a prompt word for consulting cabbage planting, calling the dynamic knowledge graph to perform knowledge reasoning on the prompt word to obtain cabbage planting data as a cabbage planting feedback result; When the to-be-processed cabbage planting interaction data represents the personalized recommendation for executing cabbage planting, the dynamic knowledge graph is called to perform knowledge reasoning on the cabbage historical planting data to obtain the recommended content of cabbage planting as the cabbage planting feedback result.

[0014] The present invention also provides a cabbage knowledge dynamic expression and interaction system based on multimodal fusion, comprising: A feature extraction module, used to obtain multimodal data of cabbage planting management, wherein the multimodal data includes image data of cabbage growth, environmental time series data, and annotated text data of cabbage planting management; A feature fusion module is used to extract features from the multimodal data to obtain multimodal features, and call a cross-modal Transformer model to perform feature alignment on the multimodal features to obtain final fusion features; A graph construction module, used to construct a dynamic knowledge graph for cabbage planting management based on the final fusion features; The dynamic interaction module is used to call the dynamic knowledge graph to search the cabbage planting interaction data to be processed, and obtain the corresponding cabbage planting feedback result.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for dynamically expressing and interacting with cabbage knowledge based on multimodal fusion as described above is implemented.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for dynamically expressing and interacting with cabbage knowledge based on multimodal fusion.

[0017] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for dynamically expressing and interacting with cabbage knowledge based on multimodal fusion.

[0018] The method and system for dynamic expression and interaction of cabbage knowledge based on multimodal fusion provided by the present invention extracts multimodal features of multimodal data of cabbage planting management, aligns the features, obtains the final fusion features, and then constructs a dynamic knowledge graph of cabbage planting management based on the final fusion features. In this way, the multimodal data is uniformly embedded into the dynamic knowledge graph framework to improve the accuracy of cabbage planting knowledge expression. In addition, the present invention also provides an interactive function, which can use the dynamic knowledge graph to search for cabbage planting interactive data for feedback, breaking the limitations of the single text or table output of traditional planting guidance, and providing farmers with more intuitive and efficient decision support. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1It is a flow chart of the cabbage knowledge dynamic expression and interaction method based on multimodal fusion provided by the present invention.

[0021] Figure 2 It is a principle framework diagram of the cabbage knowledge dynamic expression and interaction method based on multimodal fusion provided by the present invention.

[0022] Figure 3 It is a structural schematic diagram of the cabbage knowledge dynamic expression and interaction system based on multimodal fusion provided by the present invention.

[0023] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The following describes the cabbage knowledge dynamic expression and interaction method and system based on multimodal fusion of the present invention in conjunction with the accompanying drawings. Figure 1 is a flow chart of the cabbage knowledge dynamic expression and interaction method based on multimodal fusion provided by the present invention, such as Figure 1 As shown, the method includes the following steps 101 to 104, which are described in detail below.

[0026] Step 101: Acquire multimodal data of cabbage planting management.

[0027] In the process of cabbage planting and management, the cabbage growth status and environment are generally monitored in real time through sensor equipment, and the data is collected and uploaded. Here, the uploaded multimodal data of cabbage planting and management is obtained. The multimodal data includes image data of cabbage growth, environmental time series data, and annotated text data of cabbage planting and management.

[0028] The image data for cabbage growth includes the growth status of cabbage leaves, disease characteristics (such as the shape and color of downy mildew spots), and soil surface conditions. The data format is JPEG and PNG. During the data collection process, the image collection frequency is dynamically adjusted according to the cabbage growth cycle, such as collecting images once a week during the sowing period and collecting images every day during the high incidence period of diseases.

[0029] The environmental time series data includes the temperature, humidity, light intensity, soil moisture, The data is in a standardized time-series stream format (such as JSON format). The data is updated automatically every 15 minutes to ensure real-time performance.

[0030] The annotated text data for cabbage planting and management is mainly used as expert data, and its data content includes descriptions of cabbage diseases, prevention and control recommendations, and planting and management knowledge. The data is characterized by its multimodal data pairing of images and texts, which is used to build a high-quality basic knowledge graph.

[0031] Step 102: extract features from the multimodal data to obtain multimodal features, and call a cross-modal Transformer model to align the multimodal features to obtain final fusion features.

[0032] Next, feature extraction is performed on the multimodal data to obtain multimodal features. Specifically, for image data in the image modality, image feature extraction models such as convolution, residual, or Transformer are called to extract features to obtain image features. For environmental time series data in the time series modality, long short-term memory networks or Transformer models based on causal attention mechanisms are called to extract time series features to obtain time series features. For annotated text data in the text modality, text features are extracted through the Transformer model or the BERT model to obtain text features.

[0033] Next, the cross-modal Transformer model is called to align the multi-modal features to obtain the final fused features. Here, the cross-modal Transformer model is used as the feature fusion layer, and then the image features, text features, and time series features are input into the feature fusion layer. The final fused features are obtained through feature fusion, which contains multi-modal information and is used for the subsequent construction of the knowledge graph.

[0034] Step 103: construct a dynamic knowledge graph for cabbage planting management based on the final fusion features.

[0035] The construction of a dynamic knowledge graph is a key process in converting multimodal data into structured knowledge, which involves four processes: data preprocessing, entity relationship extraction, graph generation and updating, knowledge fusion and reasoning. Through these four processes, multimodal data is gradually transformed into a dynamic and reasonable graph, providing support for intelligent decision-making in agricultural application scenarios.

[0036] Here, the entity set and relationship set are constructed respectively by the final fusion features, image features, text features, time series features and other multimodal features, and then the triples are constructed to form a primary knowledge graph. The primary knowledge graph is dynamically updated according to the real-time input cabbage planting environment data to obtain a dynamic knowledge graph for cabbage planting management.

[0037] Step 104: for the cabbage planting interaction data to be processed, call the dynamic knowledge graph to search the cabbage planting interaction data to obtain the corresponding cabbage planting feedback result.

[0038] Here, the dynamic knowledge graph is used as a database for cabbage planting management. When dynamic interaction is performed, the dynamic knowledge graph can be called to retrieve the cabbage planting interaction data to be processed input by the user, and the corresponding cabbage planting feedback results can be obtained and fed back to the user for decision-making.

[0039] The cabbage planting interactive data input by the user can be a cabbage image that needs to be diagnosed for disease, a prompt word for consulting on cabbage planting, or a personalized recommendation for executing cabbage planting. The dynamic knowledge graph can complete the interaction with these cabbage planting interactive data, perform knowledge reasoning, and provide feedback such as disease diagnosis results, cabbage planting data, cabbage planting suggestions, and recommended content for cabbage planting, to facilitate users to make decisions.

[0040] In an embodiment of the present invention, multimodal features of multimodal data of cabbage planting management are extracted and feature alignment is performed to obtain the final fusion features, and then a dynamic knowledge graph of cabbage planting management is constructed based on the final fusion features. In this way, multimodal data is uniformly embedded into the dynamic knowledge graph framework to improve the accuracy of cabbage planting knowledge expression. In addition, the present invention also provides an interactive function, which can use the dynamic knowledge graph to search for cabbage planting interactive data for feedback, breaking the limitations of traditional planting guidance single text or table output, and providing farmers with more intuitive and efficient decision support.

[0041] In some embodiments, the multimodal features include image features of image data, time series features of environmental time series data, and text features of annotated text data. Figure 2 This article introduces the specific process of extracting features from multimodal data and obtaining multimodal features.

[0042] like Figure 2 As shown, in the process of feature extraction and fusion, the VisionTransformer model is called to perform disease classification tasks and leaf health status estimation tasks on the image data according to the image features of the image data, so as to obtain the image features of the image data.

[0043] Here, before image feature extraction, the image data is divided into multiple fixed-size image blocks (patches). Each patch is regarded as a sequence and input into the VisionTransformer model. The image block is encoded by multiple layers of Transformer, each layer including multi-head self-attention and feedforward network. The final output is the global feature vector , which integrates information on the shape and color of disease spots and the health status of leaves.

[0044] During the image feature extraction process of the VisionTransformer model, the disease classification task and the leaf health status estimation task are performed, wherein the disease classification task is used to classify cabbage diseases based on the image data, and the leaf health status estimation task is used to estimate the cabbage leaf health status based on the image data. The VisionTransformer model uses a shared feature extraction layer and generates two output paths at the same time. One is the classification path, which connects the classifier (including the fully connected layer and the softmax function), and the other is the regression path, which connects the regression module (including the fully connected layer and the sigmoid function). When performing the disease classification task, the global feature vector Input to the fully connected layer of the classification path for mapping processing to obtain the probability distribution of disease categories , represents the predicted probability corresponding to the nth disease category. The leaf health status estimation task belongs to the regression task. In the regression task, the global feature vector The input is sent to the fully connected layer of the regression path for mapping, and the output is the estimated value of the leaf health status. , for example [0, 1]. The higher the estimated value, the healthier the cabbage leaves.

[0045] Finally, the probability distribution of disease categories and leaf health status estimate The image features that make up the image data are denoted as .

[0046] In addition, the VisionTransformer model in the embodiment of the present invention is obtained by training with disease classification loss and state estimation loss, and multi-task learning is introduced in the training process. The specific training process is described below.

[0047] The training sample input of the VisionTransformer model is the cabbage leaf disease image, and the disease area annotation information of each image is also input, including the disease category label (used to perform the disease classification task and classify the disease) and the leaf health status label (used to perform the leaf health status estimation task and define the leaf health status level based on the leaf area and color distribution).

[0048] During the training process of multi-task learning, disease classification is performed through the disease classification task, such as powdery mildew, downy mildew, etc., and leaf health status estimation is performed through the leaf health status estimation task, such as healthy, slightly damaged, and severely damaged. When constructing the final loss function L, the loss functions of the two tasks are combined, and the formula is as follows: (1) in, is the disease classification loss function, using cross entropy loss for disease classification. is the health status estimation loss function, and the mean square error loss is used for blade health status estimation. It is a dynamically adjusted weight factor, which can be adjusted dynamically according to the importance of the task.

[0049] During the training process, the final loss function L is used to back-propagate in the VisionTransformer model to update the model parameters. The training is stopped when the maximum number of training rounds is reached or the final loss function L converges.

[0050] For the environmental time series data, the causal attention-based time series Transformer model is called here to extract features from the environmental time series data to obtain time series features.

[0051] Traditional long short-term memory networks have performance bottlenecks in long-term dependency modeling. The embodiment of the present invention uses a temporal transformer model (Temporal Causal Transformer, TCT) based on causal attention to perform temporal feature extraction. In the temporal feature extraction process, the model input is environmental time series data, such as temperature, humidity, light intensity, soil moisture, etc. Assuming that each environmental time series data contains t time steps, the feature dimension of each time step is D, and the dimension of the input matrix can be recorded as [t, D]. The core idea of ​​the causal attention mechanism is to use the causal attention mechanism to ensure that the current time step can only depend on the previous time step, and to ensure that the temporal modeling conforms to physical causality. The formula is as follows: (2) In the above formula (2), is the causal mask matrix, ensuring that the current time step only pays attention to past time steps. They are query vector, key vector and value vector respectively. is the vector dimension, and T represents the transpose of the matrix. The input environment time series data [t, D] is transformed through a linear transformation to map the features of each time step to Then, after the calculation of the causal attention mechanism, the hidden layer feature matrix [t, H] is generated, where H is the hidden layer dimension. Each hidden layer contains a causal attention module and a feedforward network, with M layers superimposed.

[0052] When the causal attention mechanism is calculating, dynamic window processing is performed on the input time series data. Specifically, the time window size is dynamically adjusted according to the fluctuation amplitude of the time series data. For example, when the environmental parameters such as temperature and humidity fluctuate slightly, a smaller window is used. When the environment changes drastically, the window is expanded to capture more contextual information. In addition, feature weighting is performed, that is, the feature weights of key time points are assigned by attention, redundant data is ignored, and the features output at the last time step are used as the time series features. , the formula is as follows: (3) In the above formula (3), Represents the output of the hidden layer at the last time step.

[0053] It is a fixed-length feature vector that contains information about the dynamic impact of environmental data on crop growth. This feature can be directly input into the subsequent feature fusion model. For the growth environment data of cabbage (temperature, humidity, light intensity, soil moisture), the model can extract key time points (such as the outbreak of downy mildew caused by continuous low temperature) and generate features that reflect dynamic changes.

[0054] For the annotated text data, the Tokenizer model is called to encode the annotated text data into a Token ID sequence. In the process of text feature extraction, the annotated text data is encoded into a Token ID sequence through the Tokenizer model, expressed as [CLS]+T+[SEP]. The Token ID sequence can directly extract semantic features, but in order to guide the model to better generate relevant semantic information in a specific field (such as the agricultural field or the cabbage planting field), the embodiment of the present invention constructs a field-specific Prompt template, such as "The best prevention and control method for the current description of cabbage diseases is", to improve the ability to understand domain knowledge.

[0055] Here, we obtain the prompt words of specific domain knowledge, and input the prompt words and Token ID sequence into the RoBERTa model to extract contextual semantic features, and obtain the text features of the annotated text data. Specifically, the input data of the RoBERTa model is in the form of "text encoding + Prompt template". First, the Prompt template is added to the text encoding (i.e., the Token ID sequence), so that the input data is expressed as [prompt] + [CLS] + T + [SEP], and then the input data is input into the RoBERTa model to extract contextual semantic features and obtain text features. , where f represents the vector labeled [CLS] and the vector of [prompt] The feature fusion function for combining. Text features Semantic information that can reflect symptoms of cabbage leaves, such as "the leaves turn yellow and have brown spots."

[0056] The embodiment of the present invention extracts multimodal features by calling different feature extraction models for multimodal data of image modality, text modality and time series modality. In the process of feature extraction, cabbage disease diagnosis, leaf health status estimation, determination of dynamic changes of environmental time series data on cabbage growth status, and construction of semantic information in specific areas of cabbage planting are realized, which increases the knowledge richness of the knowledge graph constructed subsequently.

[0057] In some embodiments, considering that in agricultural scenarios, the feature fusion of multimodal data (image data, time series data, text data) is the core step to achieve intelligent analysis and decision-making. In order to solve the problem of insufficient information interaction between modalities, a multimodal feature fusion method is proposed here, that is, calling a cross-modal Transformer model to align the multimodal features to obtain the final fusion features, which is described in detail below.

[0058] Firstly, the cross-modal attention mechanism of the cross-modal Transformer model is used to perform inter-modal interaction calculations on multimodal features, and image-time series interaction features, time series-text interaction features, and image-text interaction features are obtained respectively.

[0059] like Figure 2 As shown, in an embodiment of the present invention, a cross-modal Transformer (CMT) model is used as a feature fusion model, and a multi-head attention mechanism is used on the feature fusion layer of the model to establish inter-modal interaction, thereby generating a corresponding joint feature representation. The CMT model can combine self-supervised learning optimization to improve the semantic consistency and alignment capabilities between modalities.

[0060] The cross-modal attention mechanism aims to achieve deep interaction between image, time series data and text features, and to mine the correlation between modalities. During the calculation process, the multi-head attention mechanism uses the features of each modality as Query, Key and Vlue inputs to calculate cross-modal interaction information. The formula is as follows: (4) In the above formula (4), the query vector The feature vector representing the source mode. Key vector and value vector They all represent the eigenvectors of the target mode. is the vector dimension, and T represents the transpose of the matrix.

[0061] In the process of inter-modal interaction, there are three forms of interaction. One is image-time interaction, which uses image features With timing characteristics Perform association to obtain image-time interaction features This feature combines visual and temporal dynamic change information to explore the relationship between the environment and crop growth status. As the feature vector of the source modality, that is, the query vector , time series characteristics is the eigenvector of the target mode, i.e., the key vector Sum value vector , the formula is as follows: (5) In the above formula (5), is the calculation function of the multi-head attention mechanism. The following formulas (6) and (7) can be used for reference and will not be repeated here.

[0062] The second is time-text interaction, which uses time-series features With text features Perform association to obtain time-series-text interaction features , which connects temporal dynamics with the semantic expression of domain-specific knowledge, enhancing the understanding of procedural descriptions. As the feature vector of the source modality, that is, the query vector , text features is the eigenvector of the target mode, namely the key vector K and the value vector , the formula is as follows: (6) The third is image-text interaction, which uses image features With text features Perform association to obtain image-text interaction features , representing multi-dimensional scene descriptions. In the cross-modal attention mechanism, image features are As the feature vector of the source modality, that is, the query vector , text features is the eigenvector of the target mode, namely the key vector K and the value vector , the formula is as follows: (7) Next, the three interaction results of the features are combined to fuse the image-time sequence interaction feature, the time sequence-text interaction feature, and the image-text interaction feature to obtain a joint feature representation. , the formula is: (8) In the above formula (8), It is the processing function of feature fusion.

[0063] Finally, the joint feature representation is transformed into the final fusion feature. Here, the joint feature representation is adjusted by linear transformation and nonlinear activation function The feature dimension is expressed as: (9) In the above formula (9), represents a nonlinear activation function, and Represents the trainable parameters in the cross-modal Transformer model.

[0064] The embodiments of the present invention perform feature interaction between modalities through cross-modal attention to form multiple feature interactions, which can represent more knowledge information, and fuse multiple interactive features to effectively enhance the semantic consistency and robustness of cross-modal features.

[0065] In addition, in order to enhance the feature alignment capability of the cross-modal Transformer model as a feature fusion model, the generalization capability of the cross-modal Transformer model for feature fusion is provided. The embodiment of the present invention provides a training method for a cross-modal Transformer model. Specifically, the cross-modal Transformer model is obtained by training by constructing an overall loss function. The overall loss function is introduced below. The construction process.

[0066] The Transformer model is trained through self-supervised learning to achieve feature alignment of self-supervised learning. Its overall loss function is divided into two parts: modality contrast loss and modality reconstruction loss. The modality contrast loss is used to constrain the consistency of interactions between modalities in the semantic space, while the modality reconstruction loss is used to constrain the model to ensure information integrity and prevent feature information loss when reconstructing the original modality features.

[0067] The feature similarity between modalities is calculated for multimodal features, and the modal contrast loss is constructed based on the feature similarity. It consists of three parts, corresponding to the interaction between three modalities, including image-text modality contrast loss , Image-time modality contrast loss Time-series and text-modality contrastive loss .

[0068] Image-text modality contrast loss The construction formula is: (10) Image-Temporal Modality Contrastive Loss The construction formula is: (11) Temporal-Textual Modality Contrastive Loss The construction formula is: (12) In the above formulas (10), (11), and (12), N represents the batch size of training samples, j represents the jth training sample, represents the temperature coefficient, sim represents the function for calculating the characteristic similarity between modes, represents the time series characteristics of the i-th training sample, represents the text features of the i-th training sample, represents the image feature of the i-th training sample, and exp represents the exponential function.

[0069] As for the modal reconstruction loss, the final fusion features are reconstructed through the decoder of the cross-modal Transformer model to obtain the feature representation of each modality, and then the modal reconstruction loss is constructed based on the feature representation of each modality and the multimodal features. , the formula is as follows: (13) In the above formula (13), represents the final fusion feature, , , They represent the functions that perform reconstruction processing on the decoders of the image modality, text modality, and time series modality in the Transformer model, respectively. Represents image features, Represents text features, Represents the timing characteristics.

[0070] Finally, the modal contrast loss and modal reconstruction loss are weighted by the preset contrast weight and reconstruction weight respectively, and the weighted results are summed to obtain the overall loss function , expressed as the following formula: (14) In the above formula (14), is the comparison weight, To reconstruct the weights, Represents the modality contrast loss, and its specific value is the image-text modality contrast loss , Image-time modality contrast loss Time-series and text-modality contrastive loss The sum of the three, is the modality reconstruction loss.

[0071] During the training process, the overall loss function Backpropagation is performed in the cross-modal Transformer model to update the model parameters until the maximum number of training epochs or the overall loss function is reached. When convergence occurs, stop training.

[0072] In an embodiment of the present invention, a self-supervised learning method is adopted in the cross-modal Transformer model, and the model training is performed by constructing a modal contrast loss and a modal reconstruction loss. This enables the cross-modal Transformer model to have the ability to align features of different modalities when performing multimodal feature fusion, thereby improving the generalization ability of feature fusion, avoiding the loss of feature information, and enhancing the semantic consistency and robustness of the fused features.

[0073] The following is an introduction based on the final fusion feature The process of constructing a dynamic knowledge graph for cabbage planting management. Figure 2 As shown in the figure, the graph construction process of the dynamic knowledge graph includes data preprocessing, entity-relationship extraction, knowledge graph generation and updating, knowledge fusion and knowledge reasoning.

[0074] Firstly, the multimodal data of cabbage planting management are aligned with the final fusion features to obtain structured multimodal input data.

[0075] Here, data preprocessing needs to be performed before data alignment, which can be achieved through data processing algorithms. The data processing algorithm is used to clean the data and filter out noise information, such as removing low-quality images or abnormal environmental data. In addition, data formatting is required to convert image data, annotated text data, and environmental time series data into a standard format suitable for model input.

[0076] Then perform data alignment, which is the alignment of modal features. Here, based on the final fusion feature The timestamps in the image data, the annotated text data, and the environmental time series data are aligned respectively. For each timestamp of the image data, the annotated text data, and the environmental time series data, a final fusion feature needs to be aligned , to ensure the synchronization between multimodal data. The data processing algorithm finally outputs structured multimodal input data, providing a basis for entity relationship extraction.

[0077] Next, we perform entity-relationship extraction. The core of knowledge graph construction is to extract entities (such as "downy mildew") and relationships (such as "cause") from multimodal data. Specifically, we perform entity extraction on multimodal features, obtain corresponding entity nodes, and form entity sets. Multimodal features are also called image features. , text features , time series characteristics , the RoBERTa-Transformer-CRF model is used in the entity extraction process and combined with the final fusion feature , identify entity names and their attributes, such as "humidity = 85%" or "downy mildew risk", and finally output entity sets, such as: {high humidity, downy mildew, leaf disease}.

[0078] At the same time, we also extract relations from multimodal input data to obtain relations between entity nodes and form a relation set. Multimodal input data is the feature after data alignment. When extracting relations, we call the relation classification model of the cross-modal Transformer to capture the relations between entities in multimodal input data. Through the cross-modal attention mechanism and the final fusion feature ,identify the explicit and implicit relationships in the relationship, and finally obtain the corresponding ,relationship set, for example: {Excessive humidity leads to ,downy mildew}.

[0079] Finally, the knowledge graph is generated and updated. First, triples are constructed based on entity sets and relationship sets to obtain a primary knowledge graph. Here, the resource description framework RDF is used to construct triples (Entity1, Relation, Entity2), for example: (high humidity leads to downy mildew), and the graph database Neo4j is used to store the generated primary knowledge graph.

[0080] Then, the primary knowledge graph is dynamically updated to obtain a dynamic knowledge graph for cabbage planting management. Based on the input real-time environmental data, the entity nodes and relationships in the primary knowledge graph need to be updated, including adding, deleting, and modifying relationships.

[0081] The embodiment of the present invention uniformly embeds multimodal data and the final fusion features into the dynamic knowledge graph framework, increases the knowledge richness of the knowledge graph, improves the accuracy of cabbage planting knowledge expression, and solves the problem of single expression mode of dynamic knowledge expression system in the prior art.

[0082] The following introduces the dynamic update process of the primary knowledge graph.

[0083] The primary knowledge graph needs to be updated in real time and needs to be able to accept the input of real-time environmental data. When performing real-time updates, the real-time environmental data during the cabbage planting process is obtained, and the real-time environmental data is aggregated through a time window to obtain the window environmental features.

[0084] Real-time environmental data specifically includes environmental data for cabbage planting and corresponding structured data. The data types of real-time environmental data are: meteorological data: temperature, humidity, light intensity, etc.; soil data: pH value, water content, etc.; crop status data: leaf color, pest and disease characteristics, etc.; sensor data: real-time environmental change data collected in cabbage planting areas through IoT devices.

[0085] The structured data is to convert all the real-time environmental data into standardized feature vectors through normalization preprocessing, denoted as ,in, Represents a feature item in real-time environmental data (such as humidity or light intensity), For time point, It is the standardized feature vector after preprocessing, that is, structured data.

[0086] Then, the temporal feature extraction of real-time environment data is performed through the time window Aggregate the real-time environment data to obtain window environment features to reflect the dynamic changes of real-time environment data. , the calculation formula is as follows: (15) In the above formula (15), k is the size of the time window, Represents the structured data corresponding to the real-time environment data at time t. For explanations of other parameters, refer to .

[0087] Next, it is necessary to determine whether the primary knowledge graph needs to be updated. The judgment criterion is to determine the feature change rate of the window environment features between time windows. The calculation formula for the feature change rate is as follows: (16) In the above formula (16), k is the size of the time window, Represents the structured data corresponding to the real-time environment data at time t, represents the structured data corresponding to the real-time environment data at time t-1, and N represents the number of time windows.

[0088] When the characteristic change rate Greater than feature change threshold When the feature change rate is Less than or equal to the feature change threshold , it means that there is no need to update the knowledge graph. The following describes the update operation of the primary knowledge graph.

[0089] The update operation of the primary knowledge graph mainly updates the relationships in the relationship set, the corresponding relationship weights, and attribute values. First, based on the rule base and graph reasoning results, determine the relationships that need to be updated. Specifically, it includes removing relationships whose feature values ​​remain unchanged for a long time or exceed the association range, modifying relationship weights or attribute values ​​according to the latest environmental data, and introducing new relationships related to the current environment.

[0090] Specifically, first according to the window environment characteristics , construct the relationship change matrix before and after the update Based on the window environment characteristics, it can be determined whether a new relationship needs to be added, an outdated relationship needs to be moved, or a relationship needs to be kept unchanged, thereby mapping it into different feature matrices as the relationship change matrix , expressed as: (17) In the above formula (17), i represents the relationship before the update in the relationship set of the primary knowledge graph, that is, the relationship that still exists in the previous time window t-1, and j represents the relationship after the update in the relationship set of the primary knowledge graph, that is, the new relationship calculated in the current time window t.

[0091] Finally, within the time window, the relationship change matrix Merge with the original relation set R in the primary knowledge graph to obtain a new relation set , expressed as: (18) The embodiment of the present invention can accurately determine whether to update the relationship for the primary knowledge graph by calculating the feature change rate corresponding to the real-time environmental data through the time window. During the update, the relationship change matrix is ​​determined by the calculated window environment features, the relationship is updated, and the construction of the dynamic knowledge graph is finally completed. In this way, the dynamic adjustment of the knowledge graph is realized, the accuracy and real-time performance of the knowledge graph are ensured, and the static and lagging problems of the traditional agricultural planting knowledge base are compensated.

[0092] In some embodiments, after constructing the relationship change matrix before and after the update, a conflict detection process needs to be performed. This process occurs before performing the relationship update and is mainly used to detect whether the new knowledge added through real-time environmental data conflicts with the existing primary knowledge graph, and then perform conflict correction to ensure the consistency of the knowledge graph.

[0093] In the embodiment of the present invention, whether there is a conflict is determined by calculating the update conflict score C as a judgment criterion. Specifically, first determine the relationship set before the update and the updated set of relations The feature similarity , here we can calculate the cosine similarity between the two features. Then determine the mutual exclusion conflict scores between the relationship set before and after the update. , attribute conflict score and reasoning conflict score .

[0094] Here, the rules for detecting conflicts are determined as three types of conflicts: mutually exclusive conflicts, attribute conflicts, and reasoning conflicts. Mutually exclusive conflicts are when two or more relationships are logically mutually exclusive (such as "high humidity" and "low humidity"). The value of is 1 when the two relations are mutually exclusive, otherwise it is 0. Attribute conflict means that the attribute value of the same entity exceeds the specified range of the rule. The attribute conflict score is When the attribute value exceeds the preset range, the deviation degree is calculated and the attribute value is normalized to [0, 1]. Otherwise, no calculation is required. Reasoning conflict means that the conclusion drawn through graph reasoning is inconsistent with the existing knowledge. The reasoning conflict score is The value is 1 when the inference result contradicts the existing knowledge, otherwise it is 0.

[0095] Next, these three parts are combined to obtain the conflict degree score conf, where the mutually exclusive conflict score , attribute conflict score and reasoning conflict score Add them together to get the conflict degree score conf, expressed as: (19) Then the feature similarity Conflict Score The difference between , expressed as: (20) In the above formula (20), Represents the relationship set before update, Represents the updated relation collection.

[0096] After determining the update conflict score of the current relationship update, it can be used as a criterion to determine whether the update relationship conflicts. Greater than the preset conflict threshold , it is determined whether there is a conflict between the relationship set before the update and the relationship set after the update, otherwise there is no conflict.

[0097] When it is determined that there is a conflict, the relationship weights in the relationship set are adjusted according to the conflict degree score to correct the conflict between the relationship set before the update and the relationship set after the update.

[0098] Here, the present invention implements a conflict correction strategy by adjusting the relationship weights in the relationship set, specifically adjusting the relationship set before the update With the updated relationship set The relationship weight is used to balance the influence of feature similarity and conflict degree score. The adjustment formula is expressed as: (twenty one) In the above formula (21), represents the adjusted relationship weight, represents the relationship weight before adjustment, is the weight coefficient, Represents the relationship set before update With the updated relationship set The conflict level score.

[0099] In some embodiments, if the conflict still cannot be corrected by adjusting the relationship weight, the conflicting relationship is removed from the relationship set of the current initial knowledge graph and an alternative relationship is added, which is expressed as: (twenty two) In the above formula (22), Represents the updated relation set (new knowledge graph relation set) after replacing the conflicting relation; Represents the original relationship set, that is, the knowledge graph relationship set before conflict detection; Represents the detected conflicting relations, i.e., relations that need to be removed, such as mutually exclusive, attribute, and reasoning conflicting relations; Represents an alternative relationship, that is, a new relationship that is consistent with the current environment data of the knowledge graph, which is specifically generated based on a rule base or reasoning results.

[0100] The embodiment of the present invention proposes a conflict detection method and a conflict correction method before executing the relationship update of the initial knowledge graph, and finally completes the construction of the dynamic knowledge graph, which can ensure the knowledge consistency of the newly added knowledge and the current knowledge graph. In this way, a dynamic update method of cabbage planting knowledge is realized, which makes up for the static and lagging problems of the traditional agricultural knowledge base.

[0101] In addition, the process of constructing a dynamic knowledge graph also involves knowledge fusion and knowledge reasoning. The goal of knowledge fusion is to unify the graph content through knowledge from different sources and improve the integrity and consistency of the graph, while knowledge reasoning is to complete data-driven knowledge reasoning through a dynamic knowledge graph, which can provide support for agricultural decision-making in cabbage planting. The details are explained below.

[0102] For knowledge fusion, we first need to determine the feature similarity between any two entity nodes in the primary knowledge graph, expressed as , the calculation formula is as follows: (twenty three) In the above formula (23), Respectively represent any two entity nodes in the dynamic knowledge graph, Then they represent any two entity nodes respectively. The final fusion feature.

[0103] When the feature similarity is greater than the preset similarity threshold, the two entity nodes are merged to achieve knowledge fusion, and when the feature similarity is less than or equal to the preset similarity threshold, no knowledge fusion is required.

[0104] For knowledge reasoning, we extract entity nodes and edges from the dynamic knowledge graph, call the graph convolutional neural network (GCN) to predict the relationship between entity nodes and edges, obtain predicted nodes and predicted relationships, and use the graph convolutional neural network to predict future relationships. For example, based on "leaf humidity", it is predicted that "downy mildew" may occur. The predicted node is a modal entity, with examples of "high humidity" and "downy mildew", and the predicted edge is a relationship, with examples of "cause" and "related", and the attribute value is the information updated in time series.

[0105] Finally, the predicted nodes and predicted relationships are merged with the entity set and relationship set in the dynamic knowledge graph. Here, the inference knowledge obtained by the graph convolutional neural network prediction is also added to the knowledge graph. In this way, through knowledge fusion and knowledge reasoning, the integrity and consistency of the dynamic knowledge graph are guaranteed, which can provide agricultural decision support for cabbage planting management.

[0106] In some embodiments, by constructing a dynamic knowledge graph, decision support can be provided for cabbage planting management, and farmers can be provided with efficient, accurate and personalized agricultural planting support services.

[0107] First, for the cabbage planting interaction data to be processed, the dynamic knowledge graph is called to search the cabbage planting interaction data to obtain the corresponding cabbage planting feedback results.

[0108] Here, the dynamic knowledge graph can be modeled into a decision-making platform to achieve dynamic interaction with farmers, such as Figure 2 As shown in the figure, the decision-making platform can realize three functions: disease diagnosis, intelligent question and answer, and personalized recommendation. When farmers grow cabbage and need to provide decision support, the cabbage planting interaction data to be processed entered by the farmers can be obtained in real time. At this time, the dynamic knowledge graph is called to retrieve the cabbage planting interaction data to obtain the corresponding cabbage planting feedback results.

[0109] For the three functions of disease diagnosis, intelligent question and answer, and personalized recommendation, the cabbage planting interactive data to be processed is also divided into three cases, and the final cabbage planting feedback results also have three cases, which are explained one by one below.

[0110] When the cabbage planting interactive data to be processed is a cabbage image that needs to be diagnosed for disease, the VisionTransformer model is called to extract the disease features of the cabbage image, and the corresponding disease nodes are retrieved in the dynamic knowledge graph based on the disease features. Then, knowledge reasoning is performed on the disease nodes through the dynamic knowledge graph to obtain the disease diagnosis results of the cabbage image as the cabbage planting feedback results.

[0111] After the farmer uploads the cabbage image that needs to be diagnosed for disease, it indicates that the dynamic knowledge graph needs to be called to perform the disease diagnosis function. Here, the cabbage image is first preprocessed through the image recognition model to extract the corresponding image features. The image features are then input into the dynamic knowledge graph, and the corresponding entity nodes are retrieved as disease nodes. Inference is performed based on regional environmental data (such as climate and soil conditions) to output the most likely disease type and its associated features.

[0112] Specifically, we first extract image features from the uploaded cabbage images through a multi-task learning model and VisionTransformer as disease features (spot shape, color, etc.), and map them to image feature nodes in the dynamic knowledge graph. Then, we call the RoBERTa model to extract contextual semantic features from the annotated text data of cabbage planting management, obtain disease descriptive features, and establish semantic associations with disease nodes in the knowledge graph. At the same time, based on the environmental time series data of the uploaded cabbage images, we extract time series features through the TCT model as disease causal change features, which are used to represent time nodes and environmental influencing factors (such as climate mutations and seasonal changes).

[0113] Next, we will do disease feature matching. The first step is image matching. Using the disease features generated by the Vision Transformer, we use vector retrieval to find the disease node closest to the disease features in the dynamic knowledge graph. Then, using the disease descriptive features generated by RoBERTa, we match the corresponding disease description nodes and prevention and control measures nodes in the dynamic knowledge graph by calculating the semantic similarity of the embedding space. Next, we will do multimodal feature joint matching. Using the cross-modal Transformer model, we will jointly calculate the disease features, disease causal change features, and disease descriptive features to generate the final fusion features. In the dynamic knowledge graph, we will align the feature vectors of the multimodal disease description nodes to achieve comprehensive matching of disease types.

[0114] In the above matching process, the path search algorithm based on the knowledge graph can be called, and the Bayesian network and multimodal fusion model can be combined to perform reasoning in the dynamic knowledge graph. The disease node and its associated nodes are traversed through the path, and the most likely disease type and its transmission path are output in combination with the environmental causal relationship. Finally, the diagnosis result is output, and the disease name, cause of the disease, transmission path and specific prevention and control measures are returned. The system recommends appropriate agents and usage methods. At the same time, cabbage planting feedback results such as medication time recommendations are provided based on climate forecast data.

[0115] When the cabbage planting interaction data to be processed is a prompt word for consulting cabbage planting, the dynamic knowledge graph is called to perform knowledge reasoning on the prompt word to obtain the cabbage planting data as the cabbage planting feedback result.

[0116] Here, when farmers input planting-related questions through text as prompt words for consulting cabbage planting, such as "What crops are suitable for this piece of land?" or "When do I need to fertilize?", it means that the dynamic knowledge graph needs to be called to perform the intelligent question-answering function. First, the semantic parsing and intent recognition of the cabbage planting prompt words are completed through the natural language processing module. Then, according to the question intent, the corresponding entity nodes (such as crop types, fertilization rules) are located from the dynamic knowledge graph, and knowledge reasoning is performed in combination with real-time data input of multimodal perception (such as soil nutrient data, regional planting records). Finally, accurate answers are generated as cabbage planting data, such as outputting cabbage planting feedback results such as the types of crops suitable for planting, the types and amounts of fertilizers, and attaching relevant scientific basis (such as historical data, experimental results). In addition, farmers can add additional questions, such as "Why is this fertilizer recommended?" At this time, the dynamic knowledge graph is continued to be called to further perform knowledge reasoning and obtain explanatory answers.

[0117] When the cabbage planting interaction data to be processed represents the personalized recommendation of cabbage planting, the dynamic knowledge graph is called to perform knowledge reasoning on the historical cabbage planting data to obtain the recommended content of cabbage planting as the cabbage planting feedback result.

[0118] Here, when farmers input the cabbage planting interactive data to be processed and need to obtain personalized recommendations for cabbage planting, the farmer's portrait will be dynamically constructed based on the farmer's historical planting behavior, regional environmental characteristics, and real-time sensor data, such as the farmer's historical planting data, such as the type and time of planting crops, regional soil, climate and other environmental characteristics, farmers' operating habits, commonly used planting equipment, and planting issues of concern. Then, the dynamic knowledge graph is called to perform knowledge reasoning on the farmer's portrait to generate recommended content corresponding to cabbage planting, such as fertilization, irrigation or disease prevention and control plans for specific crops, recommending the best planting time based on weather forecasts, and providing relevant technical guidance or learning resources for common user problems, and other cabbage planting feedback results.

[0119] The embodiment of the present invention completes three interactive functions of disease diagnosis, intelligent question and answer, and personalized recommendation through the knowledge reasoning ability of the dynamic knowledge graph of cabbage planting management, realizes the dynamic interaction of cabbage planting interactive data, ensures the real-time and scientific nature of planting management knowledge, breaks the limitation of single text or table output of traditional planting guidance, and effectively provides agricultural decision-making for cabbage planting.

[0120] The following is a description of the cabbage knowledge dynamic expression and interaction system based on multimodal fusion provided by the present invention. The cabbage knowledge dynamic expression and interaction system based on multimodal fusion described below and the cabbage knowledge dynamic expression and interaction method based on multimodal fusion described above can be referred to each other. Figure 3 As shown in the figure, the cabbage knowledge dynamic expression and interaction system based on multimodal fusion includes: A feature extraction module 301 is used to obtain multimodal data of cabbage planting management, wherein the multimodal data includes image data of cabbage growth, environmental time series data, and annotated text data of cabbage planting management; a feature fusion module 302 is used to extract features from the multimodal data to obtain multimodal features, and call a cross-modal Transformer model to align the multimodal features to obtain final fused features; a graph construction module 303 is used to construct a dynamic knowledge graph of cabbage planting management based on the final fused features; a dynamic interaction module 304 is used to call the dynamic knowledge graph to retrieve the cabbage planting interaction data to be processed, and obtain corresponding cabbage planting feedback results.

[0121] It should be noted that the beneficial effects of the cabbage knowledge dynamic expression and interaction system based on multimodal fusion here and the cabbage knowledge dynamic expression and interaction method based on multimodal fusion mentioned above can correspond to each other, so the beneficial effects of the cabbage knowledge dynamic expression and interaction system based on multimodal fusion will not be repeated here.

[0122] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute a method for dynamic expression and interaction of cabbage knowledge based on multimodal fusion, the method comprising: obtaining multimodal data of cabbage planting management, the multimodal data comprising image data of cabbage growth, environmental time series data and annotated text data of cabbage planting management; extracting features from the multimodal data to obtain multimodal features, and calling a cross-modal Transformer model to align the multimodal features to obtain final fusion features; constructing a dynamic knowledge graph of cabbage planting management based on the final fusion features; for the cabbage planting interaction data to be processed, calling the dynamic knowledge graph to retrieve the cabbage planting interaction data to obtain corresponding cabbage planting feedback results.

[0123] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0124] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the cabbage knowledge dynamic expression and interaction method based on multimodal fusion provided by the above methods, and the method includes: obtaining multimodal data of cabbage planting management, the multimodal data including image data of cabbage growth, environmental time series data and annotated text data of cabbage planting management; performing feature extraction on the multimodal data to obtain multimodal features, and calling a cross-modal Transformer model to perform feature alignment on the multimodal features to obtain final fusion features; constructing a dynamic knowledge graph for cabbage planting management based on the final fusion features; for the cabbage planting interaction data to be processed, calling the dynamic knowledge graph to retrieve the cabbage planting interaction data to obtain corresponding cabbage planting feedback results.

[0125] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the cabbage knowledge dynamic expression and interaction method based on multimodal fusion provided by the above-mentioned methods, the method comprising: obtaining multimodal data of cabbage planting management, the multimodal data comprising image data of cabbage growth, environmental time series data and annotated text data of cabbage planting management; performing feature extraction on the multimodal data to obtain multimodal features, and calling a cross-modal Transformer model to perform feature alignment on the multimodal features to obtain final fusion features; constructing a dynamic knowledge graph for cabbage planting management based on the final fusion features; for the cabbage planting interaction data to be processed, calling the dynamic knowledge graph to retrieve the cabbage planting interaction data to obtain corresponding cabbage planting feedback results.

[0126] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0127] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for dynamic expression and interaction of cabbage knowledge based on multimodal fusion, characterized in that: include: Acquire multimodal data of cabbage planting management, wherein the multimodal data includes image data of cabbage growth, environmental time series data, and annotated text data of cabbage planting management; Extracting features from the multimodal data to obtain multimodal features, and calling a cross-modal Transformer model to perform feature alignment on the multimodal features to obtain final fusion features; Constructing a dynamic knowledge graph for cabbage planting management based on the final fusion features; For the cabbage planting interaction data to be processed, the dynamic knowledge graph is called to search the cabbage planting interaction data to obtain corresponding cabbage planting feedback results.

2. The method for dynamic expression and interaction of cabbage knowledge based on multimodal fusion according to claim 1 is characterized in that: The multimodal features include image features of the image data, time series features of the environment time series data, and text features of the annotated text data. The feature extraction of the multimodal data to obtain the multimodal features includes: Calling the VisionTransformer model to perform a disease classification task and a leaf health state estimation task on the image data to obtain image features of the image data, wherein the disease classification task is used to classify cabbage diseases on the image data, and the leaf health state estimation task is used to estimate the cabbage leaf health state on the image data, and the VisionTransformer model is trained by disease classification loss and state estimation loss; Calling a causal attention-based temporal Transformer model to extract features from the environmental temporal data to obtain temporal features; Calling the Tokenizer model to encode the annotated text data into a Token ID sequence; Obtain prompt words of specific domain knowledge, and input the prompt words and the Token ID sequence into the RoBERTa model to extract contextual semantic features to obtain text features of the annotated text data.

3. The cabbage knowledge dynamic expression and interaction method based on multimodal fusion according to claim 1 is characterized in that: The calling of the cross-modal Transformer model to perform feature alignment on the multi-modal features to obtain the final fusion features includes: The cross-modal attention mechanism of the cross-modal Transformer model is used to perform inter-modal interaction calculation on the multi-modal features, thereby obtaining image-time series interaction features, time series-text interaction features, and image-text interaction features respectively; Fusing the image-time sequence interaction feature, the time sequence-text interaction feature and the image-text interaction feature to obtain a joint feature representation; The joint feature representation is subjected to feature dimension transformation processing to obtain a final fusion feature.

4. The cabbage knowledge dynamic expression and interaction method based on multimodal fusion according to claim 1 is characterized in that: The cross-modal Transformer model is trained by constructing an overall loss function, and the construction process of the overall loss function includes: Calculating feature similarity between modalities for the multimodal features, and constructing modality contrast loss based on the feature similarity, wherein the modality contrast loss includes image-text modality contrast loss, image-time series modality contrast loss, and time series-text modality contrast loss; Reconstructing the final fused features through a decoder of the cross-modal Transformer model to obtain feature representations of each modality; Based on the feature representation of each modality and the multimodal feature, construct a modality reconstruction loss; The modality contrast loss and the modality reconstruction loss are weighted respectively by preset contrast weights and reconstruction weights, and the weighted results are summed to obtain an overall loss function.

5. The cabbage knowledge dynamic expression and interaction method based on multimodal fusion according to claim 1 is characterized in that: The dynamic knowledge graph for cabbage planting management is constructed based on the final fusion features, including: Performing data alignment on the multimodal data of cabbage planting management and the final fusion feature to obtain structured multimodal input data; Perform entity extraction on multimodal features to obtain corresponding entity nodes and form an entity set; Performing relationship extraction on the multimodal input data to obtain the relationship between the entity nodes and form a relationship set; Construct triples based on the entity set and the relationship set to obtain a primary knowledge graph; The primary knowledge graph is dynamically updated to obtain a dynamic knowledge graph for cabbage planting management.

6. The method for dynamic expression and interaction of cabbage knowledge based on multimodal fusion according to claim 5 is characterized in that: The dynamic update process of the primary knowledge graph includes: Acquire real-time environmental data during the cabbage planting process, and aggregate the real-time environmental data through a time window to obtain window environmental features; Determine a feature change rate of the window environment feature between time windows, and when the feature change rate is greater than a feature change threshold, trigger an update operation on the primary knowledge graph; The updating operation on the primary knowledge graph includes: According to the window environment characteristics, a relationship change matrix before and after the update is constructed; Within the time window, the relationship change matrix is ​​merged with the original relationship set in the primary knowledge graph to obtain a new relationship set.

7. The method for dynamic expression and interaction of cabbage knowledge based on multimodal fusion according to claim 6 is characterized in that: After constructing the relationship change matrix before and after the update, the method further includes: Determine the feature similarity between the relationship set before updating and the relationship set after updating; Respectively determining a mutual exclusion conflict score, an attribute conflict score, and an inference conflict score between the relationship set before the update and the relationship set after the update; Add the mutual exclusion conflict score, the attribute conflict score and the reasoning conflict score to obtain a conflict degree score; The difference between the feature similarity and the conflict degree score is used as an update conflict score, and when the update conflict score is greater than a preset conflict threshold, it is determined that there is a conflict between the relationship set before the update and the relationship set after the update; The relationship weights in the relationship set are adjusted according to the conflict degree scores to correct the conflicts between the relationship set before the update and the relationship set after the update.

8. The method for dynamic expression and interaction of cabbage knowledge based on multimodal fusion according to claim 6 is characterized in that: The method further comprises: Determine the feature similarity between any two entity nodes in the primary knowledge graph, and when the feature similarity is greater than a preset similarity threshold, merge the two entity nodes; Extract entity nodes and edges in the dynamic knowledge graph, call the graph convolutional neural network to predict the relationship between the entity nodes and the edges, and obtain predicted nodes and predicted relationships; The predicted nodes and predicted relationships are merged with the entity set and relationship set in the dynamic knowledge graph.

9. The cabbage knowledge dynamic expression and interaction method based on multimodal fusion according to claim 1, characterized in that: The method calls the dynamic knowledge graph to retrieve the cabbage planting interaction data to be processed, and obtains corresponding cabbage planting feedback results, including: When the cabbage planting interactive data to be processed is a cabbage image that needs to be diagnosed for a disease, the VisionTransformer model is called to extract the disease features of the cabbage image, and the corresponding disease node is retrieved in the dynamic knowledge graph according to the disease features; Performing knowledge reasoning on the disease node through the dynamic knowledge graph to obtain a disease diagnosis result of the cabbage image as a cabbage planting feedback result; When the cabbage planting interaction data to be processed is a prompt word for consulting cabbage planting, calling the dynamic knowledge graph to perform knowledge reasoning on the prompt word to obtain cabbage planting data as a cabbage planting feedback result; When the to-be-processed cabbage planting interaction data represents the personalized recommendation for executing cabbage planting, the dynamic knowledge graph is called to perform knowledge reasoning on the cabbage historical planting data to obtain the recommended content of cabbage planting as the cabbage planting feedback result.

10. A cabbage knowledge dynamic expression and interaction system based on multimodal fusion, characterized in that: include: A feature extraction module, used to obtain multimodal data of cabbage planting management, wherein the multimodal data includes image data of cabbage growth, environmental time series data, and annotated text data of cabbage planting management; A feature fusion module is used to extract features from the multimodal data to obtain multimodal features, and call a cross-modal Transformer model to perform feature alignment on the multimodal features to obtain final fusion features; A graph construction module, used to construct a dynamic knowledge graph for cabbage planting management based on the final fusion features; The dynamic interaction module is used to call the dynamic knowledge graph to search the cabbage planting interaction data to be processed, and obtain the corresponding cabbage planting feedback result.

Images