Multi-modal perception-driven vegetable knowledge graph construction method and multi-modal perception-driven vegetable knowledge graph construction device

Abstract

The invention provides a multi-modal perception-driven vegetable knowledge graph construction method and device, and relates to the technical field of agricultural artificial intelligence, and the method constructs a vegetable knowledge graph by obtaining the multi-modal information of each vegetable, taking a vegetable entity as a node, and taking the relationship between the multi-modal information of different vegetables as an edge. According to the multi-modal perception-driven vegetable knowledge graph construction method provided by the invention, information integration and intelligent decision support in a vegetable production process are realized, and the efficiency and quality of agricultural production are improved.

Description

Technical Field

[0001] The present invention relates to the field of agricultural artificial intelligence technology, and in particular to a method and device for constructing a vegetable knowledge graph driven by multimodal perception. Background Art

[0002] In the context of the development of digital agriculture today, the agricultural field has accumulated a large amount of data from different sources and formats, such as vegetable planting environment, growth process monitoring data, market demand information, etc. However, these agricultural data often present problems such as fragmentation and dispersion, making it difficult for farmers to obtain comprehensive and timely decision-making information.

[0003] In addition, the traditional agricultural knowledge management system used to manage these agricultural data also has many problems, such as information islands, single interaction methods, and high update and maintenance costs, which make it difficult for a large number of users to effectively use the system, resulting in a poor user interaction experience. It is also impossible to integrate information in the vegetable production process and support intelligent decision-making, which limits the effectiveness of knowledge acquisition and decision-making support, resulting in low efficiency and quality of agricultural production. Summary of the invention

[0004] The present invention provides a method and device for constructing a vegetable knowledge graph driven by multimodal perception, so as to solve the above-mentioned problems existing in the prior art.

[0005] The present invention provides a method for constructing a vegetable knowledge graph driven by multimodal perception, comprising the following steps.

[0006] Acquire multimodal information of each vegetable; the multimodal information includes at least two of text information, image information, audio information and video information; A vegetable knowledge graph is constructed by taking a vegetable entity as a node and the relationship between multimodal information of different kinds of vegetables as edges.

[0007] According to a method for constructing a vegetable knowledge graph driven by multimodal perception provided by the present invention, the relationship between the multimodal information of different types of vegetables includes one or more of the following relationships: Planting location relationship; growing season relationships; soil information relationships; Relationship with meteorological conditions; Disease relationship; Pest relations; fertilizer relations; Pesticide relations; Planting technology relationship.

[0008] According to a multimodal perception-driven vegetable knowledge graph construction method provided by the present invention, the method further includes: Perform multimodal knowledge representation learning on the multimodal information of each vegetable to obtain a high-dimensional feature vector containing semantic information; Perform cross-modal knowledge alignment on the high-dimensional feature vector.

[0009] According to a multimodal perception-driven vegetable knowledge graph construction method provided by the present invention, the method further includes: The graph reasoning algorithm is used to update and maintain the vegetable knowledge graph in real time.

[0010] According to a multimodal perception-driven vegetable knowledge graph construction method provided by the present invention, the method further includes: Obtaining the query command input by the user through the interactive interface; Based on the query instruction, the query result is obtained using the vegetable knowledge graph, and the query result is presented through an interactive interface.

[0011] According to a multimodal perception-driven vegetable knowledge graph construction method provided by the present invention, the method further includes: Based on the user's historical query records and feedback information, obtain the content that the user is interested in; Recommending content of interest to the user through the interactive interface.

[0012] The present invention also provides a multimodal perception-driven vegetable knowledge graph construction device, comprising the following modules: An acquisition module, used to acquire multimodal information of each vegetable; the multimodal information includes at least two of text information, image information, audio information and video information; The construction module is used to construct a vegetable knowledge graph with a vegetable entity as a node and the relationship between the multimodal information of different types of vegetables as edges.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for constructing a vegetable knowledge graph driven by multimodal perception as described in any one of the above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for constructing a vegetable knowledge graph driven by multimodal perception as described in any one of the above.

[0015] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the multimodal perception-driven vegetable knowledge graph construction methods described above.

[0016] The present invention provides a method for constructing a vegetable knowledge graph driven by multimodal perception. By acquiring the multimodal information of each vegetable, a vegetable entity is used as a node, and the relationship between the multimodal information of different kinds of vegetables is used as an edge to construct a vegetable knowledge graph, thereby realizing the integration of information in the vegetable production process and the support of intelligent decision-making, thereby improving the efficiency and quality of agricultural production. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of a method for constructing a vegetable knowledge graph driven by multimodal perception provided by the present invention.

[0019] Figure 2 It is a schematic diagram of a multimodal vegetable knowledge graph and an interactive system structure of a multimodal perception-driven vegetable knowledge graph construction method provided by the present invention.

[0020] Figure 3 It is a schematic diagram of a cross-modal knowledge alignment structure of a multimodal perception-driven vegetable knowledge graph construction method provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of a multimodal perception-driven vegetable knowledge graph construction device provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In the context of the development of digital agriculture today, a large amount of agricultural data from different sources and formats has accumulated in the agricultural field, such as vegetable planting environment, growth process monitoring data, market demand information, etc. These data often present problems such as fragmentation and dispersion, making it difficult for farmers to obtain comprehensive and timely decision-making information.

[0024] However, the traditional agricultural knowledge management system used to manage these agricultural data has many problems, such as information islands, single interaction mode, insufficient intelligence, etc., which limit its application effect in actual production. Therefore, in order to better support agricultural production decision-making and improve agricultural production efficiency, it is necessary to build a vegetable knowledge graph system that can integrate multiple data sources and realize multimodal interaction.

[0025] Although modern agriculture has accumulated a large amount of data from various sources and formats, including crop growth data, soil information, climate conditions, and market dynamics, there are currently some obvious technical shortcomings: (1) Information island problem: Currently available vegetable-related data are scattered in different databases and documents, and the formats are not unified, which makes it difficult to integrate information. There is a lack of effective integration and sharing mechanism between various data sources in the agricultural knowledge management system. This leads to the fragmentation and redundancy of agricultural data, making it difficult for users to obtain the required information from a comprehensive perspective.

[0026] (2) High update and maintenance costs: The update and maintenance costs of the existing system are high, requiring manual data cleaning, integration and updating, and lacking automation and real-time performance. This limits users’ timely access to the latest agricultural information and technical support, affecting the efficiency and quality of agricultural production.

[0027] (3) Single interaction mode: The user interaction mode of the existing system is single, mainly relying on text query, lacking diversified and personalized interaction modes. This makes it difficult for some users to effectively use the system, limiting the effectiveness of knowledge acquisition and decision support.

[0028] In response to various problems of traditional agricultural knowledge management systems, this paper proposes an innovative multimodal vegetable knowledge graph and interactive system, which aims to integrate multi-source heterogeneous data in the agricultural production process and realize intelligent management and decision support for the vegetable production process. Taking vegetables as the research object, multimodal data (including images, texts, etc.) are used to construct a vegetable knowledge graph. Through deep learning and knowledge representation technology, effective representation of multimodal data and cross-modal knowledge alignment are achieved, thereby providing farmers with personalized planting guidance, pest and disease identification, intelligent agricultural question and answer services, and thus improving the efficiency and quality of agricultural production.

[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] The present invention provides a multimodal perception-driven vegetable knowledge graph construction method, which aims to solve the problems existing in traditional agricultural information management systems, realize the integration of information in the vegetable production process, support intelligent decision-making, and improve the user interaction experience.

[0031] Combine the following Figures 1 to 5 The present invention describes the method and device for constructing a vegetable knowledge graph driven by multimodal perception.

[0032] The present invention provides a method for constructing a vegetable knowledge graph driven by multimodal perception, which uses multimodal data (including images, text, etc.) to construct a vegetable knowledge graph, thereby realizing the integration of information and support for intelligent decision-making in the vegetable production process. By collecting, representing and learning the multimodal data of vegetables and constructing a knowledge graph, farmers are provided with personalized planting guidance, pest and disease identification, intelligent agricultural question-and-answer services, etc., thereby improving the efficiency and quality of agricultural production. The construction process of the multimodal vegetable knowledge graph includes data collection and preprocessing, multimodal knowledge representation learning, cross-modal knowledge alignment, graph construction and updating, user interaction interface design, experimental evaluation and performance optimization, etc.

[0033] Figure 1 is a flow chart of a method for constructing a vegetable knowledge graph driven by multimodal perception provided by the present invention, such as Figure 1 As shown, the method comprises the following steps: Step 101, obtaining multimodal information of each vegetable; the multimodal information includes at least two of text information, image information, audio information and video information; Before constructing a multimodal vegetable knowledge graph, we first need to collect and preprocess data, mainly obtaining multimodal data related to vegetables from agricultural bases, agricultural product databases, the Internet and other channels, including text descriptions, images, audio and video, etc. Based on at least two of the data, such as vegetable-related text descriptions and images, the text information and image information of the vegetables are obtained, which together serve as the multimodal information of the vegetables.

[0034] In addition, the acquired data needs to be preprocessed, including removing noise, standardizing image size and format, cleaning text data, etc., to ensure the quality and consistency of the data.

[0035] Step 102: Construct a vegetable knowledge graph by taking a vegetable entity as a node and the relationship between multimodal information of different kinds of vegetables as edges.

[0036] Figure 2 : is a schematic diagram of a multimodal vegetable knowledge graph and an interactive system structure of a multimodal perception-driven vegetable knowledge graph construction method provided by the present invention, such as Figure 2 shown.

[0037] Specifically, a multimodal vegetable knowledge graph and interactive system was constructed using vegetables such as cabbage and cucumber. In the graph construction process, each vegetable variety entity was represented as a node, and the information of vegetables in multiple modalities, including text descriptions, images, and video features, was considered and integrated into a unified graph.

[0038] Optionally, the relationship between the multimodal information of different types of vegetables includes one or more of the following relationships: Planting location relationship; growing season relationships; soil information relationships; Relationship with meteorological conditions; Disease relationship; Pest relations; fertilizer relations; Pesticide relations; Planting technology relationship.

[0039] Specifically, the relationship between different vegetable entities is represented as an edge in the graph, including information about the vegetable's planting location, growing season, soil information, meteorological conditions, diseases, pests, etc., as well as connections with other related entities (such as fertilizers, pesticides, planting techniques, etc.). Users can gain a more comprehensive understanding of vegetable planting, management, and application through the constructed multimodal vegetable knowledge graph and interactive system.

[0040] Optionally, the multimodal perception-driven vegetable knowledge graph construction method further includes: Perform multimodal knowledge representation learning on the multimodal information of each vegetable to obtain a high-dimensional feature vector containing semantic information; Perform cross-modal knowledge alignment on the high-dimensional feature vector.

[0041] Specifically, when constructing a vegetable knowledge graph, the embodiment of the present invention uses natural language processing and knowledge representation technology to perform semantic modeling based on the multimodal information of each vegetable obtained, so as to realize the construction of the vegetable knowledge graph, so that the multimodal vegetable knowledge graph and the interactive system can more comprehensively understand and describe the vegetable-related information. The following takes the case where the multimodal information of each vegetable obtained includes text information and image information as an example to illustrate the cross-modal knowledge alignment in the process of constructing a vegetable knowledge graph driven by multimodal perception.

[0042] Figure 3 is a schematic diagram of a cross-modal knowledge alignment structure of a multimodal perception-driven vegetable knowledge graph construction method provided by the present invention, such as Figure 3 Before performing cross-modal knowledge alignment, it is necessary to first perform visual feature extraction based on the image information of each vegetable and perform text feature extraction based on the text information of each vegetable in order to obtain a high-dimensional feature vector containing semantic information.

[0043] In order to effectively utilize multimodal data, the embodiment of the present invention adopts an advanced two-stream structure image-text pair encoding model for knowledge representation learning. This method can convert image and text data into high-dimensional feature vectors containing semantic information to capture their semantic information and provide a basis for subsequent cross-modal alignment.

[0044] 1. Extract visual features based on the image information of each vegetable

[0045] In order to convert the image into a numerical representation that can be used for subsequent processing, the embodiment of the present invention uses a pre-trained visual encoder ResNet50 to extract image features. ResNet50 is a classic deep residual network that has achieved remarkable achievements in the field of image processing.

[0046] The embodiment of the present invention reconstructs ResNet50 in the multimodal vegetable knowledge graph and interactive system to meet the needs of vegetable knowledge graph construction, and further optimizes it. The specific optimization steps are as follows.

[0047] (1) Optimization of residual block design: The sparsity and diversity of vegetable image features are processed more finely. By introducing the attention mechanism, the model can better capture the key information in the image and improve the accuracy and robustness of feature extraction.

[0048]

[0049] In the formula, represents the attention weight, represents the weight matrix of the query, represents the weight matrix of the key, A weight matrix representing the values, and a softmax function is used to normalize the attention weights.

[0050] (2) Multi-scale feature fusion: In order to better utilize the multi-scale information in the image, the embodiment of the present invention introduces a multi-scale feature fusion module in ResNet50. This module can simultaneously consider image features at different scales, thereby improving the model's perception and expression capabilities for vegetable images.

[0051]

[0052] In the formula, is the fused feature representation, For different scale characteristics, is the weight of the corresponding scale feature.

[0053] (3) Adaptive weight learning: In order to further improve the generalization ability and adaptability of the model, the embodiment of the present invention adopts an adaptive weight learning mechanism, which can automatically learn and adjust the weight distribution in the feature extraction process according to the characteristics of different vegetable images, so that the model can better adapt to vegetable images of different types and styles.

[0054]

[0055] In the formula, is the loss function, For the model, are model parameters, is the regularization term, is the regularization coefficient.

[0056] The entire ResNet50 model and the knowledge graph construction process are end-to-end jointly optimized. By unifying the objective function of knowledge graph construction with the loss function of feature extraction, the global optimization of the model is achieved, and the image information of vegetables is converted into a high-dimensional feature vector (or image feature vector) containing semantic information, so that the model can work better between feature extraction and knowledge graph construction, thereby improving the overall performance and efficiency of the multimodal vegetable knowledge graph and interactive system.

[0057] 2. Extract text features based on the text information of each vegetable

[0058] In the multimodal vegetable knowledge graph and interactive system, RoBERTa is used as the key model for text feature extraction. RoBERTa is a pre-trained language model based on the Transformer architecture with strong semantic understanding and feature expression capabilities. This step converts each piece of text into a corresponding feature vector representation for subsequent processing and analysis.

[0059] RoBERTa uses the Transform structure The layer acts as a text encoder to extract text features, which consists of and of Layers. When the tag sequence Embedded into a word embedding matrix In , the text representation is calculated as follows:

[0060] In the formula, Represents position embedding, which is used to represent the position information of each token in the input sequence, helping the model understand the relative order of tokens. represents the initial embedding matrix, that is, the feature representation of the initial input sequence, Representation layer normalization, used to standardize the hidden layer output, represents a multi-head attention mechanism, which is used to calculate the association between each token in the input sequence. Indicates the previous layer l A hidden state of -1, Represents the fully connected layer, which performs nonlinear transformation and processing on the input features. Indicates l The intermediate output of the layer after multi-head self-attention mechanism and layer normalization, For the input text sequence The hidden state of the layer.

[0061] For each input text, it is input into the RoBERTa model. After a series of encoder layers, a high-level feature representation of the text is obtained, that is, a high-dimensional feature vector (or text feature vector) containing semantic information. These feature vectors contain the semantic information in the text information of vegetables and can capture the meaning and key information of the text. After feature extraction, each text is converted into a fixed-length feature vector. This feature vector can be regarded as a representation of the text in a high-dimensional feature space, which can fully express the semantics and meaning of the text.

[0062] Similarly, if the video information and audio information of each vegetable are obtained, visual feature extraction can be performed based on the video information of each vegetable, such as extracting video frames from the video, and then extracting image features based on the video frames; audio feature extraction can also be performed based on the audio information of each vegetable, such as extracting audio signal features from a segment of audio.

[0063] 3. Align high-dimensional feature vectors across modal knowledge

[0064] Cross-modal knowledge alignment is a key step that aims to effectively model the semantic association between image and text data. The embodiment of the present invention adopts a fine-grained contrastive learning method to achieve cross-modal alignment by measuring the similarity between images and text. The similarity between the image feature vector and the text feature vector is calculated, and a contrast loss function is designed to optimize the model parameters by maximizing the similarity between positive samples and minimizing the similarity between negative samples, thereby achieving effective association between image and text data. The content of cross-modal knowledge alignment is as follows: (1) Multimodal graph structure alignment

[0065] Multimodal graph structure alignment involves representing data of different modalities in the same graph structure and ensuring the correspondence and connectivity between them, including mapping data such as images and text to nodes in the graph structure and establishing edges between them. Using graph matching algorithms, the representation of nodes can be propagated and updated in the graph structure in order to learn the alignment relationship between nodes. In addition, by optimizing the objective function in the graph structure, including the distance loss between nodes, establishing the correspondence between modalities in the graph structure, and optimizing the connectivity between nodes, the semantic association between data of different modalities can be better reflected.

[0066] (2) Multimodal semantic alignment

[0067] Multimodal semantic alignment ensures that the semantic information of data in different modalities can match and complement each other in the graph structure. Through the deep learning model, semantic connections are established between the feature representations of different modalities, and they are aligned in the graph structure to achieve the unity of cross-modal semantics. By learning a shared embedding space, data from different modalities can be compared and associated in this space. Data from different modalities are mapped to a common embedding space so that cross-modal similarity calculations and association analysis can be performed in this space. By optimizing the distance loss in the embedding space, multimodal data can be aligned and fused at the semantic level, thereby providing more accurate and comprehensive information for subsequent knowledge reasoning and decision-making.

[0068] (3) Multi-feature splicing and relationship prediction

[0069] The extracted image and text features are concatenated to generate a richer and more comprehensive feature representation, and these features are used to predict relationships in the graph structure.

[0070] (4) Multimodal knowledge fusion

[0071] The fusion of knowledge from different modalities can produce a more comprehensive and rich agricultural knowledge graph. This process involves integrating and unifying cross-modal information so that the multimodal vegetable knowledge graph and interactive system can better provide users with personalized agricultural services and guidance.

[0072] (5) Cross-modal similarity calculation

[0073] The image feature vector and the text feature vector are used to calculate the similarity between them. The similarity can be measured by cosine similarity or Euclidean distance.

[0074] (6) Contrastive loss function

[0075] In order to achieve effective model training and parameter optimization, the present invention designs a contrast loss function to optimize model parameters by maximizing the similarity between positive samples and minimizing the similarity between negative samples. This contrast loss function can enable the model to learn a more accurate relationship between image and text data, thereby achieving effective alignment of cross-modal knowledge. The expression is as follows:

[0076] In the formula, represents the feature extraction function of the model, Indicates anchoring, represents a positive sample, represents negative samples, represents the L2 norm, is a hyperparameter of the loss function, Indicates taking the positive part.

[0077] Optionally, the multimodal perception-driven vegetable knowledge graph construction method further includes: The graph reasoning algorithm is used to update and maintain the vegetable knowledge graph in real time.

[0078] Specifically, in order to ensure the timeliness and accuracy of the knowledge graph, the present invention designs a dynamic update strategy. Using the graph reasoning algorithm, the knowledge graph is updated and maintained in real time to timely reflect new knowledge and the latest developments in vegetable planting and management.

[0079] Optionally, the multimodal perception-driven vegetable knowledge graph construction method further includes: Obtaining the query command input by the user through the interactive interface; Based on the query instruction, the query result is obtained using the vegetable knowledge graph, and the query result is presented through an interactive interface.

[0080] Optionally, the multimodal perception-driven vegetable knowledge graph construction method further includes: Based on the user's historical query records and feedback information, obtain the content that the user is interested in; Recommending content of interest to the user through the interactive interface.

[0081] Specifically, for the vegetable multimodal knowledge graph, the embodiment of the present invention designs a user-friendly interactive interface to provide a more convenient user experience. The interface allows users to interact with the knowledge graph through text query, voice recognition, image recognition and other methods to obtain personalized agricultural services and guidance.

[0082] In addition to the basic search function, the interactive interface also supports intelligent recommendation functions. By analyzing the user's historical query and feedback information, the multimodal vegetable knowledge graph and interactive system can recommend agricultural information and services related to vegetable planting and management to users, thereby improving user experience and satisfaction.

[0083] Based on the multimodal perception-driven vegetable knowledge graph construction method proposed in the present invention, the multimodal vegetable knowledge graph and interactive system designed in the embodiment of the present invention has the following advantages: 1. Multimodal data fusion: By utilizing multimodal data such as images and texts, and using natural language processing and knowledge representation technology for semantic modeling, the construction of a vegetable knowledge graph is realized, enabling the system to understand and describe vegetable-related information more comprehensively.

[0084] 2. Intelligent interactive services: The vegetable knowledge graph provides users with intelligent decision-making support, including personalized planting guidance, pest and disease identification, agricultural question and answer services, etc. Users can interact with the knowledge graph through text query, voice recognition, image recognition and other methods, realizing personalized agricultural interactive services and guidance.

[0085] 3. Dynamic update strategy: Using graph reasoning algorithm and real-time update strategy, the knowledge graph is dynamically updated and maintained to ensure the timeliness and accuracy of knowledge.

[0086] The embodiment of the present invention uses multimodal data to construct a vegetable knowledge graph, providing intelligent decision support for vegetable production. Through data collection, representation learning and knowledge graph construction, it provides users with personalized planting guidance, pest and disease identification, agricultural question and answer services, and provides agricultural information in a timely and accurate manner through dynamic updates. It also provides a convenient user experience and intelligent recommendation services through a user-friendly interactive interface, further improving the efficiency of agricultural production and enhancing farmers' satisfaction.

[0087] The present invention provides a method for constructing a vegetable knowledge graph driven by multimodal perception. The method obtains multimodal information of each vegetable, takes a vegetable entity as a node, and takes the relationship between multimodal information of different kinds of vegetables as an edge to construct a vegetable knowledge graph, thereby realizing information integration in the vegetable production process and supporting intelligent decision-making; interacts with the knowledge graph in various ways such as text query, voice recognition, and image recognition, thereby realizing personalized agricultural interactive services and guidance, and improving the user's interactive experience; dynamically updates and maintains the knowledge graph by adopting a graph reasoning algorithm and a real-time update strategy, thereby ensuring the timeliness and accuracy of knowledge and reducing the cost of updating and maintenance; thereby improving the effect of knowledge acquisition and decision support, and improving the efficiency and quality of agricultural production.

[0088] A multimodal perception-driven vegetable knowledge graph construction device provided by the present invention is described below. The multimodal perception-driven vegetable knowledge graph construction device described below and the multimodal perception-driven vegetable knowledge graph construction method described above can be referenced to each other.

[0089] Based on any of the above embodiments, Figure 4 is a schematic diagram of the structure of a multimodal perception-driven vegetable knowledge graph construction device provided by the present invention, such as Figure 4 As shown, the embodiment of the present invention provides a multimodal perception-driven vegetable knowledge graph construction device, including an acquisition module 401 and a construction module 402, wherein: The acquisition module 401 is used to obtain the multimodal information of each vegetable; the multimodal information includes at least two of text information, image information, audio information and video information; the construction module 402 is used to construct a vegetable knowledge graph with a vegetable entity as a node and the relationship between the multimodal information of different kinds of vegetables as an edge.

[0090] The present invention provides a multimodal perception-driven vegetable knowledge graph construction device, which constructs a vegetable knowledge graph by acquiring multimodal information of each vegetable, taking a vegetable entity as a node, and taking the relationship between the multimodal information of different kinds of vegetables as an edge, thereby realizing information integration in the vegetable production process and supporting intelligent decision-making; interacting with the knowledge graph through multiple methods such as text query, voice recognition, and image recognition, thereby realizing personalized agricultural interactive services and guidance, and improving the user's interactive experience; dynamically updating and maintaining the knowledge graph by adopting a graph reasoning algorithm and a real-time update strategy, thereby ensuring the timeliness and accuracy of knowledge and reducing the cost of updating and maintenance; thereby improving the effect of knowledge acquisition and decision support, and improving the efficiency and quality of agricultural production.

[0091] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communications interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the vegetable knowledge graph construction method driven by multimodal perception, and the method includes: Acquire multimodal information of each vegetable; the multimodal information includes at least two of text information, image information, audio information and video information; A vegetable knowledge graph is constructed by taking a vegetable entity as a node and the relationship between multimodal information of different kinds of vegetables as edges.

[0092] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0093] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multimodal perception-driven vegetable knowledge graph construction method provided by the above methods, and the method includes: Acquire multimodal information of each vegetable; the multimodal information includes at least two of text information, image information, audio information and video information; A vegetable knowledge graph is constructed by taking a vegetable entity as a node and the relationship between multimodal information of different kinds of vegetables as edges.

[0094] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the multimodal perception-driven vegetable knowledge graph construction method provided by the above methods, the method comprising: Acquire multimodal information of each vegetable; the multimodal information includes at least two of text information, image information, audio information and video information; A vegetable knowledge graph is constructed by taking a vegetable entity as a node and the relationship between multimodal information of different kinds of vegetables as edges.

[0095] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0096] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0097] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0098] "Determine B based on A" in the embodiments of the present application means that the factor A should be considered when determining B. It is not limited to "B can be determined based on A alone", but should also include: "Determine B based on A and C", "Determine B based on A, C and E", "Determine C based on A, and further determine B based on C", etc. In addition, it can also include taking A as a condition for determining B, for example, "When A meets the first condition, use the first method to determine B"; for another example, "When A meets the second condition, determine B", etc.; for another example, "When A meets the third condition, determine B based on the first parameter", etc. Of course, it can also be a condition that takes A as a factor for determining B, for example, "When A meets the first condition, use the first method to determine C, and further determine B based on C", etc.

[0099] In the present invention, the term "plurality" refers to two or more than two, and other quantifiers are similar to them.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a vegetable knowledge graph driven by multimodal perception, characterized in that: include: Get multimodal information about each vegetable; The multimodal information includes at least two of text information, image information, audio information and video information; A vegetable knowledge graph is constructed by taking a vegetable entity as a node and the relationship between multimodal information of different kinds of vegetables as edges.

2. The method for constructing a vegetable knowledge graph driven by multimodal perception according to claim 1 is characterized in that: The relationship between the multimodal information of different types of vegetables includes one or more of the following relationships: Planting location relationship; growing season relationships; soil information relationships; Relationship with meteorological conditions; Disease relationship; Pest relations; fertilizer relations; Pesticide relations; Planting technology relationship.

3. The method for constructing a vegetable knowledge graph driven by multimodal perception according to claim 1 is characterized in that: The method further comprises: Perform multimodal knowledge representation learning on the multimodal information of each vegetable to obtain a high-dimensional feature vector containing semantic information; Perform cross-modal knowledge alignment on the high-dimensional feature vector.

4. The method for constructing a vegetable knowledge graph driven by multimodal perception according to claim 1, characterized in that: The method further comprises: The graph reasoning algorithm is used to update and maintain the vegetable knowledge graph in real time.

5. The method for constructing a vegetable knowledge graph driven by multimodal perception according to claim 1, characterized in that: The method further comprises: Obtaining the query command input by the user through the interactive interface; Based on the query instruction, the query result is obtained using the vegetable knowledge graph, and the query result is presented through an interactive interface.

6. The method for constructing a vegetable knowledge graph driven by multimodal perception according to claim 5 is characterized in that: The method further comprises: Based on the user's historical query records and feedback information, obtain the content that the user is interested in; Recommending content of interest to the user through the interactive interface.

7. A multimodal perception-driven vegetable knowledge graph construction device, characterized in that: include: An acquisition module, used to obtain multimodal information of each vegetable; The multimodal information includes at least two of text information, image information, audio information and video information; The construction module is used to construct a vegetable knowledge graph with a vegetable entity as a node and the relationship between the multimodal information of different types of vegetables as edges.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the multimodal perception-driven vegetable knowledge graph construction method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the method for constructing a vegetable knowledge graph driven by multimodal perception as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the method for constructing a vegetable knowledge graph driven by multimodal perception as described in any one of claims 1 to 6.

Images