Cross-modal knowledge graph construction method and device

Abstract

The invention provides a cross-modal knowledge graph construction method and device, and the method comprises the steps: obtaining text data, image data and structured data of the agricultural industry; performing preprocessing and feature extraction on the text data, the image data and the structured data to obtain a text feature vector, an image feature vector and a structured feature vector; embedding and splicing the text feature vector, the image feature vector and the structured feature vector to obtain an overall feature sequence; the overall feature sequence is input into a multi-mode Transform model, and a comprehensive feature vector output by the multi-mode Transform model is obtained; the comprehensive feature vector is input into a Bi-LSTM-CRF model, and an entity and a relation label output by the Bi-LSTM-CRF model are obtained; a knowledge graph is constructed based on the entities and the relation labels, nodes of the knowledge graph are used for representing the entities, and edges of the knowledge graph are used for representing the relation labels; according to the invention, the real-time performance and the accuracy of the knowledge graph can be improved.

Description

Technical Field

[0001] The present invention relates to the field of knowledge graph technology, and in particular to a cross-modal knowledge graph construction method and device. Background Art

[0002] The varieties, regional characteristics and service information of rural specialty industries are often scattered across different channels and platforms, making it difficult to integrate information, resulting in information asymmetry.

[0003] The existing information system lacks effective cross-domain association and integration capabilities, and the collection, analysis and application of relevant data are insufficient, making it impossible to form effective decision-making support. In addition, the data sources involved in rural social services are wide and diverse, including text, images, video, audio and other modalities. These data have significant differences in format, structure and semantics, making data fusion complex and difficult, resulting in weak correlation between multimodal data.

[0004] It can be seen from this that the knowledge graph construction method in related technologies has the technical problem of being unable to accurately represent the complex relationships between multimodal data. Summary of the invention

[0005] The present invention provides a cross-modal knowledge graph construction method and device to solve the defect that the knowledge graph construction method in the prior art has difficulty in accurately representing the complex relationship between multimodal data, thereby improving the real-time and accuracy of the knowledge graph.

[0006] The present invention provides a cross-modal knowledge graph construction method, comprising the following steps.

[0007] Acquire text data, image data and structured data of the agricultural industry; preprocess and extract features of the text data, image data and structured data respectively to obtain text feature vectors, image feature vectors and structured feature vectors; embed and concatenate the text feature vectors, image feature vectors and structured feature vectors to obtain an overall feature sequence; input the overall feature sequence into a multimodal Transformer model to obtain a comprehensive feature vector output by the multimodal Transformer model; input the comprehensive feature vector into a Bi-LSTM-CRF model to obtain entity and relationship labels output by the Bi-LSTM-CRF model, wherein a mapping relationship exists between the entity and the relationship label; construct a knowledge graph based on the entity and the relationship label, wherein the nodes of the knowledge graph are used to represent the entities, and the edges of the knowledge graph are used to represent the relationship labels.

[0008] According to a cross-modal knowledge graph construction method provided by the present invention, the text data, the image data and the structured data are preprocessed and feature extracted respectively to obtain a text feature vector, an image feature vector and a structured feature vector, including: segmenting and annotating the text data to obtain processed text data, wherein the processed text data includes the variety name, text description, time and place; inputting the processed text data into a BERT model to obtain a text feature vector output by the BERT model; preprocessing the image data to obtain processed image data, wherein the processed image data includes the variety name, image file, the time and the place; inputting the processed image data into a residual network to obtain an image feature vector output by the residual network; classifying and standardizing the structured data to obtain processed structured data, wherein the processed structured data includes the service type, service organization, service object, the time and the place; word embedding the processed structured data to obtain a structured feature vector.

[0009] According to a cross-modal knowledge graph construction method provided by the present invention, the text feature vector, the image feature vector and the structured feature vector are embedded and spliced ​​to obtain an overall feature sequence, including: taking the product of the text feature vector and the embedding layer text weight matrix as the embedded text feature vector; taking the product of the image feature vector and the embedding layer image weight matrix as the embedded image feature vector; taking the product of the structured feature vector and the embedding layer structured weight matrix as the embedded structured feature vector; splicing the embedded text feature vector, the embedded image feature vector and the embedded structured feature vector to obtain an overall feature sequence.

[0010] According to a cross-modal knowledge graph construction method provided by the present invention, the overall feature sequence is input into a multimodal Transformer model to obtain a comprehensive feature vector output by the multimodal Transformer model, including: through the multimodal Transformer model, multimodal feature fusion is performed based on the overall feature sequence to obtain a multimodal query vector, a multimodal key vector and a multimodal value vector; through the multimodal Transformer model, the multimodal query vector, the multimodal key vector and the multimodal value vector are input into a multi-head self-attention mechanism to obtain the multimodal feature fusion attention output by the multi-head self-attention mechanism; through the multimodal Transformer model, the multimodal feature fusion attention is linearly transformed and layer-normalized to obtain a comprehensive feature vector.

[0011] According to a cross-modal knowledge graph construction method provided by the present invention, after constructing the knowledge graph based on the entities and the relationship labels, the method further includes: in response to identifying a new service organization, determining the confidence of the new service organization by using a Bayesian formula; when the confidence exceeds a preset confidence threshold, treating the new service organization as a valid service organization; in response to identifying a new service record, determining the maximum similarity between the new service record and a node of the knowledge graph; when the maximum similarity exceeds a preset similarity threshold, treating the new service record as a valid service record.

[0012] According to a cross-modal knowledge graph construction method provided by the present invention, the method also includes: constructing a new node in the knowledge graph based on the effective service organization, wherein the new node is used to represent the effective service organization; constructing a new edge in the knowledge graph based on the effective service record, wherein the new edge is used to connect the service organization node, variety node and regional node corresponding to the effective service record.

[0013] The present invention also provides a cross-modal knowledge graph construction device, comprising the following modules: An acquisition module is used to acquire text data, image data and structured data of the agricultural industry; a preprocessing module is used to preprocess and extract features of the text data, the image data and the structured data respectively to obtain text feature vectors, image feature vectors and structured feature vectors; a splicing module is used to embed and splice the text feature vectors, the image feature vectors and the structured feature vectors to obtain an overall feature sequence; a fusion module is used to input the overall feature sequence into a multimodal Transformer model to obtain a comprehensive feature vector output by the multimodal Transformer model; a mapping module is used to input the comprehensive feature vector into a Bi-LSTM-CRF model to obtain entity and relationship labels output by the Bi-LSTM-CRF model, wherein a mapping relationship exists between the entity and the relationship label; a construction module is used to construct a knowledge graph based on the entity and the relationship label, wherein the nodes of the knowledge graph are used to represent the entities, and the edges of the knowledge graph are used to represent the relationship labels.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a cross-modal knowledge graph construction method as described in any one of the above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the cross-modal knowledge graph construction methods described above.

[0016] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the cross-modal knowledge graph construction methods described above.

[0017] The cross-modal knowledge graph construction method and device provided by the present invention can fully utilize the complementarity of various data sources to obtain more comprehensive and multi-dimensional agricultural industry information by processing text data, image data and structured data; pre-process and feature extract different types of data to extract key information, and convert it into a numerical form (i.e., feature vector) that can be understood by the model, thereby improving data utilization and the efficiency of subsequent processing; embedding and splicing the feature vectors of text, image and structured data to form an overall feature sequence, and fusing them through a multimodal Transformer model, thereby fully utilizing the correlation and complementarity between data of different modalities and improving the richness and accuracy of feature representation; using the Bi-LSTM-CRF model to identify the entity and relationship labels of the comprehensive feature vector, the mapping relationship between the entity and the relationship label can be extracted; the knowledge graph constructed based on the entity and relationship labels can intuitively display the complex relationship between the entities and their service resources in the agricultural industry, thereby solving the technical problem that the knowledge graph construction method in the related technology is difficult to accurately represent the complex relationship between multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flowchart of the cross-modal knowledge graph construction method provided by the present invention.

[0020] Figure 2 It is an overall architecture diagram of the cross-modal knowledge graph construction method provided by the present invention.

[0021] Figure 3 This is a cross-modal data fusion flow chart provided by the present invention.

[0022] Figure 4 It is a structural schematic diagram of the cross-modal knowledge graph construction device provided by the present invention.

[0023] Figure 5 It is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The varieties, regional characteristics and service information of rural characteristic industries are often scattered across different channels and platforms, making it difficult to integrate information, resulting in information asymmetry. In addition, the existing information system lacks effective cross-domain association and integration capabilities, and the collection, analysis and application of relevant data are insufficient, making it impossible to form effective decision support. A technology that can integrate variety, region and service information scattered across different platforms and channels is needed to form a complete knowledge graph. Cross-modal association of information in different fields such as variety-region-service in the entire industrial chain of rural characteristic agricultural products can achieve richer information linkage and application, and improve the utilization efficiency of service resources.

[0026] Therefore, the present invention realizes the comprehensive integration and sharing of rural characteristic industry information by constructing a cross-modal social service map of "variety-region-service", breaking the information island. The map technology is used to accurately match service demand and supply, and improve the response speed and resource utilization efficiency of rural services.

[0027] In the related technology, a method for realizing service composition optimization by constructing a knowledge graph based on LDA cross-domain is provided. By processing information such as service description and correcting the response service description information, services in different fields can maintain their original characteristics while maintaining associations with services in similar fields, so that the associations between different service entities can be extracted more reasonably and closely connect services in different fields; entities of service field knowledge and the relationships between entities are extracted to construct a knowledge graph, and the service composition problem is modeled using a partially observable Markov decision process to achieve service composition optimization.

[0028] However, the above-mentioned related technologies mainly rely on linear discriminant analysis to process and correct service description information to achieve cross-domain service association. However, when processing text, linear discriminant analysis may not be able to fully capture the complex semantic relationships unique to the domain, resulting in inaccurate and in-depth cross-domain associations.

[0029] In the related technology, a multimodal knowledge graph construction method for the basin flood prevention four-prevention platform is provided, which obtains image data and text data from the basin flood prevention four-prevention platform data base, extracts the text data entity relationship of the basin flood prevention four-prevention platform, interprets and classifies the preprocessed image data, and performs multimodal data fusion; constructs the basin flood prevention four-prevention platform knowledge graph. Convert various forms of data into structured information and establish a knowledge graph to provide knowledge support for the four-prevention digital twin system.

[0030] However, the above-mentioned related technologies are mainly aimed at the field of river basin flood control, and the data types, processing methods and application scenarios involved are highly specialized. This makes the method less applicable in other fields and difficult to directly apply to different fields such as rural characteristic industries. In addition, the lack of data processing methods for dynamic changes may cause the knowledge graph to fail to reflect the latest information in a timely manner.

[0031] In the related technology, a method for constructing a multimodal knowledge graph is provided. First, the business data of the target business scenario is obtained, and a scenario ontology library is constructed based on the business data. The first correspondence between text entities in the text data and the second correspondence between image entities in the target video data are determined, and the third correspondence between the labeled signal entities is marked. The various correspondences are aligned and fused, and a multimodal knowledge graph is constructed based on the fusion results.

[0032] However, the above-mentioned related technologies are mainly aimed at building enterprise information data and industrial chain relationships. Therefore, when applied to rural characteristic industries, the models and algorithms may need to be adjusted to adapt to the specific characteristics and needs of rural industries. The method mentioned in the patent mainly processes text data, and there is no detailed description of the processing of multimodal data (such as images, audio, geographic information, etc.) involved in rural characteristic industries. This limitation may lead to less than ideal results when integrating multimodal information of rural industries.

[0033] In the related technology, a method for automatic graph construction is provided, which collects and preprocesses massive enterprise information data, identifies entities and entity relationships in the data, constructs and optimizes the industrial chain graph, and provides industrial chain graph query, industrial chain graph display, and industrial chain graph update services; the invented construction method uses the BERT-BiLSTM-CRF model as the industrial chain entity recognition model and Transformer as the relationship recognition model, and uses the support vector machine SVM model to adjust and optimize the industrial chain graph, which greatly improves the automation and accuracy of the industrial chain graph.

[0034] However, the above-mentioned related technologies are mainly aimed at building enterprise information data and industrial chain relationships. Therefore, when applied to rural characteristic industries, the models and algorithms may need to be adjusted to adapt to the specific characteristics and needs of rural industries. In addition, the above-mentioned methods mainly process text data, and there is no detailed description of the processing of multimodal data (such as images, audio, geographic information, etc.) involved in rural characteristic industries. This limitation may lead to less than ideal results when integrating multimodal information of rural industries.

[0035] In the related technology, a knowledge graph construction method for crop variety management is provided. The method constructs a domain ontology of crop variety management based on the subject word dictionary of crop variety management; obtains the corresponding crop variety management data, extracts knowledge from the variety management data, expands the triples through knowledge reasoning, and generates different knowledge graph data display interfaces for each user. This method reduces the degree of manual participation in the knowledge graph construction process and improves the level of automated construction.

[0036] However, the above-mentioned related technologies mainly process text data and do not fully consider the processing needs of multimodal data (such as images, audio, geographic information, etc.) in rural characteristic industries. The lack of processing and fusion of multimodal data may limit the comprehensiveness and practicality of the atlas. And the method does not specify how to handle the real-time update of data. The information of rural characteristic industries changes frequently, and there is a lack of efficient dynamic update mechanism. The atlas may not be able to reflect the latest industry information in a timely manner.

[0037] In summary, the existing relevant technologies have their own innovations in the construction of knowledge graphs, but they have some common defects, including field specialization and limited applicability. They are mainly concentrated in specific fields (such as river basin flood control, industrial chains, and crop variety management) rather than rural characteristic industries; there is insufficient processing of multimodal data, making it difficult to fully integrate and share multiple data types; there is a lack of dynamic update mechanism, which may lead to outdated graphs.

[0038] This invention proposes a cross-modal knowledge graph construction method, which is applied to rural social services. The purpose is to achieve comprehensive integration and sharing of rural characteristic industry information and break the information island by constructing a cross-modal social service graph of "variety-region-service". This technology integrates and connects multiple data types (such as text, images, geographic information, etc.) to ensure the comprehensiveness and accuracy of information. At the same time, a dynamic update mechanism is adopted to reflect the latest industry information in real time, reproduce and update the knowledge graph, thereby improving the response speed and resource utilization efficiency of rural services.

[0039] Optionally, the cross-modal knowledge graph construction method of the embodiment of the present application can be executed by a server, or by a terminal device, or jointly by a server and a terminal device, taking the cross-modal knowledge graph construction method in this embodiment executed by a server as an example.

[0040] Figure 1 is a flow chart of the cross-modal knowledge graph construction method provided by the present invention, such as Figure 1 As shown, the method includes the following steps.

[0041] Step 101, obtaining text data, image data and structured data of the agricultural industry.

[0042] In the embodiment of the present invention, rural social service resources involve text, images and structured information, and various data types including text, images, geographic information, etc. are collected from multiple data sources.

[0043] Among them, text data includes variety name, text description, time, and location; image data includes variety name, image file, time, and location; structured data includes service type, service organization, service object, time, and location

[0044] Step 102 , preprocessing and feature extraction are performed on the text data, image data and structured data respectively to obtain a text feature vector, an image feature vector and a structured feature vector.

[0045] Data preprocessing is a basic step in cross-modal data fusion. Rural social service resources involve the standardized processing of text, images and structured information.

[0046] For text data, we first use natural language processing (NLP) tools (such as Stanford NLP or spaCy) to perform word segmentation and part-of-speech tagging, segment the text describing varieties (such as a certain crop) and service resources (such as a certain social service), then remove noise data such as stop words and special characters, and then use the BERT model to convert the processed text into a high-dimensional vector representation, namely the text feature vector.

[0047] For image data, image enhancement techniques (such as rotation, flipping, and cropping) are applied to increase data diversity, and then pixel values ​​are normalized (values ​​are adjusted to the [0,1] range) and the image size is adjusted to ensure consistency. Image feature extraction uses a convolutional neural network to obtain an image feature vector.

[0048] For structured data, we process the data content, classify and standardize the variety descriptions, and ensure the consistency of different data sources. We unify the naming and description of the same variety in different places. Geographic information preprocessing converts geographic information into a standard format of longitude and latitude, and standardizes the geographic information through the Geographic Information System (GIS) tool. We classify and standardize the service types, ensure the consistency of descriptions of different data sources, and unify the classification and naming of agricultural technical services with different descriptions.

[0049] Step 103, embed and concatenate the text feature vector, the image feature vector and the structured feature vector to obtain an overall feature sequence.

[0050] For the social service data of rural characteristic industries, the main data includes three modalities: text, image and structured data. The data of each modality has been converted into feature vectors (text feature vectors, image feature vectors and structured feature vectors) after preprocessing.

[0051] In the embodiment of the present invention, firstly, an embedding layer is used to convert feature vectors of different modalities into embedded representations of the same dimension, and secondly, the embedded text, image and service features are concatenated into an overall feature sequence.

[0052] Step 104, input the entire feature sequence into the multimodal Transformer model to obtain a comprehensive feature vector output by the multimodal Transformer model.

[0053] In an embodiment of the present invention, a multimodal Transformer model is designed for feature encoding. The multimodal Transformer fuses features of different modalities through a self-attention mechanism to obtain a comprehensive feature vector.

[0054] Among them, the multimodal Transformer model is a neural network architecture based on the self-attention mechanism, which can process multimodal data (such as images, text, audio, etc.).

[0055] For example, a multimodal Transformer model includes a modality encoder and a Transformer layer, where the modality encoder is used to convert data from different modalities into a unified representation. For example, an image encoder can convert an image into a series of vector representations, and a text encoder can convert text into word embedding vectors. The Transformer layer consists of multiple self-attention layers and feed-forward layers to determine dependencies and complex features in the data. In a multimodal Transformer, these layers can process data from multiple modalities simultaneously and interact and fuse them through attention mechanisms.

[0056] Step 105, input the comprehensive feature vector into the Bi-LSTM-CRF model to obtain the entity and relationship labels output by the Bi-LSTM-CRF model, wherein there is a mapping relationship between the entity and the relationship label.

[0057] In the embodiment of the present invention, the text features, image features and structured features have been combined to generate a comprehensive feature vector through a multimodal Transformer model. Here, a Bi-LSTM-CRF model is used to input a comprehensive feature vector and output entity and relationship labels, wherein during the training process, the model learns the mapping relationship from the comprehensive feature vector to the entity and relationship labels.

[0058] Among them, the Bi-LSTM-CRF model is a classic model for sequence labeling tasks in the field of natural language processing (NLP), which combines the bidirectional long short-term memory network (Bi-LSTM) and the conditional random field (CRF).

[0059] Bidirectional long short-term memory network is a special recurrent neural network (RNN) that can capture contextual information in both the forward and backward directions of a sequence. This feature enables Bi-LSTM to more comprehensively understand the position and role of each element in the sequence when processing sequence data, thereby improving the accuracy of label prediction.

[0060] Conditional Random Field is a discriminative model for sequence labeling, which can output an optimal label sequence under the condition of a given input sequence. CRF ensures that the predicted label sequence is more semantically reasonable by considering the dependencies between labels.

[0061] The Bi-LSTM-CRF model performs sequence labeling through the following steps: Input layer: Convert sequence data into vector representation, usually using word embedding technology to convert each word into a vector of fixed dimension.

[0062] Bi-LSTM layer: The vector sequence of the input layer is passed to the Bi-LSTM layer. The Bi-LSTM layer captures the context information of the sequence through the LSTM network in the forward and backward directions respectively, and fuses this information to obtain the hidden state representation of each position.

[0063] CRF layer: The output of the Bi-LSTM layer is used as the input of the CRF layer. The CRF layer calculates the probability of each label sequence by determining the dependencies between labels, and selects the label sequence with the largest probability as the output.

[0064] Step 106, construct a knowledge graph based on entity and relationship labels, wherein the nodes of the knowledge graph are used to represent entities, and the edges of the knowledge graph are used to represent relationship labels.

[0065] In the embodiment of the present invention, nodes are created according to the identified entities, and edges are created according to the identified relationships. A graph database (such as Neo4j) is used to store the knowledge graph. Nodes and edges are stored in the form of a graph structure.

[0066] refer to Figure 2 , Figure 2 It is the overall architecture diagram of the cross-modal knowledge graph construction method provided by the present invention, which includes: cross-modal data fusion, "variety-region-service" graph construction and dynamic update mechanism.

[0067] Among them, cross-modal data fusion includes data preprocessing, feature extraction, modal alignment and modal fusion; "variety-region-service" graph construction includes entity recognition, relationship extraction, graph construction and graph representation; dynamic update mechanism includes data monitoring, threshold comparison and graph update.

[0068] The present invention relates to a cross-modal knowledge graph construction method, which realizes the comprehensive integration and sharing of rural characteristic industry information through multi-source data fusion, "variety-region-service" graph construction and dynamic update mechanism. This technology integrates multiple data types (such as text, images, geographic information, etc.) to ensure the comprehensiveness and accuracy of information, while dynamically updating the graph to highlight the field characteristics of rural socialized service resources. The main scheme includes three parts: cross-modal data fusion, "variety-region-service" graph construction and dynamic update mechanism.

[0069] Step 1: Cross-modal data fusion: First, multiple data types including text, images, and geographic information are collected from multiple data sources. The collected data is cleaned and standardized to ensure data quality. Then, multiple data types are fused and connected through a specific multimodal Transformer model to form a unified data integration model, realize cross-modal fusion of data, and lay the foundation for graph construction.

[0070] Step 2: Construction of the “Variety-Region-Service” Map: Based on the fused data, a multi-dimensional data model is designed, covering varieties (such as crops, livestock varieties, etc.), regions (such as geographical information of villages, counties, cities, etc.) and services (such as agricultural technical services, logistics, market information, etc.). Define various nodes in the map, including variety nodes (recording the biological characteristics of the variety, planting / breeding conditions, etc.), regional nodes (recording geographical location, climate conditions, soil type, traffic conditions, etc.) and service nodes (recording service content, providers, service scope, etc.). By defining the relationship between variety and region, region and service, and variety and service, a complete "variety-region-service" cross-modal social service map is constructed.

[0071] Step 3: Dynamic update mechanism: Finally, a regular data collection mechanism and real-time update algorithm are established to ensure that the map can reflect the latest information changes. A user feedback mechanism is introduced, and a user feedback module is designed to allow users to submit information feedback to further improve and update the map content. Through these dynamic update measures, the timeliness and accuracy of the map are maintained, and the allocation and utilization of rural social service resources are optimized.

[0072] Through the above steps of the embodiment of the present invention, text data, image data and structured data of the agricultural industry are obtained; the text data, image data and structured data are preprocessed and feature extracted respectively to obtain text feature vectors, image feature vectors and structured feature vectors; the text feature vectors, image feature vectors and structured feature vectors are embedded and spliced ​​to obtain an overall feature sequence; the overall feature sequence is input into a multimodal Transformer model to obtain a comprehensive feature vector output by the multimodal Transformer model; the comprehensive feature vector is input into a Bi-LSTM-CRF model to obtain entity and relationship labels output by the Bi-LSTM-CRF model, wherein there is a mapping relationship between the entity and the relationship label; a knowledge graph is constructed based on the entity and relationship labels, wherein the nodes of the knowledge graph are used to represent entities, and the edges of the knowledge graph are used to represent relationship labels; the complex relationship between entities and their service resources in the agricultural industry can be intuitively displayed, thereby solving the technical problem that the knowledge graph construction method in the related technology is difficult to accurately represent the complex relationship between multimodal data.

[0073] According to a cross-modal knowledge graph construction method provided by the present invention, text data, image data and structured data are preprocessed and feature extracted respectively to obtain text feature vectors, image feature vectors and structured feature vectors, including: Segmenting and annotating the text data to obtain processed text data, wherein the processed text data includes variety name, text description, time, and location; Input the processed text data into the BERT model to obtain the text feature vector output by the BERT model; Preprocessing the image data to obtain processed image data, wherein the processed image data includes variety name, image file, time, and location; Input the processed image data into the residual network to obtain the image feature vector output by the residual network; Classifying and standardizing the structured data to obtain processed structured data, wherein the processed structured data includes service type, service organization, service object, time, and location; The processed structured data is word embedded to obtain a structured feature vector.

[0074] In the embodiment of the present invention, data preprocessing is a basic step in cross-modal data fusion, and rural social service resources involve standardized processing of text, images and structured information.

[0075] For text data, we first use natural language processing tools (such as StanfordNLP or spaCy) to perform word segmentation and part-of-speech tagging, segment the text describing varieties (such as a certain crop) and service resources (such as a certain social service), then remove noise data such as stop words and special characters, and then use the BERT model to convert the processed text into a high-dimensional vector representation.

[0076] For example, the text t is converted into a high-dimensional vector representation T through T=BERT(t), where t represents the input text data, T represents the text feature vector, and BERT represents the pre-trained language representation model.

[0077] For image data, image enhancement techniques (such as rotation, flipping, and cropping) are applied to increase data diversity, and then pixel values ​​are normalized (adjusted to the [0,1] range) and the image size is adjusted to ensure consistency. Image feature extraction uses a convolutional neural network ResNet.

[0078] For example, the feature vector I of image i is extracted by I=CNN(i), where i represents image data, I represents the image feature vector, and CNN represents a convolutional neural network.

[0079] For structured data, the data content of structured data is processed, and the variety descriptions are classified and standardized to ensure the consistency of different data sources. The same variety in different places is uniformly named and described. Geographic information preprocessing converts geographic information into a standard format of longitude and latitude, and standardizes geographic information through geographic information system tools. Service types are classified and standardized to ensure the consistency of descriptions of different data sources, and agricultural technical services with different descriptions are uniformly classified and named.

[0080] The first-level classification of agricultural social services is divided into agricultural input supply services, seed breeding and promotion services, agricultural production services, agricultural technology promotion services, animal and plant disease prevention and control services, agricultural product quality and safety services, agricultural product processing services, agricultural product circulation services, agricultural information services, agricultural finance and insurance services, rural property rights management and transaction services, agricultural service organization construction and management services, agricultural legal services, and other agricultural social services. Under the first-level classification, specific service types are further defined as second-level types.

[0081] In the embodiment of the present invention, the Json format data processing sample (i.e., processed text data, processed image data, processed structured data) is as follows: Text data: {"Variety name":"Mincha 25","Description":"Variety description text","Time":"2023-06-10T10:00:00Z","Location":[34.0522,-118.2437]}

[0082] Image data: {"variety name":"Mincha 25","image file":image1.jpg,"time":"2023-06-10T10:05:00Z","location":[34.0522,-118.2437]}

[0083] Structured data: {"Service type":Agricultural production service-flying defense,"Service organization":"Ecological Tea Co., Ltd.","Service object":"Ecological Family Farm","Time":"2023-06-10T10:10:00Z","Location":[34.0522,-118.2437]}

[0084] After feature extraction, the text data is output as a 768-dimensional feature vector, and the image data is output as a 2048-dimensional feature vector.

[0085] Feature engineering is performed on structured data, and the discrete classification features of service type and service organization are mapped into continuous vector representations using embedding layer technology, which are represented as 16-dimensional, 128-dimensional and 128-dimensional vectors in the present invention.

[0086] {"Variety Name":"Mincha 25","Description Characteristics":[0.1,0.2,...,0.768],"Time":"2023-06-10T10:00:00Z","Location":[34.0522,-118.2437]}

[0087] {"Variety Name":"Mincha 25","Image Features":[0.05,0.1,...,0.2048],"Time":"2023-06-10T10:05:00Z","Location":[34.0522,-118.2437]}

[0088] {"Service Type":"Agricultural Production Service-Flying Defense","Service Organization":"Ecological Tea Co., Ltd.","Service Target":"Ecological Family Farm","Time":"2023-06-10T10:10:00Z","Location":[34.0522,-118.2437],"Service Feature Vector": [0.01,0.02,...,0.50,0.05,0.10,...,0.50,0.02,0.04,...,0.50,1686393000.0,34.0522,-118.2437]

[0089] }

[0090] Among them, the service feature vector (i.e., structured feature vector) is 272 dimensions.

[0091] According to a cross-modal knowledge graph construction method provided by the present invention, text feature vectors, image feature vectors and structured feature vectors are embedded and spliced ​​to obtain an overall feature sequence, including: The product of the text feature vector and the text weight matrix of the embedding layer is used as the embedded text feature vector; The product of the image feature vector and the embedding layer image weight matrix is ​​used as the embedded image feature vector; The product of the structured feature vector and the structured weight matrix of the embedding layer is used as the embedded structured feature vector; The embedded text feature vector, the embedded image feature vector and the embedded structured feature vector are concatenated to obtain the overall feature sequence.

[0092] For the socialized service data of rural characteristic industries, the main data include three modalities: text, image and structured (service) data. The data of each modality has been converted into feature vectors after preprocessing.

[0093] In the embodiment of the present invention, modal fusion is a process of combining feature vectors of different modalities into a joint representation. First, an embedding layer is used to convert feature vectors of different modalities into representations of the same dimension. For details, please refer to the following formulas (1) to (3):

[0094] T em =W T ·T (1)

[0095] Among them, Tem represents the embedded text feature vector, W T represents the embedding layer text weight matrix, and T represents the text feature vector.

[0096] I em =W I ·I (2)

[0097] Among them, I em represents the embedded image feature vector, W I represents the embedding layer image weight matrix, and I represents the image feature vector.

[0098] G em =W G ·G (3)

[0099] Among them, G em represents the embedded structured feature vector, W G represents the structured weight matrix of the embedding layer, and G represents the structured feature vector.

[0100] In the embodiment of the present invention, W T ,W I ,W G is the weight matrix of the embedding layer. Through this transformation, the feature vectors T, I, G of different modalities have the same dimension, which is convenient for subsequent fusion operations.

[0101] In the embodiment of the present invention, the text, image and structured (service) data features are converted into 1024-dimensional feature vectors; the embedded text, image and service features are spliced ​​into an overall feature sequence, which can be specifically referred to the following formula (4).

[0102] X=[T em , I em ; G em ] (4)

[0103] Among them, X represents the overall feature sequence, T em represents the embedded text feature vector, I em represents the embedded image feature vector, G em Represents an embedded structured feature vector.

[0104] According to a cross-modal knowledge graph construction method provided by the present invention, the overall feature sequence is input into a multimodal Transformer model to obtain a comprehensive feature vector output by the multimodal Transformer model, including:

[0105] Through the multimodal Transformer model, multimodal features are fused based on the overall feature sequence to obtain a multimodal query vector, a multimodal key vector, and a multimodal value vector.

[0106] Through the multimodal Transformer model, the multimodal query vector, the multimodal key vector and the multimodal value vector are input into the multi-head self-attention mechanism to obtain the multimodal feature fusion attention output by the multi-head self-attention mechanism;

[0107] Through the multimodal Transformer model, the multimodal feature fusion attention is linearly transformed and layer-normalized to obtain a comprehensive feature vector.

[0108] In the embodiment of the present invention, position coding is added to the overall feature sequence after splicing, and the sequence order information is retained by introducing the position coding. The position coding can refer to the following formula (5) and formula (6):

[0109]

[0110] Among them, PE(pos,2i) and PE(pos,2i+1) are used to represent the position encoding of position pos and dimension 2i, 2i+1 respectively, pos represents the position in the sequence, i represents the dimension index in the position encoding, and d represents the dimension of the feature.

[0111] The new feature sequence obtained by adding the input feature sequence X to the position code PE can be specifically referred to the following formula (7).

[0112] X pos =X+PE (7)

[0113] Among them, X pos Represents the new feature sequence obtained by adding the input feature sequence X to the position code PE.

[0114] In an embodiment of the present invention, a multimodal Transformer model is designed for feature encoding. The multimodal Transformer model fuses features of different modalities through a self-attention mechanism. For details, please refer to the following formulas (8) to (10):

[0115] Q T =W Q ·T em K T =W K ·T em V T =W V ·T em (8)

[0116] Q I =W Q I em K I =W K I em VI =W V I em (9)

[0117] Q G =W Q ·G em K G =W K ·G em V G =W V ·G em (10)

[0118] Where Q represents query, K represents key, V represents value, and W represents weight matrix; T, I, and G represent text modality, image modality, and structured modality respectively; T em represents the embedded text feature vector, I em represents the embedded image feature vector, G em Represents an embedded structured feature vector.

[0119] Input the query, key, and value vectors into the self-attention mechanism and calculate the attention weight. For details, please refer to the following formula (11):

[0120]

[0121] Among them, Attention represents the self-attention mechanism, Q, K, and V represent the query vector, key vector, and value vector respectively; d k represents the dimension of the key vector and T represents the transposed matrix.

[0122] In the embodiment of the present invention, the query, key and value vectors of different modalities are respectively input into the self-attention mechanism for calculation, and then their outputs are weighted summed. The correlation between different modalities can be retained, and the information of different modalities can be fused together. For details, refer to the following formula (12):

[0123] Attention multi =α1Attention(Q1,K1,V1)+α2Attention(Q2,K2V2)+α3Attention(Q3,K3,V3)…………(12) Among them, Attention multi represents multi-head attention; α1, α2, α3 represent the weights of text, image and structured (service) data respectively; Attention(Q1, K1, V1) represents text attention, Attention(Q2, K2, V2) represents image attention, and Attention(Q3, K3, V3) represents structured attention.

[0124] In the embodiment of the present invention, a multi-head self-attention mechanism is used to calculate multiple groups of attention in parallel. For details, please refer to the following formula (13):

[0125] MultiHead(Q,K,V)=Concat(head1,head2,…,headh)W o (13)

[0126] Among them, MultiHead(Q,K,V) represents a multi-head self-attention mechanism, Q, K, V represent the query matrix, key matrix and value matrix respectively, Concat means concatenating the outputs of multiple heads along a certain dimension (usually the last dimension), head1 represents the first head, head2 represents the second head, headh ​​represents the hth head, h represents the number of heads, that is, the number of attention functions running in parallel, and W o Represents the weight matrix of the linear transformation, used to project the concatenated result back to the original dimension or the required dimension.

[0127] Each head can be regarded as the calculation of the attention output after the fusion of different modal features. For details, please refer to the following formula (14):

[0128] headi=Attentionmulti(Q i ,K i ,V i ) (14)

[0129] Among them, headi represents the i-th head, Attentionmulti(Q i ,K i ,V i ) represents the attention output after the fusion of different modal features.

[0130] In the embodiment of the present invention, the attention output after the fusion of multimodal features is linearly transformed and layer-normalized to obtain the final comprehensive feature vector, which can be specifically referred to the following formula (15):

[0131] Z = LayerNorm(Attention(Q T ,K T ,V T )+Attention(Q I ,K I ,V I )+

[0132] Attention(Q i ,K i ,V i ))(15)

[0133] Among them, Z represents the comprehensive feature vector, LayerNorm represents layer normalization, Attention(Q T ,K T ,V T ) represents the text modality self-attention, Attention(Q I ,K I ,V I ) represents the image modality self-attention, Attention(Q i ,K i ,V i ) represents structured modal self-attention.

[0134] In an embodiment of the present invention, a multimodal alignment task is designed to learn the association and co-occurrence relationship between data of different modalities (text, image, structured data), wherein the contrast loss function can refer to the following formula (16):

[0135] L contrastive =∑ (i,j)∈P max(0,ms ij )+∑ (i,j)∈N s ij (16)

[0136] Among them, L contrastive represents the contrast loss function, P represents the set of positive sample pairs (similar data pairs), N represents the set of negative sample pairs (dissimilar data pairs), m represents the boundary hyperparameter used to control the separation between positive sample pairs and negative sample pairs, and s ij Represents different modal data x i and x j The similarity is calculated by the dot product of the comprehensive feature vectors.

[0137] refer to Figure 3 , Figure 3 It is a cross-modal data fusion flow chart provided by the present invention, which includes: data preprocessing and feature extraction, feature embedding and multimodal Transformer model training.

[0138] Data preprocessing and feature extraction include text preprocessing through the BERT model to obtain text features, image preprocessing through ResNet to obtain image features, and structured data preprocessing through the embedding layer to obtain structured data features.

[0139] Feature embedding includes concatenating text features (1024 dimensions), image features (1024 dimensions), and structured data features (1024 dimensions) to obtain the overall feature sequence X = [T em , I em ; Gem ].

[0140] Multimodal Transformer training includes position encoding, self-attention mechanism, multi-head self-attention mechanism, comprehensive feature vector (Z) and multimodal alignment task. The above process can refer to the above embodiment, and the present invention will not be repeated here.

[0141] According to a cross-modal knowledge graph construction method provided by the present invention, before inputting the comprehensive feature vector into the Bi-LSTM-CRF model to obtain the entity and relationship labels output by the Bi-LSTM-CRF model, the method further includes:

[0142] Build a graph based on variety entities, regional entities, and service entities.

[0143] In an embodiment of the present invention, entities in a graph are designed and relationships are defined to construct a basic framework of the graph, wherein some entities and some relationships are designed as shown below.

[0144] Variety: Contains information on the type of agricultural product, such as name, classification, biological characteristics of the variety, planting (breeding) conditions, etc.

[0145] Region: involves attributes related to geographic information, such as place names, longitude and latitude, climate, soil type, etc.

[0146] Service Provider: includes detailed information about the service provider, such as name, type (such as supply and marketing cooperatives, farmers' professional cooperatives, etc.), business status, region, introduction, service items, etc.

[0147] Production entity: includes detailed information of the production entity, such as name and description.

[0148] Breed-area relationships: describe the association between a particular breed and the area in which it is distributed.

[0149] Region-Service Provider Relationship: describes the geographical location of the service provider and the region that the service provider provides services to.

[0150] Service subject-production subject: describes the services provided by the service subject to the production subject, such as service time, service type, etc.

[0151] Service Entity-Variety: Describes the types of services provided by the service entity.

[0152] According to a cross-modal knowledge graph construction method provided by the present invention, the comprehensive feature vector is input into the Bi-LSTM-CRF model to obtain the entity and relationship labels output by the Bi-LSTM-CRF model.

[0153] In some embodiments, entity relationships are identified, and a comprehensive feature vector is generated by combining text features, image features, and structured features through a multimodal Transformer model. Here, a Bi-LSTM-CRF model is used, which inputs a comprehensive feature vector and outputs entity and relationship labels. During the training process, the model learns the mapping relationship from the comprehensive feature vector to the entity and relationship labels.

[0154] The comprehensive feature vector X = [x1, x2, ..., x n ] is input into the Bi-LSTM-CRF model, where Bi-LSTM is used to process time series data and capture context information. For details, please refer to the following formulas (17) to (19):

[0155] Front and back LSTM

[0156] Backward LSTM

[0157] Bidirectional LSTM output:

[0158] Among them, x t is the representation of the input comprehensive feature vector at time step t, h t is the hidden state vector at time step t.

[0159] The output hidden state vector of Bi-LSTM is mapped to the relation label space through a fully connected layer. For details, please refer to the following formula (20):

[0160] o t =W·h t +b (20) Among them, t is the output vector, W is the weight matrix, h t is the hidden state vector at time step t, and b is the bias term. The CRF layer is used to globally optimize the output of the fully connected layer to ensure the legitimacy of the output relationship label sequence. The scoring function is as follows: the score of the given input sequence X and the output label sequence y is calculated. For details, please refer to the following formula (21): Among them, emit(y t ,o t ) indicates label y t The probability at time step t, trans(y t-1 ,y t ) indicates that from label y t-1 Transfer to y t probability. The identified entities or relationships are finally output, for example, "service relationship = [("Ecological Tea Co., Ltd.", "provide", "flying defense service", "ecological family farm")]". According to a cross-modal knowledge graph construction method provided by the present invention, a knowledge graph is constructed based on entity and relationship labels. Create nodes based on the identified entities and edges based on the identified relationships. Use a graph database (such as Neo4j) to store the knowledge graph. Store nodes and edges in a graph structure. CREATE(p: variety {name: 'Mincha 25'}) CREATE(r: region {name: 'Fujian Province'}) CREATE(s:service subject{name:'Ecological Tea House',type:'Supply and Marketing Cooperative',status:'Active'}) CREATE (p)-[:located]->(r) Next, the cross-modal fused feature vector (i.e., comprehensive feature vector) is applied to the representation of the knowledge graph and combined with the embedding of the TransE algorithm. The TransE algorithm is used to embed entities and relationships in the knowledge graph and embeds them into a low-dimensional vector space using the following formula (22): score(h,r,t)=‖h+rt‖ (22) Among them, h is the embedding vector of the head entity, r is the embedding vector of the relation, and t is the embedding vector of the tail entity. In order to combine the feature vector of cross-modal fusion with the embedding of TransE, assume that the comprehensive feature vector is v t , the embedding vector of TransE is h, which can be fused by the following formula (23): v f =α·v t +β·h (23) Among them, α and β are fusion coefficients, v t represents the comprehensive feature vector, h represents the embedding vector of TransE, controls the weights of the comprehensive feature vector and the embedding vector, and the fused feature vector v f (i.e. fused feature vectors). In the embodiment of the present invention, the fusion feature vector v f Can be stored in a knowledge graph as the final representation of an entity or relationship, for example:

[0161] "Entity Identifier": "Mincha25",

[0162] "Fused feature vector": [0.20, 0.30, ..., 0.85]

[0163] }

[0164] In the query and reasoning process, the fused feature vectors are used for similarity calculation and relationship inference. This method makes full use of the rich features of multimodal data fusion to improve the representation ability and reasoning accuracy of the knowledge graph.

[0165] According to a cross-modal knowledge graph construction method provided by the present invention, after constructing the knowledge graph based on entity and relationship tags, the method further includes: In response to identifying a new service organization, determining the confidence level of the new service organization by using a Bayesian formula; When the confidence exceeds a preset confidence threshold, the new service organization is regarded as a valid service organization; In response to identifying a new service record, determining a maximum similarity between the new service record and a node of the knowledge graph; When the maximum similarity exceeds a preset similarity threshold, the new service record is taken as a valid service record.

[0166] In the embodiment of the present invention, when new service organizations and service record data appear, it is necessary to add these new information to the existing "category-region-service" map through a dynamic update mechanism.

[0167] Data change detection is used to identify newly emerged service organizations and service records. Embodiment of the present invention. A probability and threshold-based method is introduced to implement it, and a probability model is introduced to evaluate the effectiveness of new service organizations. The Bayesian formula is used to calculate the confidence of the new service organization, and a threshold is set for screening. For details, please refer to the following formula (24): (twenty four)

[0168] in, The given data After-sales service organization The probability of occurrence, The given data In service organizations The probability of existence, Is a service organization The prior probability of occurrence, The given data The total probability is, if Exceeding the set threshold (pre-set confidence threshold), the service organization is considered It is an effective new service organization.

[0169] The goal of new service record detection is to identify new service records and their relationship with varieties and regions. Graph matching algorithms and semantic analysis are used to verify the validity of new service records. The semantic similarity between new service records and related nodes in the existing graph is calculated to determine their relevance. For details, please refer to the following formula (25).

[0170] (25)

[0171] in, represents the maximum similarity between the service record r and the nodes in the graph G, represents the cosine similarity between service record r and graph node g. Exceeding the set threshold (preset similarity threshold), the service record r is considered to be a valid new service record.

[0172] According to a cross-modal knowledge graph construction method provided by the present invention, the method further includes: Building new nodes in the knowledge graph based on effective service organizations, where the new nodes are used to represent effective service organizations; New edges are constructed in the knowledge graph based on valid service records, where the new edges are used to connect service organization nodes, product nodes, and region nodes corresponding to the valid service records.

[0173] In an embodiment of the present invention, the graph is updated based on the result of change detection. The graph update includes adding new nodes and adding new edges. For new service organizations detected, new nodes are added to the graph. For new service records detected, new edges are added to the graph to connect the corresponding service organizations with the variety and region nodes.

[0174] The dynamic weight adjustment mechanism of the edge is adopted to dynamically adjust the weight of the edge according to the frequency and importance of the service record. The weight of connecting service organization s and product / region node p is , please refer to the following formula (26): (26)

[0175] in, is the frequency of service logging, The importance of service records. and is the weight adjustment factor.

[0176] The present invention realizes efficient management and maintenance of rural social service resources through cross-modal data fusion, knowledge graph construction and dynamic update mechanism. First, a unified feature vector is generated through multimodal data preprocessing and feature extraction; then, the multimodal Transformer model is used to fuse the features of different modalities, and the model is optimized through comparative learning; then, a knowledge graph containing information such as varieties, regions, and services is constructed, and the graph representation capability is improved using an embedding algorithm; finally, a dynamic update mechanism is designed to ensure that the graph can reflect the latest service information in a timely manner.

[0177] The method proposed by the present invention has the following characteristics: Innovative data preprocessing and feature extraction methods. The present invention extracts features from text and image data through the BERT model and the ResNet model respectively; uses the embedding layer technology to map structured data into vector representation, and converts text, image and service feature vectors into a unified 1024-dimensional representation through the embedding layer. This method ensures that data of different modalities can be fused and operated in the same dimensional space.

[0178] Multimodal Transformer model fusion. The present invention utilizes a multimodal Transformer model to deeply fuse the features of different modalities through self-attention mechanism and multi-head attention mechanism, and introduces a contrast loss function to learn the correlation between modalities, thereby enhancing the generalization ability and accuracy of the model when processing multimodal data.

[0179] The comprehensive knowledge graph construction method invented defines entities such as varieties, regions, service entities and production entities and their relationships, generates a comprehensive feature vector through a multimodal fusion model, combines the Bi-LSTM-CRF model for entity and relationship recognition, creates a knowledge graph in the graph database, and embeds and fuses it with the TransE algorithm, which significantly improves the representation ability and retrieval efficiency of the knowledge graph.

[0180] The dynamically updated graph management mechanism innovatively introduces probability and threshold-based methods to identify new service organizations and service records, uses Bayesian formulas and graph matching algorithms to verify the validity of new data, and dynamically adjusts the edge weights in the graph to reflect the latest information, ensuring that the graph can be updated in a timely manner and accurately reflect the current service situation.

[0181] In summary, the present invention significantly improves the efficiency and accuracy of rural social service resource management through innovative data preprocessing and feature extraction methods, unified feature embedding technology, multimodal Transformer model fusion, comprehensive knowledge graph construction method, and dynamically updated graph management mechanism.

[0182] First, through the multimodal fusion of text, image and structured data, the effective integration and unified processing of different data sources are achieved, ensuring the standardization and consistency of data. Secondly, the multimodal Transformer model is used to deeply fuse the features of different modalities, and the self-attention mechanism and multi-head attention mechanism are used to enhance the model's learning ability of the correlation between data, thereby improving the generalization ability and accuracy of the model. In terms of knowledge graph construction, the entity and relationship recognition is carried out through the comprehensive feature vector and Bi-LSTM-CRF model, and the embedding fusion is combined with the TransE algorithm, which greatly improves the representation ability and retrieval efficiency of the knowledge graph, and can better display and manage the complex relationship of rural social service resources. The dynamic update mechanism ensures that the knowledge graph can reflect the latest service information in a timely manner, identifies new data through probability and threshold-based methods, and dynamically adjusts the edge weights in the graph, ensuring the real-time and accuracy of the graph.

[0183] In summary, the innovation and application of the present invention in data fusion, knowledge graph construction and dynamic updating not only improve the intelligent level of rural social service resource management, but also provide more accurate and comprehensive data support for relevant decision-making, thereby promoting the development of rural economy and informatization construction.

[0184] The cross-modal knowledge graph construction device provided by the present invention is described below. The cross-modal knowledge graph construction device described below and the cross-modal knowledge graph construction method described above can be referenced to each other.

[0185] refer to Figure 4 , Figure 4 It is a structural schematic diagram of the cross-modal knowledge graph construction device provided by the present invention, which includes: an acquisition module 401, a preprocessing module 402, a splicing module 403, a fusion module 404, a mapping module 405 and a construction module 406.

[0186] An acquisition module 401 is used to acquire text data, image data and structured data of the agricultural industry; A preprocessing module 402 is used to preprocess and extract features from the text data, the image data and the structured data to obtain a text feature vector, an image feature vector and a structured feature vector; A splicing module 403 is used to embed and splice the text feature vector, the image feature vector and the structured feature vector to obtain an overall feature sequence; A fusion module 404 is used to input the overall feature sequence into a multimodal Transformer model to obtain a comprehensive feature vector output by the multimodal Transformer model; A mapping module 405 is used to input the comprehensive feature vector into the Bi-LSTM-CRF model to obtain entity and relationship labels output by the Bi-LSTM-CRF model, wherein there is a mapping relationship between the entity and the relationship label; The construction module 406 is used to construct a knowledge graph based on the entity and the relationship label, wherein the nodes of the knowledge graph are used to represent the entity, and the edges of the knowledge graph are used to represent the relationship label.

[0187] Specifically, the above-mentioned cross-modal knowledge graph construction device provided by the present invention can implement all the method steps implemented by the above-mentioned cross-modal knowledge graph construction method embodiment, and can achieve the same technical effect. The parts and beneficial effects that are the same as the method embodiment in this embodiment will not be described in detail here.

[0188] refer to Figure 5 , Figure 5 is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute a cross-modal knowledge graph construction method, which includes: obtaining text data, image data and structured data of the agricultural industry; preprocessing and feature extraction of the text data, image data and structured data respectively to obtain text feature vectors, image feature vectors and structured feature vectors; embedding and splicing the text feature vectors, image feature vectors and structured feature vectors to obtain an overall feature sequence; inputting the overall feature sequence into a multimodal Transformer model to obtain a comprehensive feature vector output by the multimodal Transformer model; inputting the comprehensive feature vector into a Bi-LSTM-CRF model to obtain entity and relationship labels output by the Bi-LSTM-CRF model, wherein there is a mapping relationship between entity and relationship labels; constructing a knowledge graph based on entity and relationship labels, wherein the nodes of the knowledge graph are used to represent entities, and the edges of the knowledge graph are used to represent relationship labels.

[0189] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0190] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the cross-modal knowledge graph construction method provided by the above methods, which method includes: obtaining text data, image data and structured data of the agricultural industry; preprocessing and feature extraction of the text data, image data and structured data respectively to obtain text feature vectors, image feature vectors and structured feature vectors; embedding and splicing the text feature vectors, image feature vectors and structured feature vectors to obtain an overall feature sequence; inputting the overall feature sequence into a multimodal Transformer model to obtain a comprehensive feature vector output by the multimodal Transformer model; inputting the comprehensive feature vector into a Bi-LSTM-CRF model to obtain entity and relationship labels output by the Bi-LSTM-CRF model, wherein there is a mapping relationship between the entity and the relationship label; constructing a knowledge graph based on the entity and relationship labels, wherein the nodes of the knowledge graph are used to represent entities, and the edges of the knowledge graph are used to represent relationship labels.

[0191] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the cross-modal knowledge graph construction method provided by the above-mentioned methods, the method comprising: obtaining text data, image data and structured data of the agricultural industry; preprocessing and feature extraction of the text data, image data and structured data respectively to obtain text feature vectors, image feature vectors and structured feature vectors; embedding and splicing the text feature vectors, image feature vectors and structured feature vectors to obtain an overall feature sequence; inputting the overall feature sequence into a multimodal Transformer model to obtain a comprehensive feature vector output by the multimodal Transformer model; inputting the comprehensive feature vector into a Bi-LSTM-CRF model to obtain entity and relationship labels output by the Bi-LSTM-CRF model, wherein there is a mapping relationship between the entity and the relationship label; constructing a knowledge graph based on the entity and relationship labels, wherein the nodes of the knowledge graph are used to represent entities, and the edges of the knowledge graph are used to represent relationship labels.

[0192] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0193] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-modal knowledge graph construction method, characterized in that: include: Obtain text data, image data, and structured data of the agricultural industry; Preprocessing and feature extraction are performed on the text data, the image data, and the structured data to obtain a text feature vector, an image feature vector, and a structured feature vector; Embedding and concatenating the text feature vector, the image feature vector and the structured feature vector to obtain an overall feature sequence; Inputting the overall feature sequence into a multimodal Transformer model to obtain a comprehensive feature vector output by the multimodal Transformer model; Inputting the comprehensive feature vector into the Bi-LSTM-CRF model to obtain entity and relationship labels output by the Bi-LSTM-CRF model, wherein a mapping relationship exists between the entity and the relationship label; A knowledge graph is constructed based on the entities and the relationship labels, wherein the nodes of the knowledge graph are used to represent the entities, and the edges of the knowledge graph are used to represent the relationship labels.

2. The cross-modal knowledge graph construction method according to claim 1, characterized in that: The preprocessing and feature extraction of the text data, the image data and the structured data are performed respectively to obtain a text feature vector, an image feature vector and a structured feature vector, including: Segmenting and annotating the text data to obtain processed text data, wherein the processed text data includes variety name, text description, time, and location; Inputting the processed text data into a BERT model to obtain a text feature vector output by the BERT model; Preprocessing the image data to obtain processed image data, wherein the processed image data includes the variety name, image file, time, and location; Inputting the processed image data into a residual network to obtain an image feature vector output by the residual network; Classifying and standardizing the structured data to obtain processed structured data, wherein the processed structured data includes service type, service organization, service object, time, and location; The processed structured data is word-embedded to obtain a structured feature vector.

3. The cross-modal knowledge graph construction method according to claim 1, characterized in that: The embedding and splicing of the text feature vector, the image feature vector and the structured feature vector to obtain an overall feature sequence includes: The product of the text feature vector and the embedding layer text weight matrix is ​​used as the embedded text feature vector; The product of the image feature vector and the embedding layer image weight matrix is ​​used as the embedded image feature vector; The product of the structured feature vector and the structured weight matrix of the embedding layer is used as the embedded structured feature vector; The embedded text feature vector, the embedded image feature vector and the embedded structured feature vector are concatenated to obtain an overall feature sequence.

4. The cross-modal knowledge graph construction method according to claim 1, characterized in that: The step of inputting the overall feature sequence into a multimodal Transformer model to obtain a comprehensive feature vector output by the multimodal Transformer model comprises: By using a multimodal Transformer model, multimodal feature fusion is performed based on the overall feature sequence to obtain a multimodal query vector, a multimodal key vector, and a multimodal value vector; The multimodal query vector, the multimodal key vector and the multimodal value vector are input into a multi-head self-attention mechanism through a multimodal Transformer model to obtain a multimodal feature fusion attention output by the multi-head self-attention mechanism; Through the multimodal Transformer model, the multimodal feature fusion attention is linearly transformed and layer-normalized to obtain a comprehensive feature vector.

5. The cross-modal knowledge graph construction method according to claim 1, characterized in that: After constructing the knowledge graph based on the entity and the relationship label, the method further includes: In response to identifying a new service organization, determining the confidence of the new service organization by using a Bayesian formula; When the confidence exceeds a preset confidence threshold, taking the new service organization as a valid service organization; In response to identifying a new service record, determining a maximum similarity between the new service record and a node of the knowledge graph; When the maximum similarity exceeds a preset similarity threshold, the new service record is taken as a valid service record.

6. The cross-modal knowledge graph construction method according to claim 5, characterized in that: The method further comprises: Constructing a new node in the knowledge graph based on the valid service organization, wherein the new node is used to represent the valid service organization; A new edge is constructed in the knowledge graph based on the valid service record, wherein the new edge is used to connect the service organization node, the variety node and the region node corresponding to the valid service record.

7. A cross-modal knowledge graph construction device, characterized in that: include: An acquisition module, used to acquire text data, image data and structured data of the agricultural industry; A preprocessing module, used for preprocessing and feature extraction of the text data, the image data and the structured data, respectively, to obtain a text feature vector, an image feature vector and a structured feature vector; A splicing module, used for embedding and splicing the text feature vector, the image feature vector and the structured feature vector to obtain an overall feature sequence; A fusion module, used for inputting the overall feature sequence into a multimodal Transformer model to obtain a comprehensive feature vector output by the multimodal Transformer model; A mapping module, used for inputting the comprehensive feature vector into a Bi-LSTM-CRF model to obtain entity and relationship labels output by the Bi-LSTM-CRF model, wherein a mapping relationship exists between the entity and the relationship label; A construction module is used to construct a knowledge graph based on the entities and the relationship labels, wherein the nodes of the knowledge graph are used to represent the entities, and the edges of the knowledge graph are used to represent the relationship labels.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the cross-modal knowledge graph construction method as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the cross-modal knowledge graph construction method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the cross-modal knowledge graph construction method as described in any one of claims 1 to 6.

Images