Knowledge extraction and mining method for smart home multi-modal dialogue system

Abstract

The present application relates to the technical field of multi-modal dialogue, and especially relates to a knowledge extraction and mining method for a smart home multi-modal dialogue system, which comprises the following steps: S1. smart home multi-modal dialogue system input data acquisition; S2. multi-modal explicit knowledge graph construction; S3. multi-modal explicit knowledge representation learning; S4. multi-modal explicit knowledge reasoning and relationship completion; and S5. multi-modal implicit knowledge mining and representation learning. The present application extracts and aligns knowledge from massive unstructured texts, pictures and videos, constructs a multi-modal knowledge graph, performs representation learning on the graph based on semantic meta paths, extracts rich subgraph information through graph convolution operation to complete knowledge reasoning, and further completes the missing relationships. In addition, based on a large-scale pre-training model, a pre-training task for mining implicit knowledge is designed to guide the model to learn implicit knowledge such as emotion, which can improve the logical reasoning ability of the dialogue system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal dialogue, in particular to a knowledge extraction and mining method for a smart home multi-modal dialogue system. BACKGROUND

[0002] Although the dialogue system of single modal information has achieved certain success, with people's higher level pursuit of a better and happier life, single modal is difficult to meet people's lively expression of complex intent and intuitive presentation of required information. Realizing the multi-modal dialogue system with illustrations is a new development trend. Applying the multi-modal dialogue system to smart home can help people balance diet and improve human-computer interaction experience.

[0003] According to whether knowledge can be clearly expressed in a certain code system, it can be divided into two dimensions of explicit and implicit. For explicit knowledge, although it can be constructed and represented by structured method, there are still problems such as imperfect knowledge construction and inaccurate knowledge representation. For implicit knowledge, it is difficult to mine and represent by structured method. The defects of the expression of the two kinds of knowledge limit the extraction and mining of multi-modal heterogeneous knowledge by the dialogue system and the dialogue logic reasoning ability.

[0004] To solve the above problems, the present application provides a knowledge extraction and mining method for a smart home multi-modal dialogue system. The present application aims to extract and align massive unstructured text, pictures and videos, complete the construction of multi-modal explicit knowledge graph, and perform graph representation learning based on semantic meta path. Rich subgraph information is extracted by graph convolution operation to complete machine reasoning and further complete the missing relationship. In addition, based on large-scale pre-training model, a pre-training task for mining implicit knowledge is designed to guide the model to learn implicit knowledge such as emotion. SUMMARY

[0005] The present application is developed to solve the problems of the prior art.

[0006] The technical scheme for solving the technical problem of the present application is as follows: a knowledge extraction and mining method for a smart home multi-modal dialogue system, comprising the following steps:

[0007] S1. Smart home multi-modal dialogue system input data acquisition: multi-modal data of a user is captured by various sensors in the smart home, and then a multi-modal data set is obtained ;

[0008] S2. Multi-modal explicit knowledge graph construction: the multi-modal data set is obtained through a BERT-based bidirectional recurrent neural network model to obtain an entity set , a relation set and a triple set , based on the entity set , the relation set and the triple set to construct a multi-modal explicit knowledge graph ;

[0009] S3. Multi-modal explicit knowledge representation learning: the multi-modal explicit knowledge graph is processed by an explicit knowledge representation learning model to obtain feature representation vectors of entity nodes and relation representation vectors between entity nodes;

[0010] S4. Inference and relation completion of multi-modal explicit knowledge: the multi-modal explicit knowledge graph and the feature representation vectors of the entity nodes and the relation representation vectors between the entity nodes are input into a multi-modal explicit knowledge graph knowledge inference model for inference, to complete the missing triple knowledge in the knowledge graph, to obtain new graph data in the multi-modal explicit knowledge graph, and thus obtain a complete multi-modal explicit knowledge graph;

[0011] S5. Mining and representation learning of multi-modal implicit knowledge: the multi-modal data set obtains a probability distribution vector of the current user's emotional state through a pre-training sentiment classification model based on BERT, and then obtains a fine-tuned model, and inputs the complete multi-modal explicit knowledge graph into the fine-tuned model to obtain the implicit emotional information in the multi-modal data.

[0012] Specifically, the multi-modal data in step S1 includes an audio data set , a video data set , a text data set and an image data set , and the multi-modal data set .

[0013] 3. The knowledge extraction and mining method of the intelligent home-oriented multi-modal dialogue system according to claim 1, characterized in that step S2 is specifically:

[0014] S21. The audio data set is converted into text data by using an audio-to-text model and added to the text data set , the single text in the text data set is subjected to sentence feature extraction by a BERT model based on bidirectional encoding of Transformer to obtain text sequence features , the text sequence features are subjected to final text sequence features by a bidirectional LSTM model, and the formula is as follows:

[0015] ,

[0016] ，

[0017] wherein, denotes a BERT model based on a bidirectional encoding of a Transformer, denotes a bidirectional LSTM model,

[0018] denotes a feature vector at position on a text sequence feature , denotes a word corresponding memory at the previous step sequence position , denotes a feature vector at position on the final text sequence feature ; then, a linear transformation and Softmax are used to obtain the classification of each word in the final text sequence feature on relationship labels, wherein denotes relationship categories, at the same time, vector and matrix multiplication are used to obtain the relevance between entities, and finally the BERT model based on bidirectional encoding of the Transformer and the bidirectional LSTM model are trained using the label information contained in the data to obtain the triple information in the text , and then the triple information is input into the general large model LLaMA3.1 to complete the effective verification of the triplets;

[0019] S22. Detecting entity targets by using ViT and maximum pooling Maxpooling on the image dataset , processing a single picture, and finally obtaining entity targets in the picture and the relationship between the entity targets , ; the single picture processing process is as follows: a single picture is divided into image patches with position encoding, , , denotes the length and width of the picture, The size of the patch is represented, the image patch is mapped to an embedding vector combined with a linear transformation, and a position encoding embedding is added in the embedding vector to obtain an output feature in the format of a class token, and the output feature is classified by a multi-layer perception (MLP);

[0020] S23. Key frame extraction of the video dataset The video is compressed into a video key frame image data set with h pictures by key frame extraction of the video in the video dataset through a difference frame method, and then the entities and the relationships between the entities in the single picture in the key frame image data set are obtained by using the method in step S22, the image of the key frame is converted into text by using the pre-trained model CLIP, and the triple information in the text is obtained by using the method of processing the text in step S21.

[0021] S24. Based on the information of different modalities obtained in steps S21, S22, and S23, the knowledge between the modalities is aligned by calculating the semantic similarity by using a feature comparison method, and the knowledge is refined into triples to obtain a multi-modal explicit knowledge graph .

[0022] Specifically, in step S3:

[0023] The explicit knowledge representation learning model includes a subgraph structure acquisition module and a multi-layer graph convolution module.

[0024] The multi-modal explicit knowledge graph obtains a subgraph structure containing semantics through a semantic meta-path based method by using the subgraph structure acquisition module, and further aggregates the information of adjacent nodes of the subgraph structure by combining graph convolution operations The formula is as follows:

[0025] ,

[0026] wherein, and represent the feature representation vectors of adjacent entity nodes , and represent the feature representation vectors of the entity nodes at the layer and the layer of the meta-path, respectively, represents the initial feature vector randomly generated by a random function for the node , represents the neural network weight of the layer of the meta-path, represents the relationship vector between the entity nodes and , Indicates semantic meta-path, This represents the convolution operation;

[0027] The adjacent nodes of the subgraph structure The information is processed by a multi-layer graph convolution module to perform multi-layer graph convolution operations, calculating entity and relation vectors, and then constructing positive examples of triples. and negative examples and the loss function To optimize, the loss function is calculated using the following formula:

[0028] ,

[0029] in, Represents the distance factor. Indicates the number of random replacements. The entity vector of each entity node. Indicates the number of random replacements. The entity vector of each entity node. This represents a positive example of a triple. Represents the distance function. This means mapping heterogeneous multimodal entities and relationships to a unified vector space, and finally obtaining the feature representation vectors of entity nodes and the relationship representation vectors between entity nodes in the multimodal explicit knowledge graph through training.

[0030] Specifically, in step S4:

[0031] The multimodal explicit knowledge graph knowledge reasoning model includes a graph convolution module, a graph pooling module, and an attention mechanism module;

[0032] The multimodal explicit knowledge graph, along with the feature representation vectors of the entity nodes and the relationship representation vectors between the entity nodes, are input into a graph convolution module for convolution to obtain a graph of convolutional triples. Then, a graph pooling module aggregates the nodes in the graph of the convolutional triples, taking the average of the neighboring nodes as the latest node, resulting in an aggregated graph. This aggregated graph is then input into an attention mechanism module. By calculating the feature vectors of each node and its neighboring nodes, as well as the attention weights between the current node and its neighbors, the module identifies closely related neighboring nodes. The features of these neighboring nodes are then weighted and summed to obtain a new feature representation for the current node, ultimately yielding the multimodal explicit knowledge graph. The calculation formula for the new graph data is as follows:

[0033] ,

[0034] in, This represents the subgraph features obtained using a graph convolutional neural network (GCN). , Representing nodes respectively The degree, Represents a linear transformation matrix. Indicates the offset. Indicates adjacent nodes eigenvectors, Represents a node The set of adjacent nodes, Indicates the process after attention matrix Weighted subgraph features, This represents a neural network used for classification—a two-layer fully connected neural network. This represents the i-th entity node. This represents the j-th entity node. Represents the activation function. This represents the sigmoid activation function.

[0035] Specifically, in step S5:

[0036] The BERT-based pre-trained sentiment classification model uses the BERT model as its base model and is constructed by combining it with a bidirectional recurrent neural network.

[0037] The pre-training objective function of the BERT-based pre-trained sentiment classification model for:

[0038] ,

[0039] in, Represents the entirety of the sentiment lexicon. One word, Indicates the first A sentiment category tag, and This represents the weight of each task in the objective function. Indicates the number of sentiment categories. Represents a probability distribution. This represents the loss function.

[0040] Specifically, the formula for calculating the probability distribution vector is:

[0041] ,

[0042] in, This represents the probability distribution of the model output. Represents multimodal tuples Related sentiment category tags, Data representing visual modalities, Data representing text modality, express distribution, representing the occluded emotional words, representing the length of the text.

[0043] Specifically, the triple set , respectively represent the head entity, the tail entity and the relationship from the head entity to the tail entity.

[0044] Specifically, the BERT-based bidirectional recurrent neural network model comprises an audio-to-text model, a BERT model based on bidirectional encoding of Transformer, a bidirectional LSTM model, a ViT, a maximum pooling Maxpooling, a multilayer perceptron MLP and a pre-training model CLIP.

[0045] The present application has the advantages of:

[0046] The present application extracts and aligns knowledge from massive unstructured texts, pictures and videos, constructs a multi-modal knowledge graph, performs representation learning of the graph based on semantic meta paths, extracts rich subgraph information through graph convolution operation to complete machine reasoning, and further completes the missing relationships. In addition, based on a large-scale pre-training model, a pre-training task for mining implicit knowledge is designed to guide the model to learn implicit knowledge such as emotion. Through the technical scheme of the present application, the explicit knowledge and implicit knowledge are enriched, and the logical reasoning ability is given, which can improve the intelligence quotient of the dialogue system. BRIEF DESCRIPTION OF DRAWINGS

[0047] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, which together with the embodiments of the present application, serve to explain the present application, and do not constitute a limitation of the present application.

[0048] Figure 1 is a flowchart of the present application;

[0049] Figure 2 is a flowchart of explicit knowledge construction in the present application;

[0050] Figure 3 is a flowchart of explicit knowledge relationship completion in the present application;

[0051] Figure 4 is a flowchart of the pre-training emotion classification model based on BERT in the present application. DETAILED DESCRIPTION

[0052] With reference to the drawings and embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0053] Embodiment 1

[0054] In this embodiment, a knowledge extraction and mining method for a smart home multi-modal dialogue system is provided, including the following steps:

[0055] S1. Smart home multi-modal dialogue system input data acquisition: multi-modal data of a user is captured by various sensors in a smart home, and then a multi-modal data set is obtained , the multi-modal data including an audio data set , a video data set , a text data set and an image data set ;

[0056] S2. Multi-modal explicit knowledge graph construction: the multi-modal data set is subjected to a bidirectional recurrent neural network model based on BERT to obtain an entity set , a relation set and a triple set , and based on the entity set , the relation set and the triple set , a multi-modal explicit knowledge graph is constructed, the triple set , respectively representing a head entity, a tail entity and a relation from the head entity to the tail entity;

[0057] S3. Multi-modal explicit knowledge representation learning: the multi-modal explicit knowledge graph is processed by an explicit knowledge representation learning model to obtain feature representation vectors of entity nodes and relation representation vectors between entity nodes;

[0058] S4. Inference and relation completion of multi-modal explicit knowledge: the multi-modal explicit knowledge graph and the feature representation vectors of the entity nodes and the relation representation vectors between the entity nodes are input into a multi-modal explicit knowledge graph knowledge inference model for inference, to complete the missing triple knowledge in the knowledge graph, to obtain new graph data in the multi-modal explicit knowledge graph, and then to obtain a complete multi-modal explicit knowledge graph;

[0059] S5. Mining and representation learning of multi-modal implicit knowledge: the multi-modal data set The probability distribution vector of the current user's emotional state is obtained by using a BERT-based pre-trained sentiment classification model, and then a fine-tuned model is obtained. The complete multimodal explicit knowledge graph is input into the fine-tuned model to obtain the implicit sentiment information in the multimodal data.

[0060] Furthermore, the construction of the multimodal explicit knowledge graph in step S2 specifically involves:

[0061] S21. Transfer the audio dataset Use an audio-to-text model to convert the audio data into text data and add it to a text dataset. In the text dataset single text in Sentence features are extracted using a BERT model based on Transformer bidirectional encoding to obtain text sequence features. The text sequence features The final text sequence features are obtained through a bidirectional LSTM model. The formula is expressed as follows:

[0062] ,

[0063] ，

[0064] in, This represents the BERT model based on bidirectional encoding using Transformer. This represents a bidirectional LSTM model.

[0065] Indicates features in text sequence The upper position is eigenvectors, Indicates the position of the previous sequence step Memorize by word correspondence. Indicates the features in the final text sequence. The upper position is The feature vectors are obtained; then, linear transformation and Softmax are used to obtain the final text sequence features. Each word in Categories on relational labels, among which express The system identifies relationship categories and uses vector and matrix multiplication to obtain the associations between entities. Finally, it uses the label information contained in the data to train a Transformer-based bidirectional encoding BERT model and a bidirectional LSTM model to obtain triple information in the text. Then input the triple information into the general large model LLaMA3.1 to complete the effective verification of the triple;

[0066] S22. Utilize ViT and maximum pooling Maxpooling to detect the image dataset to obtain entity targets, process a single picture, and finally obtain entity targets in the picture and the relationship between the entity targets ; , ; The single picture processing process is as follows: the single picture is divided into image patches with position encoding, , , represent the length and width of the picture, represent the size of the patch, combine linear transformation to map the image patch to an embedding vector, and add position encoding embedding in the embedding vector to obtain an output feature in the class token format, and use a multilayer perception MLP to classify the output feature;

[0067] S23. Key frame extraction is performed on the video in the video dataset by a difference frame method, the video is compressed into a video key frame image data group with h pictures, then the method in step S22 is used to obtain entities and relationships between entities in a single picture in the key frame image data group, the pre-trained model CLIP is used to convert the key frame image into text, and the method of processing text in step S21 is used to obtain triple information in the text;

[0068] S24. Based on the different modal information obtained in steps S21, S22, and S23, use the feature comparison method to align the knowledge between the multi-modal by calculating the semantic similarity, and refine the knowledge into triples to obtain a multi-modal explicit knowledge graph .

[0069] As shown in Figure 2 , each sensor in the smart home captures audio data, video data, text data, and image data such as "fruit, vegetable, potato, watermelon, cabbage, spinach" in the scientific diet application scenario, obtains a multi-modal dataset about "fruit, vegetable, potato, watermelon, cabbage, spinach", and the multi-modal data obtains the relationship between each entity such as "fruit, vegetable, potato, watermelon, cabbage, spinach, apple, kiwi" through the above steps, and finally forms a multi-modal knowledge graph.

[0070] Further, the multi-modal explicit knowledge representation learning in step S3 is specifically:

[0071] The explicit knowledge representation learning model includes a subgraph structure acquisition module and a multi-layer graph convolution module;

[0072] The multimodal explicit knowledge graph is processed by the subgraph structure acquisition module to obtain semantically meaningful subgraph structures using a semantic metapath-based approach. Furthermore, the adjacent nodes of the subgraph structures are aggregated using graph convolution operations. The information is expressed by the following formula:

[0073] ,

[0074] in, and Representing adjacent entity nodes respectively The feature representation vector, and Representing the metapath, respectively Layer and First Feature representation vectors of layer entity nodes express The nodes use initial feature vectors randomly generated by a random function. Indicates the metapath number The weights of the neural network layers, Represents entity nodes and Relationship vectors Indicates semantic meta-path, This represents the convolution operation;

[0075] The adjacent nodes of the subgraph structure The information is processed by a multi-layer graph convolution module to perform multi-layer graph convolution operations, calculating entity and relation vectors, and then constructing positive examples of triples. and negative examples and the loss function The optimization is performed using the following formula:

[0076] ,

[0077] in, Represents the distance factor. Indicates the number of random replacements. The entity vector of each entity node. Indicates the number of random replacements. The entity vector of each entity node. This represents a positive example of a triple. Represents the distance function. The mapping of the heterogeneous multi-modal entities and relations into a unified vector space, and finally obtaining the feature representation vectors of the entity nodes and the relation representation vectors between the entity nodes in the knowledge graph through training.

[0078] Further, the multi-modal explicit knowledge reasoning and relation completion in step S4 are specifically:

[0079] The multi-modal explicit knowledge graph knowledge reasoning model comprises a graph convolution module, a graph pooling module and an attention mechanism module.

[0080] The multi-modal explicit knowledge graph and the feature representation vectors of the entity nodes and the relation representation vectors between the entity nodes are input into the graph convolution module for convolution operation to obtain a graph of the convolved triplets, then the graph pooling module is used to aggregate the nodes in the graph of the convolved triplets, take the average of adjacent nodes as the latest node, obtain an aggregated graph, then the aggregated graph is input into the attention mechanism module, the feature vectors of each node and its neighbor nodes are calculated, as well as the attention weight of the current node and the neighbor nodes, the neighbor nodes closely related to the current node are obtained, then the features of the obtained neighbor nodes are weighted and summed to obtain the new feature representation of the current node, and finally the multi-modal explicit knowledge graph new graph data, and the calculation formula is as follows:

[0081] ,

[0082] wherein, represents the subgraph feature obtained by using a graph convolutional neural network GCN, , respectively represent the degrees of nodes , represents a linear transformation matrix, represents an offset, represents the feature vector of an adjacent node , represents the adjacent node set of node , represents the subgraph feature weighted by the attention matrix , represents a neural network for classification, i.e., a two-layer fully connected neural network, represents the i-th entity node, represents the j-th entity node, represents an activation function, represents a sigmoid activation function.

[0083] As Figure 3As shown, the multi-modal knowledge graph of the relationship between "fruit, vegetable, potato, watermelon, cabbage, spinach, apple, kiwi" in the scientific diet application scenario is completed through graph convolution, graph pooling and attention mechanism.

[0084] Further, the step S5 is described as the mining and representation learning of multi-modal implicit knowledge, specifically:

[0085] The pre-training sentiment classification model based on BERT takes the BERT model as the basic model, and is constructed by combining a bidirectional recurrent neural network.

[0086] The pre-training target function of the pre-training sentiment classification model based on BERT is:

[0087] ,

[0088] Among them, represents all words in the sentiment vocabulary, represents the th sentiment classification label, and represents the weight of each task in the target function, represents the number of sentiment classification, represents the probability distribution.

[0089] Further, the probability distribution vector calculation formula is:

[0090] ,

[0091] Among them, represents the probability distribution output by the model, represents the sentiment classification label associated with the multi-modal tuple , represents the data of the visual mode, represents the data of the text mode, represents distribution, represents the occluded sentiment word, represents the text length.

[0092] As Figure 4 shown, the multi-modal corpus (which stores the multi-modal information data of the user ​The data in the text corpus is input into the sentiment classification model to obtain multi-modal implicit knowledge (sentiment classification), such as the like degree of watermelon (positive, neutral, and negative). The data in the pure text corpus (which stores the text information of the user) is input into the sentiment classification model to obtain multi-modal implicit knowledge (sentiment cue classification), such as the nutritional status of vegetables (nutritious).

[0093] Embodiment 2

[0094] In this embodiment, Table 1 is the experimental comparison data of the method of the present application and the prior art. The area under the ROC curve (AUC) and the average reciprocal rank (MRR) are used as evaluation indexes, and the following advanced methods at home and abroad are selected for comparison: (1) PAI-BPR is an attribute-level modeling scheme, which extracts and mines knowledge by positioning inharmonious attributes and harmonious attributes. (2) HFGN builds a hierarchical graph network to model the relationship between multiple nodes and extracts and mines knowledge. (3) GP-BPR is a comprehensive preference modeling scheme that jointly explores multi-modal data (such as pictures and text descriptions) to extract and mine knowledge. As can be seen from Table 1, the effect of the method of the present application is optimal, proving the effectiveness of the method of the present application.

[0095] Table 1 Comparison of the effect of the method of the present application and the prior art

[0096]

[0097] In this embodiment, Table 1 shows the effect comparison of the method of the present application and the advanced methods at home and abroad. As shown in Table 1, the present application is superior to other methods in multiple indicators, fully embodying its effectiveness and advancement in knowledge extraction and mining of intelligent home multi-modal dialogue systems. This proves that by carrying out multi-modal explicit knowledge graph construction and representation learning, multi-modal explicit knowledge reasoning and relationship completion, and multi-modal implicit knowledge mining and representation learning, the present application can significantly improve the knowledge extraction and mining of intelligent home multi-modal dialogue systems.

[0098] Finally, it should be noted that: the above only describes the preferred embodiments of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent replacements to some technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1.A method for knowledge extraction and mining of a smart home multi-modal dialogue system, characterized by: Comprising the following steps: S1. Input data acquisition of smart home multi-modal dialogue system: multi-modal data of users are captured by various sensors in the smart home, and then a multi-modal data set is obtained ; S2. Multi-modal explicit knowledge graph construction: multi-modal data set Through the BERT-based bidirectional recurrent neural network model, an entity set , a relation set and a triple set are obtained, and a multi-modal explicit knowledge graph is constructed based on the entity set , the relation set and the triple set ;​ S3. Multimodal explicit knowledge representation learning: the multimodal explicit knowledge graph is processed by an explicit knowledge representation learning model to obtain feature representation vectors of entity nodes and relationship representation vectors between entity nodes; The explicit knowledge representation learning model comprises a subgraph structure acquisition module and a multi-layer graph convolution module, and the specific steps comprise: The multi-modal explicit knowledge graph obtains the sub-graph structure containing semantics by the sub-graph structure acquisition module in a manner based on a semantic meta path (SemanticMetaPath), and further aggregates information of adjacent nodes of the sub-graph structure in combination with graph convolution operation is expressed by a formula as follows: ， wherein, and respectively represent the feature representation vector of the adjacent entity node , and respectively represent the feature representation vector of the entity node of the first layer and the second layer of the meta-path, represents the initial feature vector randomly generated by the node using a random function, represents the neural network weight of the first layer of the meta-path, represents the relationship vector between the entity node and , represents the semantic meta-path, represents the convolution operation; Adjacent nodes of a subgraph structure The information is subjected to multi-layer graph convolution operation by a multi-layer graph convolution module, entity and relation vectors are calculated, positive examples and negative examples of triplets are constructed , and a loss function is optimized, and the loss function calculation formula is:​ ， in, Represents the distance factor. Indicates the number of random replacements. The entity vector of each entity node. Indicates the number of random replacements. The entity vector of each entity node. This represents a positive example of a triple. Represents the distance function. This means mapping heterogeneous multimodal entities and relationships to a unified vector space, and finally obtaining the feature representation vectors of entity nodes and the relationship representation vectors between entity nodes in the multimodal explicit knowledge graph after training; S4. Inference and relationship completion of multimodal explicit knowledge: the multimodal explicit knowledge graph and the feature representation vectors of the entity nodes and the relationship representation vectors between the entity nodes are input into a multimodal explicit knowledge graph knowledge inference model for inference, the missing triple knowledge in the knowledge graph is completed, new graph data in the multimodal explicit knowledge graph is obtained, and then a complete multimodal explicit knowledge graph is obtained; The multimodal explicit knowledge graph knowledge inference model comprises a graph convolution module, a graph pooling module and an attention mechanism module, and the specific steps comprise: The multi-modal explicit knowledge graph is input into a graph convolution module for convolution operation to obtain a graph of the convolved triplets, then the graph of the convolved triplets is input into a graph pooling module, nodes in the graph of the convolved triplets are aggregated, an average value of adjacent nodes is taken as a latest node to obtain an aggregated graph, then the aggregated graph is input into an attention mechanism module, feature vectors of each node and neighbor nodes thereof are calculated, and attention weights of the current node and the neighbor nodes are calculated to obtain neighbor nodes closely related to the current node, then features of the neighbor nodes are weighted and summed to obtain new feature representation of the current node, and finally the multi-modal explicit knowledge graph is obtained The new graph data is calculated according to the following formula: ， wherein, denotes subgraph features obtained using a graph convolutional neural network GCN, , denote degrees of nodes , denotes a linear transformation matrix, denotes an offset, denotes a feature vector of an adjacent node , denotes a set of adjacent nodes of a node , denotes subgraph features weighted by an attention matrix , denotes a neural network for classification - a two-layer fully connected neural network, denotes an i-th entity node, denotes a j-th entity node, denotes an activation function, denotes a sigmoid activation function; S5. Mining and representation learning of multi-modal implicit knowledge: the multi-modal dataset The probability distribution vector of the current user's emotional state is obtained through the pre-training emotion classification model based on BERT, and then the fine-tuned model is obtained. The complete multi-modal explicit knowledge graph is input into the fine-tuned model to obtain the implicit emotional information in the multi-modal data. 2.The knowledge extraction and mining method for the smart home multi-modal dialogue system according to claim 1, characterized by the step of The multi-modal data described in S1 includes an audio data set , a video data set , a text data set , and an image data set The multi-modal data set . 3.The knowledge extraction and mining method for the smart home multi-modal dialogue system according to claim 1, characterized in that, Step S2 is specifically: S21. The audio dataset is converted into text data using an audio-to-text model and added to a text dataset S22. The text data in the text dataset is converted into text sequence features using a BERT model S23. The text sequence features are converted into final text sequence features using a bidirectional LSTM model S24. The final text sequence features are input into a text classification model to obtain a text classification result S25. The text classification result is output S26. The text classification result is output S27. The text classification result is output ​ ， ， wherein, denotes a BERT model based on a bidirectional encoding of a Transformer, denotes a bidirectional LSTM model, Indicates features in text sequence The upper position is eigenvectors, Indicates the position of the previous sequence step Memorize by word correspondence. Indicates the features in the final text sequence. The upper position is The feature vectors are obtained; then, linear transformation and Softmax are used to obtain the final text sequence features. Each word in Categories on relational labels, among which express The system identifies relationship categories and uses vector and matrix multiplication to obtain the associations between entities. Finally, it uses the label information contained in the data to train a Transformer-based bidirectional encoding BERT model and a bidirectional LSTM model to obtain triple information in the text. Then, the triplet information is input into the general large model LLaMA3.1 to complete the valid verification of the triplet; S22. Using ViT and Maxpooling on image datasets Entity targets are detected, and individual images are processed to obtain the final result. Individual targets and entity targets Relationship between , The single image processing procedure is as follows: [Process the single image...] Segmented into positions with encoding One image patch, , , Indicates the length and width of the image. The size of the patch is represented by a linear transformation, which maps the image patch to an embedding vector. A positional encoding embedding is added to the embedding vector to obtain output features in class token format. The output features are then classified using a multilayer perceptron (MLP). S23. Key frame extraction is performed on the video data set by a difference frame method The video is compressed into a video key frame image data set with h pictures by key frame extraction on the video in the difference frame method. Then, the method in step S22 is used to obtain entities and relationships between entities in a single picture in the key frame image data set. The pre-trained model CLIP is used to convert the key frame image into text. The way of processing text in step S21 is adopted to obtain triple information in the text. S24. Based on the information of different modalities obtained in steps S21, S22 and S23, the knowledge among the multi-modalities is aligned by calculating semantic similarity using a feature comparison method, and the knowledge is refined into triples to obtain a multi-modal explicit knowledge graph . 4.The knowledge extraction and mining method for the smart home multi-modal dialogue system according to claim 1, characterized in that, Step S5 is specifically: The pre-training sentiment classification model based on BERT takes a BERT model as a basic model and is constructed in combination with a bidirectional recurrent neural network; The pre-training objective function of the BERT-based pre-training sentiment classification model is: ， wherein, denotes all words in the sentiment lexicon, denotes the i-th sentiment classification label, and denotes the weight of each task in the objective function, denotes the number of sentiment classifications, denotes the probability distribution.​​ 5.The knowledge extraction and mining method for the smart home multi-modal dialogue system of claim 1, characterized in that, The probability distribution vector calculation formula is: ， wherein, represents a probability distribution of the model output, represents a multi-modal tuple a related sentiment classification label, represents data of a visual modality, represents data of a textual modality, represents a distribution, represents an occluded sentiment word, represents a text length. 6.The knowledge extraction and mining method for the smart home multi-modal dialogue system of claim 1, characterized in that, the set of triples , respectively represent the head entity, the tail entity and the relation from the head entity to the tail entity. 7.The knowledge extraction and mining method for the smart home multi-modal dialogue system according to claim 3, characterized in that, The bidirectional recurrent neural network model based on BERT comprises an audio-to-text model, a BERT model based on bidirectional encoding of a Transformer, a bidirectional LSTM model, a ViT, a maximum pooling Maxpooling, a multi-layer perception MLP and a pre-training model CLIP.

Images