A multimodal-based smart building knowledge extraction model and method

Abstract

The present invention provides a multimodal-based smart building knowledge extraction model and method, belonging to the field of natural language processing technology, to solve the problems faced by existing information extraction technologies in terms of data quality, scale, and task complexity. The model and method include: a gating module, a data generation module, a data diversion module, and a modal information extraction module. The gating module is mainly responsible for encoding the original input data and determining the subsequent data processing module; the data generation module is mainly responsible for generating corresponding text descriptions for single visual modal data; the data diversion module is mainly responsible for selecting different information extraction models for multimodal data based on the degree of matching between visual-textual multimodal data; and the modal information extraction module is mainly responsible for extracting knowledge from input information. The present invention implements a flexible, accurate, and comprehensive smart building information extraction model. Each functional module is independent of each other, capable of achieving high performance and enabling information extraction from different types of input.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a multimodal-based smart building knowledge extraction model and method. Background Art

[0002] Information extraction (IE) is a technique for extracting valuable information from unstructured data and integrating it into structured knowledge. It automatically mines entities, relationships, and events within text, making it crucial for processing massive amounts of unstructured text data. It is widely used in fields such as knowledge graph construction, intelligent question-answering systems, and public opinion analysis. IE integrates and innovates multiple technologies, including natural language processing, machine learning, knowledge representation, and reasoning, to better meet the accuracy and scalability requirements of knowledge extraction tasks. Key implementation approaches include rule-based, statistical, and deep learning-based methods. Current research in IE technology focuses on deep learning methods such as recurrent neural networks, long short-term memory networks, convolutional neural networks, and the Transformer. Furthermore, due to the limited availability of text data, multimodal information extraction techniques, using images as auxiliary cues for information extraction, have also significantly improved IE performance.

[0003] These methods each have their own advantages, but they also face limitations in data quality, scale, and task complexity. For example, the quality of joint multimodal information extraction is closely related to the quality of the training data. When visual and textual modal information are highly aligned, performance is significantly better than unimodal information extraction. However, performance degrades when cross-modal information is not highly aligned. Summary of the Invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a multimodal-based smart building knowledge extraction model and method.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0006] A multimodal smart building knowledge extraction model includes a gating module, a data generation module, a data diversion module and a modal information extraction module; the gating module is used to process the original input data, obtain the data embedding vector representation and encode it, and select the data to be input into the data generation module, the data diversion module or the modal information extraction module according to the different types of the original input data; the data generation module is used to perform data generation, perform text generation tasks on the single visual modal data encoded by the gating module, obtain image-text data and input it into the modal information extraction module. Extraction module; the data diversion module is used for data quality screening, distinguishing the data quality of the visual-text multimodal data encoded by the gating module, and selecting the single-modal information extraction module or the multimodal information extraction module to input it into the modal information extraction module; the modal information extraction module is used to perform information extraction, extracting information from the single-text modal data encoded by the gating module, extracting information from the single-visual modal data passing through the gating module and the data generation module, and extracting information from the visual-text multimodal data passing through the gating module and the data diversion module.

[0007] In this technical solution, the system designed by the present invention has a clear hierarchical structure, and each functional module is independent of each other, which can achieve higher performance and realize information extraction of different types of inputs. It effectively solves the problem that traditional multimodal information extraction methods are greatly affected by noise data, and is conducive to completing information extraction tasks more efficiently, accurately and flexibly, providing users with a better user experience.

[0008] Preferably, the data diversion module and the data generation module are also used for knowledge extraction model training;

[0009] The data generation module is used to receive visual modality data eliminated after the data diversion module in the training phase, obtain image-text pair data, and select data with better evaluation results to form a new training set for training the knowledge extraction model.

[0010] A multimodal-based smart building knowledge extraction method includes:

[0011] The original input data is processed by the gating module to obtain the data embedding vector representation and encode it. According to the different types of original input data, the data is input into the data generation module, data diversion module or modal information extraction module;

[0012] The data generation module performs data generation and performs text generation on the single visual modality data encoded by the gating module to obtain image-text pair data and input it into the modality information extraction module.

[0013] The data quality is screened by the data diversion module, and the visual-text multimodal data encoded by the gating module is differentiated in terms of data quality, and the data is selected to be input into the single-modal information extraction module or the multimodal information extraction module of the modal information extraction module;

[0014] Information extraction is performed through the modal information extraction module. Information extraction is performed on the single text modal data encoded by the gating module, on the single visual modal data that passes through the gating module and the data generation module, and on the visual-text multimodal data that passes through the gating module and the data diversion module.

[0015] Furthermore, the gate control module processes the original input data in the following manner:

[0016] A1. Obtain embedded vector representation for the original input data and encode it;

[0017] A2. Based on the input data characteristics, the gating network is used to send the data to the data generation module, data diversion module or modal information extraction module.

[0018] The data generation module specifically performs data generation in the following manner:

[0019] B1. During the training phase of the model, an image-based text transformer decoder is used as the data generator and the decoder parameters are initialized.

[0020] B2. During the model training phase, perform text generation tasks to generate image-text pair data;

[0021] B3. During the model usage phase, the text generation task is performed on the input single visual modality data to generate image-text pair data.

[0022] The data distribution module performs data classification in the following manner:

[0023] C1. Input the visual-textual multimodal data encoding information output by the gating module into the classifier layer, analyze the similarity characteristics between the two vectors, and output the probability that the visual-textual multimodal data is suitable for multimodal information joint extraction;

[0024] C2. According to the output probability value, the visual-text multimodality is sent to the corresponding information extraction module.

[0025] The modal information extraction module performs information extraction in the following manner:

[0026] D1. Use a single modal model for information extraction;

[0027] D2. Use multimodal models for information extraction.

[0028] This technical solution implements integrated processing of unimodal and multimodal raw data by setting up a gating module. The model learns the mapping relationship between input data and processing modules through a gating network. When the input raw data is single-visual modality data, the data is encoded and sent to a trained data generation module to obtain text information reflecting the image content, forming cross-modal information for multimodal information joint extraction. When the input raw data is single-text modality data, the data is encoded and sent to a single-modal information extraction module for information extraction. When the input data is multimodal visual-textual data, the data quality is distinguished by a data diversion module, and information extraction is performed using either a unimodal or multimodal extraction model. To better train the knowledge extraction model and minimize labor costs, two types of smart building datasets were constructed: the first type of dataset is composed of a small amount of manually annotated data for pre-training; the second type of dataset includes a large amount of building-related data obtained through crawling. In the first stage of training, the entire model is trained using the manually constructed dataset. In the second phase of training, the data diversion module filters the second dataset, selecting high-quality data to be added to the training set of the multimodal knowledge extraction module. Simultaneously, the data generation module receives the visual modality information eliminated by the data diversion module and generates new multimodal data. Selected high-quality data is also added to the training set, and the multimodal knowledge extraction model is trained using this final training set. The data diversion module also processes the noisy data in the multimodal dataset. Based on the correlation between the visual and textual modal data, data suitable for information extraction using the multimodal model is selected. Because multimodal information extraction can compromise model performance when data mismatches occur, the data diversion module distinguishes data quality and effectively avoids interference from noisy data. After data screening, the data diversion module does not discard all data unsuitable for multimodal information extraction. Instead, its textual information is retained and used for unimodal information extraction. Since unimodal information extraction tasks are relatively mature, multimodal noisy datasets can also achieve good results in unimodal models without interference from visual modality information, resulting in better performance. The present invention provides a model paradigm for multimodal information extraction in the field of smart buildings. Different information extraction models can be replaced according to different information extraction subtasks, and can be better migrated to different information extraction tasks.

[0029] Furthermore, the method for obtaining the data embedding vector representation in A1 is:

[0030] A101. For visual modality information, the visual modality data is divided into multiple patches. Assuming that the original input image size is H*W*C and the size of each patch is P*P, the number of patches N is calculated as follows:

[0031]

[0032] The one-dimensional vector size obtained from the image patch sequence is P*P*C, and the overall input change is N×(P 2 *c) Perform a linear transformation on each vector to map it into a high-dimensional space and obtain the patch embedding through the transformer encoder;

[0033] A102: For text modal information, input the text into the embedding layer and use the word2vec method and position embedding to obtain a word embedding vector representation containing position information.

[0034] A103, input the input vector to the transformer encoder for encoding;

[0035] A104. Input the encoded modal information into the gating network.

[0036] In this technical solution, it should be noted that computer programs are not good at processing raw image data and text data, and the raw data needs to be converted into a series of codes. For visual modal information, the patch embedding method is used to block the raw data, that is, flatten the image into a sequence of multiple patches, and input it into the encoder of the transformer model. The use of the patch embedding method helps to reduce computational complexity and increase the training speed of the model. At the same time, the vectors mapped to high-dimensional space are conducive to enhancing the model's expressive ability, capturing more complex visual modal features, and can be well integrated with the transformer structure. For text modal information, the classic word2vec method is used to predict words by given context, which is computationally efficient.

[0037] Furthermore, the method for the gate control module to execute the selection of the data subsequent processing module is:

[0038] A201. Decompose the information extraction task into data generation, data screening, and modal information extraction subtasks. Train an expert model on each subtask and learn which expert model to use based on the input to be predicted.

[0039] A202: When the input information is single visual modality data, the data is sent to the data generation module;

[0040] A203. When the input information is single text modal data, the data is sent to the modal information extraction module;

[0041] A204. When the input information is visual-text modality data, the data is sent to the data diversion module.

[0042] In this technical solution, it is important to note that in actual applications, question-answering systems may receive information in different modalities, such as single visual modality information, single textual modality information, or visual-textual modality information. The system needs to select the appropriate information extraction model based on the different inputs to complete the task. This model uses the concept of a gated network to derive weights for selecting different models based on input features. The model with the highest weight is selected to process the data, making the model more flexible and able to better meet application needs.

[0043] Furthermore, the method for the data generation module to perform model parameter initialization is:

[0044] B101. Use the manually annotated COCO dataset to train the generation module;

[0045] B102. Use an image-based text transformer decoder as a data generator;

[0046] B103, use the special mark [MASK] to token n Subsequent information is concealed;

[0047] B104, feed the encoded data into the cross-attention layer so that the model can learn the cross-modal fusion information in the training data;

[0048] B105. Use language modeling loss (LM) as the training objective to maximize the likelihood of text generation in an autoregressive manner. The objective function is as follows:

[0049]

[0050] Where k is the window size, and BLEU is used as the evaluation metric for training. For each n-gram, the ratio of the number of n-grams in the generated text that matches the reference text to the total number of n-grams in the generated text is calculated;

[0051] B106. The decoder model used in the data generation module and the text encoding model of the modal information extraction module share parameters in the cross-attention mechanism layer.

[0052] In this technical solution, it should be noted that the present invention provides a method for multimodal text generation, generating corresponding text based on visual modal information. This method adopts an autoregressive strategy, masking the subsequent text and predicting the current output using the dependency relationship of the output before the current time step (i.e., the previous text), which is more direct and effective in generative tasks. Through the text generation task, the visual modal information is converted into text, thereby obtaining the information provided by the single visual modal input.

[0053] Furthermore, the data generation module in the data generation module performs a method for generating graphic and text modal data in the training phase as follows:

[0054] B201. During the training phase, the data generation module receives the visual modality information output from the data diversion module and the visual modality data from the gating module;

[0055] B202. Performing a text generation task on the separately extracted visual modal information to generate new cross-modal information data;

[0056] B203, send the generated data to the data diversion module.

[0057] In this technical solution, it should be noted that the present invention provides a training method for effectively utilizing data, which can effectively utilize the available information in noisy data. There are relatively few high-quality datasets for multimodal information extraction, while datasets constructed by automatically crawling websites are large in size but of poor quality, posing a challenge to the multimodal information extraction task. By generating and filtering the training dataset constructed by automatically crawling on top of a model based on initial training (training with a clean dataset), a new multimodal training dataset is formed, providing the model with more high-quality training data, which is conducive to improving model quality.

[0058] Furthermore, the method for the data generation module to generate the usage phase data is as follows:

[0059] B301, the data generation module receives the visual modality encoding information from the gating module;

[0060] B302, inputting the visual coding information into the decoder;

[0061] B303. The decoder outputs the generated text modal information and the visual modal information and sends them to the modal interaction module.

[0062] The data diversion module inputs the vector output by the baseline model into the classifier layer, analyzes the similarity characteristics between the two vectors, and outputs the probability that the image and text modality data is suitable for multimodal information joint extraction. The process is as follows:

[0063] C101. To make classification task training more efficient, a hard negative sampling strategy is adopted to sample negative samples closest to positive samples and maintain a negative sample queue of size M.

[0064] C102. Compare and study the information encoded in the visual modality and the textual modality, and learn the following similarity function:

[0065]

[0066] Among them, g v and g t is a linear transformation that maps the joint representation of modal information into a normalized low-dimensional representation, g′ v (v′ cls ) and g′ t (t′ cls ) is the feature representation from the motion encoder, and the image-text / text-image correlation is further defined based on the similarity. The calculation formula is as follows:

[0067]

[0068] Among them, τ is the temperature coefficient which is a trainable hyperparameter, and N is the batch size of training samples;

[0069] C103: The visual modality data and text modality encoded data obtained through contrastive learning are fed into a fully connected layer with softmax for classification to determine whether the image-text pair matches.

[0070] C104, the training phase is based on the ITM task. The ITM loss function is as follows:

[0071]

[0072] Among them, y itm It is a two-dimensional one-hot vector representation, indicating the true similarity, with a positive sample probability of 1 and a negative sample probability of 0. H(·) represents the cross entropy function.

[0073] C105. Output the probability p that the visual modality data matches the text modality data.

[0074] In this technical solution, it should be noted that the present invention provides a method for aligning and fusing multimodal information, which aligns information of different modalities through a comparative learning strategy before inputting the multimodal information into a cross-modal encoder for cross-modal interaction, so that the similarity of the positive samples of the image-text pair is as high as possible. The data diversion module is trained using a hard negative sampling strategy, and according to the scoring of the data pairs by comparative learning, visual modal data that has a high score for a certain text modal data but is a negative sample and text modal data that has a high score for Mog visual modal data but is a negative sample are selected, thereby expecting to improve the model's ability to distinguish difficult negative samples. This method makes it easier for the multimodal encoder to perform cross-modal learning on the one hand, and on the other hand, through the sampling strategy, it can better mine data information.

[0075] Furthermore, the process of the data diversion module sending the vector data to the information extraction module is as follows:

[0076] C201. Using the probability of visual-textual modal information matching as the criterion for performing multimodal information joint extraction, it is believed that when the matching probability is greater than the mismatch probability, the data performs better under the multimodal information joint extraction model, and vice versa, it performs better under the unimodal information extraction model;

[0077] C202. During the training phase, each piece of data is selected with probability p and 1-p to perform joint extraction of multimodal information and single modal information extraction;

[0078] C203. During the training phase, the image-text data pairs selected for single-modal information extraction are split. The image modal data is fed into the data generation module to generate new image-text modal data pairs, and the text modal data is directly fed into the modal information extraction module to perform information extraction.

[0079] C204. During the usage phase, a sampling strategy is adopted based on the probability output. Specifically, the data classification label with the output probability p>0.5 is marked as "1", and the rest of the data labels are marked as "0". The data labeled "1" is selected to perform multimodal information joint extraction, and the data labeled "0" is selected to perform single-modal information extraction;

[0080] C205. In the use phase, the image-text data pair for performing single-modal information extraction is selected to discard the image modal data and only the text modal information is sent to the modal information extraction module to perform single-modal information extraction;

[0081] C206, selecting the image and text data for performing multimodal information joint extraction, and feeding it to the modal information extraction module to first perform image and text information interaction;

[0082] In this technical solution, it is important to note that the present invention provides a data classification training strategy. The data splitting module essentially performs a binary classification task. Compared to outputting hard labels of "1" and "0", the smoother softmax output during the training phase contains more information, which helps enhance the model's generalization ability.

[0083] Furthermore, the method for extracting single-modal information by the modal information extraction module is as follows (taking the named entity recognition task as an example):

[0084] D101, feed the text encoding information into the bidirectional LSTM layer, obtain context information, and feed the information into the hidden layer;

[0085] D102, input the hidden layer output information into the CRF layer and output the text annotation sequence;

[0086] In this technical solution, it should be noted that the present invention provides a text named entity recognition model, in which the text information that has been encoded in the gating module learns contextual knowledge through LSTM+CRF, further enhances the extraction of text semantic information, and obtains a text annotation sequence.

[0087] Furthermore, the method for performing multimodal information extraction by the modal information extraction module is as follows (taking the named entity recognition task as an example):

[0088] D201, feed the text modality encoding information into the bidirectional LSTM layer and hidden layer;

[0089] D202, the visual modality information and the text encoding information are sent to the cross attention layer together with the output of the hidden layer;

[0090] D203, input the cross-modal fusion information into the CRF layer, output sequence annotation, and use softmax combined with cross entropy loss as the training target for training. The cross entropy loss formula is as follows:

[0091]

[0092] Among them, M is the number of entity labels, y ij is the label, if the true label of sample i is consistent with j, it is 1, otherwise it is 0, q ij is the probability that observation sample i belongs to category j. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 This is the multimodal smart building knowledge extraction model of the present invention;

[0094] Figure 2 This is a data processing flow chart of the gate control module of the present invention;

[0095] Figure 3 This is the overall operation flow chart of the present invention;

[0096] Figure 4 This is the model training flow chart in this embodiment;

[0097] Figure 5 This is a flow chart of the model processing when the user inputs multimodal data in this embodiment;

[0098] Figure 6 This is a flowchart of the model processing when the user inputs single visual modality data in this embodiment. DETAILED DESCRIPTION

[0099] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0100] Example 1

[0101] like Figure 1 As shown, the present invention provides a multimodal-based smart building knowledge extraction model. The model includes: a gating module, a data generation module, a data diversion module and a modal information extraction module. The gating module is used to process the original input data, obtain the embedded representation of the modal data and encode it, and select the subsequent input module according to the different types of data input from the outside; the data generation module is used to generate corresponding text for a single image input and generate more training data for model training; the data diversion module is used to screen the data generated by the generation module and the data obtained by the crawler to determine whether the input data is more suitable for extraction using a single modal or multimodal model; the modal information extraction module is used to perform information extraction, extract information from the single text modal data encoded by the gating module, extract information from the single visual modal data that has passed through the gating module and the data generation module, and extract information from the visual-text multimodal data that has passed through the gating module and the data diversion module.

[0102] Example 2

[0103] like Figure 2 As shown, the present invention provides a multimodal-based smart building knowledge extraction method, comprising: a gate control module, a data generation module, a data diversion module and a modal information extraction module;

[0104] The gating stage includes the following steps:

[0105] A1. Process the original input data to obtain the embedded vector representation and encode it. The specific steps are as follows:

[0106] A101, the smart building knowledge extraction model receives visual modal information related to the smart building (e.g., building camera images) and text modal information related to the smart building (e.g., "unit door camera is damaged and hanging in the air");

[0107] A102. For the smart building visual modal information input to the model, the image is divided into multiple patches. Assuming that the original input image size is H*W*C and the size of each patch is P*P, the number of patches N is calculated as follows:

[0108]

[0109] The one-dimensional vector size obtained from the image patch sequence is P*P*C, and the overall input change is N×(P 2 *C) Perform a linear transformation on each vector to map it into a high-dimensional space, and obtain the patch embedding through the transformer encoder, denoted as V;

[0110] A103: For text modal information, input the text into the embedding layer, use the word2vec method and position embedding to obtain a word embedding vector representation containing position information, and encode the word vector representation through the transformer encoder. The output is recorded as T;

[0111] A104. Input the encoded modal information into the gating network.

[0112] A2. Based on the input data characteristics, the gating network is used to send the data to different subsequent modules. The specific steps are as follows:

[0113] A201. Decompose the smart building knowledge extraction task into three subtasks based on the input information: text data generation, modal data screening, and modal information extraction. The three subtasks correspond to the three modules of the model.

[0114] A202, the gated network uses a multi-layer perceptron network (MLP) to learn the mapping relationship from input to output. It consists of multiple fully connected layers and a softmax layer. The output is the probability p of sending the input information to the subsequent module after receiving the modal coding information. ci , ci (i = 1 ~ 3) correspond to the generation module, diversion module and single modal information extraction module respectively;

[0115] A203, compare the output probability, if max(p c1 ,p c2 ,pc3 )=p c1 , the model determines that the input information is single visual modality information, that is, picture input without text description, then the data will be sent to the data generation module, such as Figure 5 Indicates that if max(p c1 ,p c2 ,p c3 )=p c2 , the model determines that the input information is multimodal information, that is, image-text pairs, then the data will be sent to the data diversion module, such as Figure 4 Indicates that if max(p c1 ,p c2 ,p c3 )=p c3 , the model determines that the input information is single text modal information, that is, only text is input, then the data will be sent to the single modal information extraction module under the modal information extraction module, such as Figure 6 express;

[0116] The data generation module includes the following steps:

[0117] B1. During the training phase of the model, the decoder parameters are trained in the first phase, such as Figure 3 It means that the implementation method is as follows:

[0118] B101. Use a manually labeled few-shot dataset to perform preliminary training on the generation module.

[0119] B102. Use transformer decoder as data generator;

[0120] B103, use the special mark [MASK] to token n The subsequent information is masked, and the decoder generates the words in the description one by one according to the input image features, and takes the first k words of the true description as the input of the current time step;

[0121] B104. Use language modeling loss (LM) as the training objective to maximize the likelihood of text generation in an autoregressive manner. The objective function is as follows;

[0122]

[0123] Where k is the window size;

[0124] B105. Use BELU as the initial training evaluation metric. Calculate the ratio of the number of n-grams (n = 1 to 4) in the generated text that match the reference text to the total number of n-grams in the generated text to obtain the precision of each n-gram, and perform a weighted sum.

[0125] B106. The decoder model used in the data generation module and the text encoding model of the modal information extraction module share parameters in the cross-attention mechanism layer.

[0126] B2. In the second stage of model training, the downstream task of text generation is performed to generate image-text pair data, such as Figure 3 express:

[0127] B201, the data generation module after preliminary training receives the visual modality information output from the data diversion module and the visual modality information input from the gating module;

[0128] B202. Perform an image description generation task on the smart building visual modal information. The generated image description is used as text modal data and combined with the original input to form new cross-modal information data.

[0129] B203. Use the automatic evaluation indicators BELU and SPICE for evaluation, supplemented by manual evaluation, and store the generated data with better quality as a new training set for training other modules of the model.

[0130] B3. In the model usage phase, text generation is performed for the input single image modality data, such as Figure 6 It means that the implementation method is as follows:

[0131] B301, receiving smart building visual modal information output by gated network coding;

[0132] B302, inputting the visual coding information into the decoder;

[0133] B303. The decoder outputs the generated text modal information and the visual modal information and sends them to the modal interaction module.

[0134] The data diversion module includes the following steps:

[0135] C1. Input the encoded information output by the gating module into the classifier layer, analyze the similarity characteristics between the two vectors, and output the probability that the image and text modal data is suitable for multimodal information joint extraction. The implementation method is as follows:

[0136] C101. To make classification task training more efficient, a hard negative sampling strategy is adopted to sample negative samples closest to positive samples and maintain a negative sample queue of size M.

[0137] C102. Compare and study the information encoded in the visual modality and the textual modality, and learn the following similarity function:

[0138]

[0139] Among them, gv and g t is a linear transformation that maps the joint representation of modal information into a normalized low-dimensional representation, g′ v (v′ cls ) and g′ t (t′ cls ) is the feature representation from the motion encoder, and the image-text / text-image correlation is further defined based on the similarity. The calculation formula is as follows:

[0140]

[0141] Among them, τ is the temperature coefficient which is a trainable hyperparameter, and N is the batch size of training samples;

[0142] C103: The visual modality data and text modality encoded data obtained through contrastive learning are fed into a fully connected layer with softmax for classification to determine whether the image-text pair matches.

[0143] C104, the training phase is based on the ITM task. The ITM loss function is as follows:

[0144]

[0145] Among them, y itm It is a two-dimensional one-hot vector representation, indicating the true similarity, with a positive sample probability of 1 and a negative sample probability of 0. H(·) represents the cross entropy function.

[0146] C105. Output the probability p that the visual modality data matches the text modality data.

[0147] C2. According to the output probability value, the modal data is sent to the corresponding information extraction module. The implementation method is as follows:

[0148] C201, using the probability of visual-textual modal information matching as the criterion for performing multimodal information joint extraction, it is believed that when the matching probability is greater than 0.5, the data performs better under the multimodal information joint extraction model, and vice versa, it performs better under the unimodal information extraction model;

[0149] C202. During the training phase, each piece of data is selected with probability p and 1-p to perform joint extraction of multimodal information and single modal information extraction;

[0150] C203. In the training phase, the image-text data pairs selected for single-modal information extraction are split. The image modal data are fed to the data generation module to generate new image-text modal data pairs as training data. The text modal data are directly fed to the modal information extraction module as training data for the single-modal information extraction module.

[0151] C204. During the usage phase, the model adopts a sampling strategy based on the probability output. Specifically, the data classification label with the output probability p>0.5 is marked as "1", and the rest of the data labels are marked as "0". The data labeled "1" is selected to perform multimodal information joint extraction, and the data labeled "0" is selected to perform single-modal information extraction;

[0152] C205. During the usage phase, the smart building image-text data pairs selected for single-modal information extraction are considered to be unmatched data pairs, that is, the image information in the data interferes with the text knowledge extraction. The image modal data is discarded, and only the text modal information is sent to the modal information extraction module to perform single-modal information extraction; the smart building image-text data pairs selected for multi-modal information extraction are all retained, and the encoded information is sent to the modal fusion layer for information extraction.

[0153] The modal information extraction module includes the following steps:

[0154] D1. Use a single-modal model to extract information. The implementation method is as follows:

[0155] D101, feed the text encoding information into the bidirectional LSTM layer, obtain context information, and feed the information into the hidden layer;

[0156] D102. Input the hidden layer output information into the CRF layer and output the text annotation sequence.

[0157] D2. Use a multimodal model to extract information. The implementation method is as follows:

[0158] D201, feed the text modality encoding information into the bidirectional LSTM layer and hidden layer;

[0159] D202, the visual modality information and the text encoding information are sent to the cross attention layer together with the output of the hidden layer;

[0160] D203, input the cross-modal fusion information into the CRF layer, output sequence annotation, and use softmax combined with cross entropy loss as the training target for training. The cross entropy loss formula is as follows:

[0161]

[0162] Among them, M is the number of entity labels, y ij is the label, if the true label of sample i is consistent with j, it is 1, otherwise it is 0, q ij is the probability that observation sample i belongs to category j.

[0163] In this embodiment, if Figure 2As shown in the figure, the system includes a gating module, a data generation module, a data diversion module, and an information extraction module. Input data first passes through the gating module to select the corresponding processing module. Single visual modality data enters the data generation module, and multimodal data enters the data diversion module. After completing the corresponding stage tasks, it enters the information extraction stage. Single text modality data directly enters the modal information extraction module.

[0164] In this embodiment, the user inputs multimodal data, and the model runs in the following specific process: Figure 4 As shown in the figure, multimodal data passes through the gating module, and different modal data are embedded in vector representation and encoded respectively. The data is then sent to the data diversion module. The data diversion module classifies the input model data according to data quality, dividing it into data suitable for multimodal information extraction and data not suitable for multimodal information extraction. The two types of data are then sent to the information extraction module for multimodal information extraction and unimodal information extraction respectively.

[0165] In this embodiment, the user inputs single visual modality data, and the model runs in the following specific process: Figure 5 As shown in Figure 2, the single visual modality information data is patch-embedded by the gating module to extract visual feature information. The data is then fed into the data generation module, which generates descriptive text based on the visual information. This text is then combined with the visual modality information to form a multimodal information pair. This is then fed into the information extraction module to perform the multimodal information extraction task.

[0166] In this embodiment, the user inputs single text modal data, and the model runs in the following specific process: Figure 6 If only text modal information is input, the data will be encoded by the encoder and then sent directly to the modal information extraction module by the gating module for information extraction.

[0167] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0168] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0169] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0170] The present invention only describes the method of using multimodal information extraction in the field of smart buildings, but it should be noted that the multimodal information extraction model designed by the present invention is not limited to the field of smart buildings. It can be used as a general knowledge extraction model in fields such as finance, medical care, education, etc. The data set used for training can be replaced with the data set of the corresponding field, and a manually labeled small sample data set and a massive data set constructed by crawlers can be used. Then, a domain knowledge extraction model can be constructed through the multimodal knowledge extraction model designed by the present invention.

[0171] Specific embodiments are used in the present invention to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

[0172] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A multimodal smart building knowledge extraction method, characterized by: include: The original input data is processed by the gating module to obtain the data embedding vector representation and encode it. According to the different types of original input data, the data is input into the data generation module, data diversion module or modal information extraction module; The data generation module performs data generation and performs text generation on the single visual modality data encoded by the gating module to obtain image-text pair data and input it into the modality information extraction module. The data quality is screened by the data diversion module, and the visual-text multimodal data encoded by the gating module is differentiated in terms of data quality, and the data is selected to be input into the single-modal information extraction module or the multimodal information extraction module of the modal information extraction module; Information extraction is performed through the modal information extraction module. Information extraction is performed on the single text modal data encoded by the gating module, on the single visual modal data that passes through the gating module and the data generation module, and on the visual-text multimodal data that passes through the gating module and the data diversion module. The specific steps of screening the data quality by the data diversion module, distinguishing the data quality of the visual-textual multimodal data encoded by the gating module, and selecting the single modal information extraction module or the multimodal information extraction module of the modal information extraction module to input the data are as follows: C1. Input the visual-textual multimodal data encoding information output by the gating module into the classifier layer, analyze the similarity characteristics between the two vectors, and output the probability that the visual-textual multimodal data is suitable for multimodal information joint extraction, specifically: C101. To make classification task training more efficient, a hard negative sampling strategy is adopted to sample negative samples closest to positive samples and maintain a negative sample queue of size M. C102. Compare and study the information encoded in the visual modality and the textual modality, and learn the following similarity function: Among them, g v and g t is a linear transformation that maps the joint representation of modal information into a normalized low-dimensional representation, g′ v (v′ cls ) and g′ t (t′ cls ) is the feature representation from the motion encoder, and the image-text / text-image correlation is further defined based on the similarity. The calculation formula is as follows: Among them, τ is the temperature coefficient which is a trainable hyperparameter, and N is the batch size of training samples; C103: The visual modality data and text modality encoded data obtained through contrastive learning are fed into a fully connected layer with softmax for classification to determine whether the image-text pair matches. C104, the training phase is based on the ITM task. The ITM loss function is as follows: Among them, y itm It is a two-dimensional one-hot vector representation, which indicates the true similarity, with a positive sample probability of 1 and a negative sample probability of 0. H() represents the cross entropy function. C105, output the probability p that the visual modality data matches the text modality data; C2. Based on the output probability value, the visual-textual multimodal data is sent to the corresponding single-modal information extraction module or multimodal information extraction module in the modal information extraction module, specifically: C201. Using the probability of visual-textual modal information matching as the criterion for performing multimodal information joint extraction, it is believed that when the matching probability is greater than the mismatch probability, the data performs better under the multimodal information joint extraction model, and vice versa, it performs better under the unimodal information extraction model; C202. During the training phase, each piece of data is selected with probability p and 1-p to perform joint extraction of multimodal information and single modal information extraction; C203. During the training phase, the image-text data pairs selected for single-modal information extraction are split. The image modal data is fed into the data generation module to generate new image-text modal data pairs, and the text modal data is directly fed into the modal information extraction module to perform information extraction. C204. During the usage phase, a sampling strategy is adopted based on the probability output. Specifically, the data classification label with the output probability p>0.5 is marked as "1", and the rest of the data labels are marked as "0". The data labeled "1" is selected to perform multimodal information joint extraction, and the data labeled "0" is selected to perform single-modal information extraction; C205. In the use phase, the image-text data pair for performing single-modal information extraction is selected to discard the image modal data and only the text modal information is sent to the modal information extraction module to perform single-modal information extraction; C206. Select the graphic and text data for performing multimodal information joint extraction, and feed it to the modal information extraction module to first perform graphic and text information interaction.

2. The multimodal smart building knowledge extraction method according to claim 1 is characterized in that: The processing of the original input data by the gating module specifically includes the following steps: A1. Process the original input data to obtain an embedded vector representation and encode it as follows: For visual modality information, the visual modality data is divided into multiple patches. Assuming that the original input image size is H*W*C and the size of each patch is P*P, the number of blocks N is calculated as follows: The size of the one-dimensional vector obtained from the image patch sequence is P*P*C, and the overall input change is N× (P 2 *c) Perform a linear transformation on each vector to map it into a high-dimensional space, obtain the patch embedding through the transformer encoder, denoted as V, and input the encoded modal information into the gating network; For text modal information, the text is input into the embedding layer, and the word2vec method and positionembedding are used to obtain the word embedding vector representation containing position information; the input vector is input into the transformer encoder for encoding, which is recorded as T; Input the encoded modality information into the gating network; A2. Based on the input data characteristics, the gating network is used to send the data to the data generation module, data diversion module, or modal information extraction module, as follows: Decompose the information extraction task into data generation, data screening, and modal information extraction subtasks. Train an expert model on each subtask, learn which expert to use based on the input to be predicted, and combine the prediction results. When the input information is single visual modality data, the data is sent to the data generation module; When the input information is single text modal data, the data is sent to the modal information extraction module; When the input information is visual-text multimodal data, the data is sent to the data diversion module.

3. The multimodal smart building knowledge extraction method according to claim 1 is characterized in that: The data generation module specifically uses the following method to perform data generation: During the training phase of the model, an image-based text transformer decoder is used as the data generator and the decoder parameters are initialized; During the model training phase, the text generation task is performed on the input single visual modality data to generate image-text pair data; During the model usage phase, the text generation task is performed on the input single visual modality data to generate image-text pair data.

4. The multimodal smart building knowledge extraction method according to claim 1 is characterized in that: The method for initializing the decoder parameters is: B101. Use manually annotated smart building datasets to train the generation module; B102. Use an image-based text transformer decoder as a data generator; B103, use the special mark [MASK] to token n Subsequent information is concealed; B104, feed the encoded data into the cross-attention layer so that the model can learn the cross-modal fusion information in the training data; B105. Use language modeling loss as the training objective to maximize the likelihood of text generation in an autoregressive manner. The objective function is as follows: Where token is a word-level unit, k is the window size, n indicates that the token is the nth token, and θ is the hyperparameter to be learned; B106. The decoder model used in the data generation module and the text encoding model of the modal information extraction module share parameters in the cross-attention mechanism layer.

5. The multimodal-based smart building knowledge extraction method according to claim 1 is characterized in that: During the training phase of the model, the text generation task is performed on the input single visual modality data to generate image-text pair data, specifically: B201. During the training phase, the data generation module receives the visual modality information output from the data diversion module; B202. Performing a text generation task on the separately extracted visual modal information to generate new cross-modal information data; B203. Store the generated data with better quality as new training set data.

6. The multimodal smart building knowledge extraction method according to claim 1 is characterized in that: In the model usage phase, the text generation task is performed on the input single visual modality data to generate image-text pair data, specifically: B301, the data generation module receives the visual modality encoding information from the gating module; B302, inputting the visual coding information into the decoder; B303. The decoder outputs the generated text modal information and the visual modal information and sends them to the modal interaction module.

7. The multimodal smart building knowledge extraction method according to claim 1 is characterized in that: The modal information extraction module performs information extraction using the following method: D1. Use a single modal model for information extraction; D101, feed the text encoding information into the bidirectional LSTM layer, obtain context information, and feed the information into the hidden layer; D102, input the hidden layer output information into the CRF layer and output the text annotation sequence; D2. Use multimodal models for information extraction; D201, feed the text modality encoding information into the bidirectional LSTM layer and hidden layer; D202, the visual modality information and the text encoding information are sent to the cross attention layer together with the output of the hidden layer; D203, input the cross-modal fusion information into the CRF layer, output sequence annotation, and use softmax combined with cross entropy loss as the training target for training. The cross entropy loss formula is as follows: Among them, M is the number of entity labels, y ij is a label. If the true label of sample i is consistent with j, it is 1, otherwise it is 0. ij is the probability that observation sample i belongs to category j.

8. A multimodal smart building knowledge extraction model, characterized by: A multimodal smart building knowledge extraction method according to any one of claims 1 to 7 is adopted, comprising a gating module, a data generation module, a data diversion module, and a modal information extraction module; The gating module is used to process the original input data, obtain the data embedding vector representation and encode it, and select the data input into the data generation module, data diversion module or modal information extraction module according to the different types of the original input data; The data generation module is used to perform data generation, perform text generation tasks on the single visual modality data encoded by the gating module, obtain image-text pair data and input it into the modality information extraction module; The data diversion module is used for data quality screening, distinguishing the data quality of the visual-textual multimodal data encoded by the gating module, and selecting the single-modal information extraction module or the multimodal information extraction module of the modal information extraction module to input it; The modal information extraction module is used to perform information extraction, extracting information from the single text modal data encoded by the gating module, extracting information from the single visual modal data that passes through the gating module and the data generation module, and extracting information from the visual-text multimodal data that passes through the gating module and the data diversion module.

9. The multimodal smart building knowledge extraction model according to claim 8 is characterized in that: The data diversion module and data generation module are also used for knowledge extraction model training; The data generation module is used to receive visual modality data eliminated after the data diversion module in the training phase, obtain image-text pair data, and select data with better evaluation results to form a new training set for training the knowledge extraction model.

Images