A knowledge-based multi-modal meeting summary generation method

Abstract

This invention provides a knowledge-based multimodal meeting summarization method, comprising the following steps: S1, constructing a multimodal meeting knowledge graph; S2, a meeting knowledge embedding model based on knowledge aggregation; S3, multimodal embedding based on image-audio encoders and text encoders; S4, a facial emotion recognition collaborative training mechanism; and S5, meeting decoding based on Transformer-Decoder. This method achieves the following: realizing the important role of audio, video, and emotion data in understanding meeting content, breaking down data type barriers, and comprehensively integrating and mining meeting information for summarization generation.

Description

Technical Field

[0001] This invention relates to the field of conference summarization technology, and specifically to a knowledge-based multimodal conference summarization generation method. Background Technology

[0002] With the rapid development of communication technology, online meeting tools such as Tencent Meeting, DingTalk, Zoom, Lark, and Huawei Cloud Meeting have become emerging models for cross-enterprise and cross-regional communication and collaboration. Online meetings have the unique advantage of not being limited by time and space, allowing people to have "face-to-face" discussions anytime, anywhere, truly realizing the saying "Though separated by vast distances, true friends are never far apart." According to the 50th Statistical Report on Internet Development in China, the number of online office users in my country has reached 461 million, accounting for 43.8% of all internet users, leading to the widespread use of online meeting software. However, with the growth of users, online meeting platforms have also generated a large amount of meeting data, including video, audio, and text information, which poses challenges to meeting recording and extraction. During meetings, users are often surrounded by lengthy or fragmented information. How to help users quickly locate the core content from complex meeting information and free up their energy has become an urgent need. Today, summarization technology is widely used to extract core information from corpora, which greatly facilitates people's understanding of important content and saves reading time.

[0003] Based on different classification criteria, abstract generation tasks can be divided into different subtasks. From the perspective of input data, they can be divided into news abstracts, patent abstracts, paper abstracts, and conference abstracts, etc. From the perspective of generation mode, they can be divided into extractive abstracts and generative abstracts: In recent years, generative abstracts have mainly adopted encoding and decoding architectures, including mainstream technologies such as seq2seq and transformers.

[0004] Generative methods, which align more closely with human thinking habits, have been widely studied in the industry. However, existing generative models are not effectively applicable to meeting scenarios and still suffer from the following problems: 1) Unlike single-text data scenarios, online meeting scenarios contain rich audio and video data. Therefore, it is necessary to break down data type barriers and comprehensively integrate and mine meeting information for summary generation. 2) Most existing models mainly focus on text data, spending a lot of effort learning structural and role information, while neglecting the supplementary role of audio and video data in understanding meeting content. Research shows that language conveys only 7% of the total information, while facial expressions and voice convey much more. Therefore, it is necessary to leverage multimodal information to enhance the performance of summarization models. Summary of the Invention

[0005] The present invention aims to provide a knowledge-based multimodal meeting summary generation method to solve the above problems.

[0006] The technical solution of this invention is: a knowledge-based multimodal meeting summary generation method, comprising the following steps:

[0007] S1, Construct a multimodal meeting knowledge graph;

[0008] S2, a conference knowledge embedding model based on knowledge aggregation;

[0009] S3, a multimodal embedding based on image-audio encoders and text encoders;

[0010] S4, a collaborative training mechanism for facial emotion recognition;

[0011] S5, a conference decoding based on Transformer-Decoder.

[0012] Preferably, the multimodal meeting knowledge graph in S1 includes a three-layer architecture to effectively organize the knowledge from online meetings; specifically:

[0013] The first layer is the basic information layer G. basic The task is to collect information on meeting participants and their basic attributes, including the participants, their relationships, and the meeting location.

[0014] The second layer is the multimodal information layer G. multimodal This is used to store multimodal information during a meeting, including audio, visual, text, and shared files; where shared files are processed as images and considered as supplementary visual information; specifically, given a text sequence S = {S1, S2, ..., S...} of a meeting... n For each text node S k For each element, 1 ≤ k ≤ n, record its corresponding text information, visual information, and audio information respectively. For each text node, the proposed storage model focuses on its corresponding textual, visual, and audio information. The textual and audio information are fully preserved in the multimodal layer, while the visual information only retains images from three key moments: the participant's statement S. k The image information corresponding to the start, middle, and end times; in addition, S is also considered. k The corresponding screenshot of the shared document was combined with images from the three time points to form S. k Visual information;

[0015] The third layer is the meeting knowledge layer G. knowledge This is used to integrate conference domain knowledge and common-sense knowledge from multimodal information, including extracting domain knowledge from conference text, introducing common-sense knowledge into dialogues from the large-scale common-sense knowledge base ConceptNet, and extracting facial and spoken emotions from images and audio; specifically including:

[0016] First, text is extracted from screenshots using an OCR text recognition tool. Then, DeepDive is used to extract domain knowledge entities, relationships, and attributes from meeting minutes and screenshot text.

[0017] Secondly, S k Each word is treated as a query term. Common sense knowledge tuples with the query term as the core are searched in ConceptNet, and tuples with confidence scores greater than the threshold are added to the conference knowledge layer.

[0018] Finally, considering that emotional information is implicit in the participants' voices and images, for the same text, facial emotions and spoken emotions are extracted from images and audio respectively, and these two emotions are regarded as important knowledge and stored in the meeting knowledge layer.

[0019] Preferably, S2 specifically includes the following steps:

[0020] S21, Apply the TransE tool to process the initial knowledge representation of entities and relationships in the knowledge graph learning graph;

[0021] S22, for each text node S k Centered on it, the knowledge graph links are used to traverse the multi-hop neighbor nodes in sequence to aggregate the knowledge representation.

[0022] S23, for the m-th hop neighbor node, use a local attention activation component to identify important knowledge nodes and adaptively assign weights, using the weighted sum of the embeddings of the knowledge nodes as S. k The knowledge representation of the m-th jump; specifically:

[0023] Given text node S k Initialization indicates Its m-th hop neighbor knowledge node For any knowledge node j The weights are calculated as follows:

[0024]

[0025] in, Represents a knowledge node j Knowledge weight, Knowledge node j The vector transpose representation of . Represents a knowledge node p The vector transpose of , where exp represents an exponential function with the natural constant e as the base;

[0026] After obtaining the weight coefficients, S is aggregated by calculating the weighted sum of all knowledge nodes in this hop.k Knowledge representation:

[0027]

[0028] S represents k Knowledge representation of the m-th jump Represents a knowledge node j Vectorized representation of;

[0029] S24, due to its distance from S k The further away a knowledge node is, the smaller its knowledge influence. Therefore, a knowledge weakening coefficient is designed to reduce S. k The knowledge vector of the meeting is obtained by summing the knowledge representations of each jump.

[0030]

[0031] in, This represents the knowledge reduction coefficient, used to reduce the impact of m-hop knowledge nodes on S. k The impact; S represents k Knowledge representation of the m-th jump S represents k Knowledge representation of the first jump.

[0032] Preferably, S3 specifically includes the following steps:

[0033] S31, for S k and the corresponding image I k and audio Au k The initial vector representations of the text, images, and audio are obtained respectively;

[0034] S32, will Sentence S k The nth word is input into the Bert tool for token embedding, segment embedding, and position embedding. The feature vectors obtained from the three embeddings are summed to obtain S. k Word-level embedding representation

[0035] S33, for image I k Fine-grained visual feature representations are obtained using ResNet processing.

[0036] S34, for audio Au k Fine-grained auditory features were obtained by processing with the VGGish tool.

[0037] S35, for the image-to-audio encoder, use a CNN-Encoder to... and Embedded into a multimodal common space, forward multimodal features are obtained. Specifically:

[0038] The encoder uses CNN-Encoder to... and Embedded into a multimodal common space, where CNN-Encoder represents a convolutional neural network-based encoder, whose function is to reduce the dimensionality of the original vectors to obtain the implicit vector representations of the original images, audio, and text:

[0039]

[0040]

[0041] in, and It uses the multimodal latent representations obtained by the encoder, and adds the two together to form the forward multimodal features.

[0042]

[0043] S36, For the text encoder, use CNN-Encoder to... Embedded into a multimodal common space, backward multimodal features are obtained.

[0044]

[0045] S37, multimodal embedding training is performed by constraining the forward and backward multimodal features of positive examples to be close in the multimodal common space, and the negative examples to be far apart; specifically:

[0046] For image-audio and text positive examples (I) k Au k S k For example, the forward and backward multimodal characteristics of the constrained solution are similar in the multimodal common space; conversely, for negative examples (I... k Au k S n Since n ≠ k, the forward and backward multimodal features obtained from the constraint solution are far apart in the multimodal common space; therefore, the following loss function is designed to train the above encoder:

[0047]

[0048] Where D is a dataset containing positive and negative examples, ε is a constant, and S nIt randomly extracts text from the corpus that does not match the images and audio. For I k and Au k Forward multimodal features, For example S k Backward multimodal features, For negative example S n The backward multimodal features are used, and the cosine similarity function is used to evaluate the distance between the forward and backward multimodal features in the multimodal space.

[0049] Preferably, S4 is used to enhance the generalization capability of multimodal public spaces, specifically including the following steps:

[0050] S41, based on forward multimodal representation The PyTorchDNNClassifier classifier neural network is used to identify the current user's emotion. The loss function for the emotion recognition task is as follows:

[0051]

[0052] Where L represents the total number of emotional expression labels, primarily considering 7 categories: anger, disgust, fear, happiness, sadness, surprise, and normal; y i ∈{0,1} is the truth value of emotion, indicating whether the current user has expressed the i-th type of emotion. This represents the predicted probability of the classifier whether the user represents the i-th type of emotion. The emotion recognition task is trained using the multi-class cross-entropy loss function.

[0053] S42, co-training of meeting multimodal information embedding is performed by minimizing the total loss of multimodal embedding and emotion recognition. After training, the average forward and backward multimodal features are used as the multimodal vector representation of the meeting. Specifically, it includes:

[0054]

[0055] Where K represents the total amount of text, This represents the multimodal embedding loss. This indicates a loss of emotion recognition. The overall loss function is used to train the forward and backward multimodal features of the meeting by jointly optimizing the total loss of the two tasks, namely minimizing the multimodal embedding loss and the emotion recognition loss;

[0056] For S k Its multimodal vector as follows:

[0057]

[0058] Here, Average represents the average pooling of vectors. Forward multimodal features, It is a backward multimodal feature;

[0059] S43, Representing Knowledge Vectors and multimodal vector representation By combining them, a text S is constructed. k The final vector representation

[0060] S44, representing the vector sequence of meeting text. Inputting the data into a Long Short-Term Memory (LSTM) network yields the hidden vector representation M of the meeting. hidden |S| represents the number of text sequences.

[0061] Preferably, S5 specifically includes the following steps:

[0062] S51, representing the implicit vector of the meeting as M hidden The input is given to the decoder. At the start of decoding, a start token is input to activate the decoder, and combined with M... hidden We obtain an output prediction;

[0063] S52, in each subsequent decoding step, the decoder uses M... hidden The output of the previous step is used as input to generate the next output prediction;

[0064] S53, repeat the above process until a special termination symbol is reached.

[0065] Preferably, during the S52 and S53 decoding processes, the output of each time step is provided to the bottom of the decoder in the next time step; specifically, this includes the following steps:

[0066] S54 embeds word vectors and adds positional encoding as input to the decoder to represent the position of each word;

[0067] S55 outputs a real number vector through structures such as multi-head attention layers, regularization, and feedforward networks;

[0068] S56 performs a linear transformation and the Softmax function on the real number vector. The Softmax function will transform the vector into a positive probability value with an upper limit of 1.0.

[0069] S57, the cell with the highest probability value is selected, and its corresponding word is used as the output of this time step to generate the final conference summary;

[0070] S58, for any conference M and its annotated conference abstract Sum M={word1,word2,...,word n The training objective of this invention is to minimize the prediction loss between the predicted word and the target word.

[0071]

[0072] The beneficial effects of this invention are as follows:

[0073] This invention constructs a multimodal meeting knowledge graph by integrating multidimensional and fragmented information from meetings. Combining the complementarity between multimodal information, a multimodal embedding method is proposed to capture a fusion of visual and auditory meeting context representations. Knowledge features are extracted from the meeting knowledge graph based on graph network algorithms, enhancing the understanding of meeting content with the help of knowledge information. Furthermore, high-quality meeting summaries are generated by comprehensively utilizing multimodal and knowledge information from the meeting. This demonstrates the crucial role of audio-visual and emotional data in understanding meeting content, breaking down data type barriers and comprehensively integrating and mining meeting information for summary generation. Attached Figure Description

[0074] Figure 1 A flowchart illustrating a knowledge-based multimodal meeting summary generation method provided in this embodiment of the invention;

[0075] Figure 2 This is a schematic diagram illustrating the construction of a multimodal meeting knowledge graph used in a knowledge-based multimodal meeting summary generation method provided in an embodiment of the present invention.

[0076] Figure 3 A schematic diagram of a conference multimodal information embedding framework based on joint coding, used in a knowledge-based multimodal conference summary generation method provided in an embodiment of the present invention;

[0077] Figure 4 A schematic diagram of the LSTM-based meeting latent vector solving model used in a knowledge-based multimodal meeting summary generation method provided in an embodiment of the present invention;

[0078] Figure 5 This is a schematic diagram of a conference summary decoder used in a knowledge-based multimodal conference summary generation method provided in an embodiment of the present invention. Detailed Implementation

[0079] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. The embodiments of the present invention are not limited thereto.

[0080] Example 1

[0081] like Figure 1As shown, a knowledge-based multimodal meeting summary generation method includes the following steps:

[0082] S1, Construct a multimodal meeting knowledge graph;

[0083] S2, a conference knowledge embedding model based on knowledge aggregation;

[0084] S3, a multimodal embedding based on image-audio encoders and text encoders;

[0085] S4, a collaborative training mechanism for facial emotion recognition;

[0086] S5, a conference decoding based on Transformer-Decoder.

[0087] like Figure 2 As shown, the multimodal meeting knowledge graph in S1 comprises a three-layer architecture, effectively organizing the knowledge from online meetings; specifically:

[0088] The first layer is the basic information layer G. basic The task is to collect information on meeting participants and their basic attributes, including the participants, their relationships, and the meeting location.

[0089] The second layer is the multimodal information layer G. multimodal This is used to store multimodal information during a meeting, including audio, visual, text, and shared files; where shared files are processed as images and considered as supplementary visual information; specifically, given a text sequence S = {S1, S2, ..., S...} of a meeting... n For each text node S k For each element, 1 ≤ k ≤ n, record its corresponding text information, visual information, and audio information respectively. For each text node, the proposed storage model focuses on its corresponding textual, visual, and audio information. The textual and audio information are fully preserved in the multimodal layer, while the visual information only retains images from three key moments: the participant's statement S. k The image information corresponding to the start, middle, and end times; in addition, S is also considered. k The corresponding screenshot of the shared document was combined with images from the three time points to form S. k Visual information;

[0090] The third layer is the meeting knowledge layer G. knowledge This is used to integrate conference domain knowledge and common-sense knowledge from multimodal information, including extracting domain knowledge from conference text, introducing common-sense knowledge into dialogues from the large-scale common-sense knowledge base ConceptNet, and extracting facial and spoken emotions from images and audio; specifically including:

[0091] First, text is extracted from screenshots using an OCR text recognition tool. Then, DeepDive is used to extract domain knowledge entities, relationships, and attributes from meeting minutes and screenshot text.

[0092] Secondly, S k Each word is treated as a query term. Common sense knowledge tuples with the query term as the core are searched in ConceptNet, and tuples with confidence scores greater than the threshold are added to the conference knowledge layer.

[0093] Finally, considering that emotional information is implicit in the participants' voices and images, for the same text, facial emotions and spoken emotions are extracted from images and audio respectively, and these two emotions are regarded as important knowledge and stored in the meeting knowledge layer.

[0094] S2 specifically includes the following steps:

[0095] S21, Apply the TransE tool to process the initial knowledge representation of entities and relationships in the knowledge graph learning graph;

[0096] S22, for each text node S k Centered on it, the knowledge graph links are used to traverse the multi-hop neighbor nodes in sequence to aggregate the knowledge representation.

[0097] S23, for the m-th hop neighbor node, use a local attention activation component to identify important knowledge nodes and adaptively assign weights, using the weighted sum of the embeddings of the knowledge nodes as S. k The knowledge representation of the m-th jump; specifically:

[0098] Given text node S k Initialization indicates Its m-th hop neighbor knowledge node For any knowledge node j The weights are calculated as follows:

[0099]

[0100] in, Represents a knowledge node j Knowledge weight, Knowledge node j The vector transpose representation of . Represents a knowledge node p The vector transpose of , where exp represents an exponential function with the natural constant e as the base;

[0101] After obtaining the weight coefficients, S is aggregated by calculating the weighted sum of all knowledge nodes in this hop. kKnowledge representation:

[0102]

[0103] S represents k Knowledge representation of the m-th jump Represents a knowledge node j Vectorized representation of;

[0104] S24, due to its distance from S k The further away a knowledge node is, the smaller its knowledge influence. Therefore, a knowledge weakening coefficient is designed to reduce S. k The knowledge vector of the meeting is obtained by summing the knowledge representations of each jump.

[0105]

[0106] in, This represents the knowledge reduction coefficient, used to reduce the impact of m-hop knowledge nodes on S. k The impact; S represents k Knowledge representation of the m-th jump S represents k Knowledge representation of the first jump.

[0107] like Figure 3 As shown, S3 specifically includes the following steps:

[0108] S31, for S k and the corresponding image I k and audio Au k The initial vector representations of the text, images, and audio are obtained respectively;

[0109] S32, will Sentence S k The nth word is input into the Bert tool for token embedding, segment embedding, and position embedding. The feature vectors obtained from the three embeddings are summed to obtain S. k Word-level embedding representation

[0110] S33, for image I k Fine-grained visual feature representations are obtained using ResNet processing.

[0111] S34, for audio Au k Fine-grained auditory features were obtained by processing with the VGGish tool.

[0112] S35, for the image-to-audio encoder, use a CNN-Encoder to... and Embedded into a multimodal common space, forward multimodal features are obtained. Specifically:

[0113] The encoder uses CNN-Encoder to... and Embedded into a multimodal common space, where CNN-Encoder represents a convolutional neural network-based encoder, whose function is to reduce the dimensionality of the original vectors to obtain the implicit vector representations of the original images, audio, and text:

[0114]

[0115]

[0116] in, and It uses the multimodal latent representations obtained by the encoder, and adds the two together to form the forward multimodal features.

[0117]

[0118] S36, For the text encoder, use CNN-Encoder to... Embedded into a multimodal common space, backward multimodal features are obtained.

[0119]

[0120] S37, multimodal embedding training is performed by constraining the forward and backward multimodal features of positive examples to be close in the multimodal common space, and the negative examples to be far apart; specifically:

[0121] For image-audio and text positive examples (I) k Au k S k For example, the forward and backward multimodal characteristics of the constrained solution are similar in the multimodal common space; conversely, for negative examples (I... k Au k S n Since n ≠ k, the forward and backward multimodal features obtained from the constraint solution are far apart in the multimodal common space; therefore, the following loss function is designed to train the above encoder:

[0122]

[0123] Where D is a dataset containing positive and negative examples, ε is a constant, and S nIt randomly extracts text from the corpus that does not match the images and audio. For I k and Au k Forward multimodal features, For example S k Backward multimodal features, For negative example S n The backward multimodal features are used, and the cosine similarity function is used to evaluate the distance between the forward and backward multimodal features in the multimodal space.

[0124] S4 is used to enhance the generalization ability of multimodal public spaces, specifically including the following steps:

[0125] S41, based on forward multimodal representation The PyTorchDNNClassifier classifier neural network is used to identify the current user's emotion. The loss function for the emotion recognition task is as follows:

[0126]

[0127] Where L represents the total number of emotional expression labels, primarily considering 7 categories: anger, disgust, fear, happiness, sadness, surprise, and normal; y i ∈{0,1} is the truth value of emotion, indicating whether the current user has expressed the i-th type of emotion. This represents the predicted probability of the classifier whether the user represents the i-th type of emotion. The emotion recognition task is trained using the multi-class cross-entropy loss function.

[0128] S42, co-training of meeting multimodal information embedding is performed by minimizing the total loss of multimodal embedding and emotion recognition. After training, the average forward and backward multimodal features are used as the multimodal vector representation of the meeting. Specifically, it includes:

[0129]

[0130] Where K represents the total amount of text, This represents the multimodal embedding loss. This indicates a loss of emotion recognition. The overall loss function is used to train the forward and backward multimodal features of the meeting by jointly optimizing the total loss of the two tasks, namely minimizing the multimodal embedding loss and the emotion recognition loss;

[0131] For S k Its multimodal vector as follows:

[0132]

[0133] Here, Average represents the average pooling of vectors. Forward multimodal features, It is a backward multimodal feature;

[0134] S43, Representing Knowledge Vectors and multimodal vector representation By combining them, a text S is constructed. k The final vector representation

[0135] like Figure 4 As shown in S44, the vector representation of the conference text sequence is... Inputting the data into a Long Short-Term Memory (LSTM) network yields the hidden vector representation M of the meeting. hidden |S| represents the number of text sequences.

[0136] like Figure 5 As shown, S5 specifically includes the following steps:

[0137] S51, representing the implicit vector of the meeting as M hidden The input is given to the decoder. At the start of decoding, a start token is input to activate the decoder, and combined with M... hidden We obtain an output prediction;

[0138] S52, in each subsequent decoding step, the decoder uses M... hidden The output of the previous step is used as input to generate the next output prediction;

[0139] S53, repeat the above process until a special termination symbol is reached.

[0140] Preferably, during the S52 and S53 decoding processes, the output of each time step is provided to the bottom of the decoder in the next time step; specifically, this includes the following steps:

[0141] S54 embeds word vectors and adds positional encoding as input to the decoder to represent the position of each word;

[0142] S55 outputs a real number vector through structures such as multi-head attention layers, regularization, and feedforward networks;

[0143] S56 performs a linear transformation and the Softmax function on the real number vector. The Softmax function will transform the vector into a positive probability value with an upper limit of 1.0.

[0144] S57, the cell with the highest probability value is selected, and its corresponding word is used as the output of this time step to generate the final conference summary;

[0145] S58, for any conference M and its annotated conference abstract Sum M ={word1,word2,...,word n The training objective of this invention is to minimize the prediction loss between the predicted word and the target word.

[0146]

[0147] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the processes depicted in the drawings are not necessarily essential for implementing the present invention.

Claims

1. A knowledge-based multimodal meeting summary generation method, characterized in that, Includes the following steps: S1, Construct a multimodal meeting knowledge graph; S2, a conference knowledge embedding model based on knowledge aggregation; S3, a multimodal embedding based on image-audio encoders and text encoders; S4, facial emotion recognition collaborative training mechanism; S5, a conference decoding system based on Transformer-Decoder; Specifically, S2 includes the following steps: S21, Apply the TransE tool to process the initial knowledge representation of entities and relationships in the knowledge graph learning graph; S22, for each text node around which, along the links of the knowledge graph, the multi-hop neighbor nodes are sequentially traversed to aggregate the knowledge representation; S23, for the first hop neighbor nodes, using a local attention activation component to identify important knowledge nodes and adaptively assigning weights to the weighted sum of the embeddings of the knowledge nodes of the first hop knowledge representation; specifically: Given a text node , initialize a representation , which is the first hop neighbor knowledge node , for any knowledge node , the weight is calculated as follows: ； in, Representing knowledge nodes Knowledge weight, Knowledge Nodes The vector transpose representation of . Representing knowledge nodes The vector transpose of , where exp represents an exponential function with the natural constant e as the base; After obtaining the weight coefficients, the knowledge representation of this hop is aggregated by calculating the weighted sum of all knowledge nodes of this hop ​ ； express No. Knowledge representation of jumping, Representing knowledge nodes Vectorized representation of; S24, due to distance The further away a knowledge node is, the smaller its knowledge influence. Therefore, a knowledge weakening coefficient is designed to... The knowledge vector of the meeting is obtained by summing the knowledge representations of each jump. : ； in, Represents the knowledge weakening coefficient, used to weaken... Skipping knowledge nodes is important. The impact; express No. Knowledge representation of jumping, express Knowledge representation of the first jump; S3 specifically includes the following steps: S31, for and the corresponding image and audio The initial vector representations of the text, images, and audio are obtained respectively; S32, will , Representative sentence The Take a word, input it into the Bert tool, and perform token embedding, segment embedding, and position embedding. Sum the feature vectors obtained from the three embeddings to get... Word-level embedding representation ; S33, for the image , using ResNet to obtain its fine-grained visual feature representation ; S34, for audio , process to get fine-grained auditory features with VGGish tool ; S35, for image-to-audio encoders, uses Will and Embedded into a multimodal common space, forward multimodal features are obtained. Specifically: Encoder use Will and Embedded into a multimodal common space, where CNN-Encoder represents a convolutional neural network-based encoder, whose function is to reduce the dimensionality of the original vectors to obtain the implicit vector representations of the original images, audio, and text: ； ； wherein, and are multimodal latent representations obtained using an encoder, adding both as forward multimodal features : ； S36, for text encoders, utilizes Will Embedded into a multimodal common space, backward multimodal features are obtained. : ； S37, multimodal embedding training is performed by constraining the forward and backward multimodal features of positive examples to be close in the multimodal common space, and the negative examples to be far apart; specifically: For positive examples of image-audio and text ( The forward and backward multimodal features of the constrained solution are similar in the multimodal common space; conversely, for negative examples ( ), The forward and backward multimodal features obtained from the constraint solution are far apart in the multimodal common space; therefore, the following loss function is designed to train the encoder: ； in, It is a dataset containing both positive and negative examples. It is a constant. It randomly extracts text from the corpus that does not match the images and audio. for and Forward multimodal features, As a positive example backward multimodal features, negative example The backward multimodal features are used, and the cosine similarity function is used to evaluate the distance between the forward and backward multimodal features in the multimodal space; S4 is used to enhance the generalization ability of multimodal public spaces, specifically including the following steps: S41, based on the forward multi-modal representation The PyTorch DNNClassifier docking classification neural network identifies the current user's emotion, and the loss function of the emotion recognition task is represented as follows: ； where L is the total number of emotion expression labels, considering 7 categories: anger, disgust, fear, happiness, sadness, surprise, normal, is the emotion ground truth, indicating whether the current user is expressing the i-th emotion, is the prediction probability of the classifier on whether the user is expressing the i-th emotion, and the multi-class cross-entropy loss function is used to train the emotion recognition task. S42, cooperative training of conference multi-modal information embedding is performed by minimizing the total loss of multi-modal embedding and emotion recognition, and after the training is completed, the average forward-backward multi-modal features and the multi-modal vector representation of the conference ; specifically comprising: ； in, Indicates the total amount of text. This represents the multimodal embedding loss. This indicates a loss of emotion recognition. The overall loss function is used to train the forward and backward multimodal features of the meeting by jointly optimizing the total loss of the two tasks, namely minimizing the multimodal embedding loss and the emotion recognition loss; For its multi-modal vector is as follows: ； wherein, Average represents the average pooling of the vector, is the forward multimodal feature, is the backward multimodal feature; S43, knowledge vector representation and multimodal vector representation are concatenated to build a final vector representation of the text , ;​ S44, vector representation of the sequence of meeting text input long short-term memory network, obtaining a latent vector representation of the meeting , denotes the number of text sequences; S5 specifically includes the following steps: S51, the implicit vector representation of the meeting is obtained The input to the decoder, at the start of decoding, is a start token that activates the decoder and combines An output prediction is obtained; S52, in each subsequent decoding step, the decoder takes as input and the output of the previous step, to generate the next output prediction; S53, repeat the above process until a special termination symbol is reached. 2.The knowledge-based multi-modal meeting summarization method of claim 1, wherein, The S1 multimodal meeting knowledge graph comprises a three-layer architecture, effectively organizing the knowledge from online meetings; specifically: The first layer is a basic information layer , responsible for collecting information of participants and basic attribute information of the conference, including conference participants, relationships and conference locations; The second layer is the multimodal information layer. This is used to store multimodal information during meetings, including audio, visual, text, and shared files; wherein shared files are processed as images and considered as supplementary visual information; specifically, given a text sequence of a meeting... For each text node , Record the corresponding text information, visual information and audio information respectively. Specifically, for each text node, the proposed storage model focuses on its corresponding textual, visual, and audio information; among them, the textual and audio information are fully preserved in the multimodal layer, while the visual information only retains images from three important moments, namely, the participants' statements. Image information corresponding to the start, middle, and end times; in addition, it also considers The corresponding screenshot of the shared document was combined with images from the three different times as... Visual information; The third layer is the meeting knowledge layer. This is used to integrate conference domain knowledge and common sense knowledge from multimodal information, including extracting domain knowledge from conference text, introducing common sense knowledge into dialogues from the large-scale common sense knowledge base ConceptNet, and extracting facial and spoken emotions from images and audio; specifically including: First, text is extracted from screenshots using an OCR text recognition tool. Then, DeepDive is used to extract domain knowledge entities, relationships, and attributes from meeting minutes and screenshot text. Secondly, Each word is treated as a query term. Common sense knowledge tuples with the query term as the core are searched in ConceptNet, and tuples with confidence scores greater than the threshold are added to the conference knowledge layer. Finally, considering that emotional information is implicit in the participants' voices and images, for the same text, facial emotions and spoken emotions are extracted from images and audio respectively, and these two emotions are regarded as important knowledge and stored in the meeting knowledge layer.

3. The method of using the knowledge-based multimodal meeting summary generation method according to claim 1, characterized in that, During S52 and S53 decoding, the output of each time step is provided to the bottom of the decoder in the next time step; specifically, the following steps are included: S54 embeds word vectors and adds positional encoding as input to the decoder to represent the position of each word; S55 outputs a real number vector through structures such as multi-head attention layers, regularization, and feedforward networks; S56 performs a linear transformation and the Softmax function on the real number vector. The Softmax function will transform the vector into a positive probability value with an upper limit of 1.

0. S57, the cell with the highest probability value is selected, and its corresponding word is used as the output of this time step to generate the final conference summary; S58, for any meeting and its annotated meeting summary The training objective is to minimize the prediction loss between the predicted word and the target word. 。

Images