A Method for Extracting Unstructured Intelligence Topics from Conversational Interactions Using a Multimodal Transformer Model

Abstract

This invention discloses a method for extracting unstructured intelligence topics through conversational interaction using a multimodal Transformer model. The method includes: inputting multimodal data; preprocessing the multimodal data; inputting the preprocessed multimodal data into a Transformer model for cross-fusion and decoding to obtain intelligence topic text; calculating the topic coverage loss of the generated intelligence topic text based on a coverage function with adjustable parameters, updating the parameters of the Transformer model to enable it to output customized intelligence topic text; allowing the user to input text information to further specify the intelligence topic and implement multi-round interactions; extracting the interaction information; and iteratively extracting topics to deeply understand user needs and contextual intelligence information. This invention can effectively extract multimodal intelligence topics, improving the comprehensiveness of intelligence topic extraction while further enhancing the accuracy of intelligence extraction and optimizing the user experience through conversational interaction.

Description

Technical Field

[0001] This invention relates to a method for extracting unstructured intelligence topics, specifically a method for extracting unstructured intelligence topics from conversational interactions using a multimodal Transformer model. Background Technology

[0002] In today's information-saturated era, massive amounts of unstructured intelligence data, such as social media content, online news, and scientific papers, flood our field of vision, posing enormous challenges to information processing and analysis. Topic extraction, as a crucial step in information processing, aims to extract representative and meaningful thematic information from this data to help users quickly understand and obtain the information they need. However, traditional topic extraction methods are often limited to single types of data, failing to fully utilize the correlations between different modalities of data, and lacking effective user interaction, resulting in low accuracy and insufficient targeting of the extraction results.

[0003] Currently, methods combining single-modal text generation and multimodal fusion mechanisms have some shortcomings. There may be an imbalance in the quantity of data across different modalities, leading to the neglect or underestimation of certain modal information during the fusion process, thus affecting the comprehensiveness and objectivity of topic extraction. Furthermore, the modal fusion strategies are insufficient; current multimodal fusion mechanisms may have limitations. These methods may fail to fully exploit the inherent connections and semantic information between different modalities, resulting in poor fusion effects and impacting the accuracy and comprehensiveness of topic extraction. Summary of the Invention

[0004] The purpose of this invention is to provide a method for extracting unstructured intelligence topics from conversational interactions using a multimodal Transformer model, thereby helping users to accurately understand and utilize key information in multimodal intelligence through multi-turn conversational interactions.

[0005] To achieve the above objectives, the specific solution of the present invention is as follows:

[0006] Step S101: Input multimodal data, which includes audio, text, and images;

[0007] Step S102: Select the respective encoding schemes for the input multimodal data and perform data preprocessing to obtain the embedding sequence of each modality. The audio is encoded using the log-Mel spectrum embedding model; the text is first encoded using one-hot encoding for word vectors, and then positional encoding information is added; the image is processed using the ViT model, and positional encoding information of the image is added.

[0008] Step S103: Input the embedding sequence of each modality into the Transformer encoder containing the cross-fusion layer to generate a multimodal fusion sequence; output the decoded sequence through the Transformer decoder, use SoftMax to calculate the probability of words containing intelligence topic information, select the word with the highest probability, and iteratively generate the next output character until the complete intelligence topic text is generated;

[0009] Step S104: Calculate the topic coverage loss of the generated intelligence topic text based on the coverage function of the adjustable parameters, update the parameters of the Transformer model, and enable the Transformer model to output customized intelligence topic text.

[0010] Step S105: The user inputs text information to make further requests on the intelligence topic and realizes multiple rounds of interaction. A user interaction information database is established, which records user information for each session. The interaction information is extracted, encoded according to the data type, and preprocessed to obtain the corresponding embedding sequence. The embedding sequence containing the interaction information and the embedding sequence of the preprocessed multimodal data in step S102 are input into the Transformer model, and then steps S103, S104, and S105 are executed until the user no longer makes further requests on the intelligence topic.

[0011] Furthermore, in step S102, the data preprocessing of the audio includes:

[0012] First, convert the audio data into a log-Mel spectrogram;

[0013] Then, a feature encoder consisting of multiple convolutional layers is used, where the stride of the convolutional layers is set to 2 to achieve audio downsampling, until the last convolutional layer outputs an embedding sequence with a specified feature dimension;

[0014] Learnable location coding information is superimposed to generate an audio embedding sequence containing location information.

[0015] Furthermore, in step S102, the data preprocessing of the text includes:

[0016] All words are organized into a word list, using the smallest granularity unit of word position. After the word list is built, the number of word lists is recorded as V, and the length of the recorded text intelligence data is recorded as N. Each word is converted into a one-hot encoded vector to obtain an N×V matrix, which is used as a training sample.

[0017] Initialize a trainable random word embedding matrix, input one-hot encoding into the trainable random word embedding matrix, and obtain word vectors;

[0018] The absolute positional encoding method is used to add positional information to each character vector to obtain the text embedding sequence. The positional encoding formula is:

[0019]

[0020] Where k is the position of the character in the text data, and the value of k ranges from 1 to k to N. k,2i p k,2i+1 These are the 2i and 2i+1 components in the absolute position encoding vector of position k, where i = 0, 1, ..., d / 2, and d is the dimension of the absolute position encoding vector.

[0021] Furthermore, in step S102, the data preprocessing of the image includes:

[0022] To unify the image size, the image is cut into multiple small squares of a fixed size.

[0023] Flatten these small squares into one-dimensional vectors and feed them into the block embedding layer of the ViT model;

[0024] Learnable location encoding information is added to generate image embedding sequences.

[0025] Furthermore, in step S103, the Transformer encoder processes the embedding sequences of each input mode, specifically including the following steps:

[0026] Step S103.1: Input the preprocessed audio embedding sequence, text embedding sequence and image embedding sequence into the self-attention mechanism respectively to establish the association between different positions of the same modality;

[0027] Step S103.2: Input the self-attention output of the audio, text or image embedded sequence into the cross-modal attention mechanism to capture the semantic association between different modalities and generate audio-guided fusion representation, text-guided fusion representation and image-guided fusion representation;

[0028] Step S103.3: The audio-guided fusion representation, the text-guided fusion representation, and the image-guided fusion representation are input into the feedforward layer for processing and then the fusion representations are spliced ​​together to obtain a fusion sequence that integrates the three modalities of information.

[0029] Step S103.4: Input the multimodal fusion representation into the Transformer decoder for decoding, and calculate the word probability of the decoded sequence through the SoftMax layer. Output the words with the highest probability in sequence to complete the information feature extraction.

[0030] Furthermore, in step S104, the topic coverage loss of the generated intelligence topic text calculated by the coverage function of the adjustable parameter is:

[0031]

[0032] Where, x t p is a specific vector for the target topic t, representing the specific topic or category that the model hopes to generate. t Let x be the conditional probability distribution of the target topic, β be the parameter of the coverage loss function, and x be the value of x. i For the decoded sequence (x1, x2, ..., x...) n The i-th vector in ).

[0033] Furthermore, in step S105, the user's requirements include the text's temporal and geographical scope, text subject, topic depth and breadth, semantic level and abstraction, domain specificity, text style and intent.

[0034] Compared with existing intelligence topic extraction methods, the present invention has the following advantages:

[0035] 1. The method of this invention uses audio preprocessing based on word embedding model, text preprocessing based on one-hot coding positional coding, and image preprocessing based on ViT model. When processing multimodal data, the characteristics of multimodal data are comprehensively considered. Preprocessing and alignment are required to ensure that they are comparable in feature space, thereby more comprehensively grasping the content and features of the topic.

[0036] 2. This invention utilizes a multimodal Transformer model for cross-fusion of multimodal intelligence data. Data from various modalities, after feature concatenation, are input into the Transformer model. A cross-modal mechanism generates a fused representation guided by different modal data. Then, through a specific cross-fusion strategy, the features of each modality are fused to generate a more comprehensive and integrated multimodal representation. The cross-fusion layer facilitates the complementarity and synergy of information between different modalities, improving the accuracy and comprehensiveness of topic extraction.

[0037] 3. This invention uses an adjustable coverage loss function to calculate the coverage loss. By using an adjustable coverage loss β as a parameter of the coverage loss function, the weight of the coverage loss can be dynamically adjusted according to the content and context of the current session, ensuring the accuracy of the intelligence topics generated by the model. At the same time, the coverage loss is used to train the model to maximize model performance and the comprehensiveness of the generated results.

[0038] 4. This invention utilizes a multi-round, user-interactive approach to intelligence topic extraction. Compared to traditional static analysis methods, the interactive session makes the topic extraction process interactive and ensures the accuracy of intelligence topics based on the user's input information in each round. Furthermore, the user interaction information is reconstructed into a database, allowing the model to further learn the differences in user interaction information from each round, dynamically adjusting the topic extraction strategy and model to better understand user needs and contextual information. Attached Figure Description

[0039] Figure 1 This is a simplified flowchart of the method of the present invention;

[0040] Figure 2 This is a flowchart of the method of the present invention;

[0041] Figure 3 This is a flowchart of audio data preprocessing provided in a specific embodiment of the present invention;

[0042] Figure 4 A flowchart of text data preprocessing provided in a specific embodiment of the present invention;

[0043] Figure 5 This is a flowchart of image data preprocessing provided in a specific embodiment of the present invention;

[0044] Figure 6 This is a block diagram of a multimodal Transformer model provided in a specific embodiment of the present invention. Detailed Implementation

[0045] To make the features and advantages of the present invention more apparent and understandable, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] like Figure 1 As shown, the method of the present invention includes:

[0047] Step S101: Input multimodal data, which includes audio, text, and images;

[0048] Step S102: Preprocess the multimodal data.

[0049] That is, based on the input multimodal data, the respective encoding schemes are selected for data preprocessing to obtain the embedding sequences of each modality. The audio is first converted into a log-Mel spectrogram, and after the features are extracted by a multi-layer convolutional encoder, positional encoding information is added. The text is first encoded using one-hot encoding for word vectors, and then positional encoding information is added. The image is processed using the ViT model, and positional encoding information of the image is added.

[0050] Step S103: Input the preprocessed multimodal data into the Transformer model for cross-fusion and decoding to obtain the intelligence topic text.

[0051] The embedding sequences of each modality are input into the Transformer encoder, which includes a cross-fusion layer, to generate a multimodal fusion sequence. The decoded sequence is output by the Transformer decoder. SoftMax is used to calculate the probability of words containing intelligence topic information. The word with the highest probability is selected and the next output character is generated iteratively until the complete intelligence topic text is generated.

[0052] Step S104: Calculate the topic coverage loss of the generated intelligence topic text based on the coverage function of the adjustable parameters, update the parameters of the Transformer model, and enable the Transformer model to output customized intelligence topic text.

[0053] Step S105: The user inputs text information to make further requests on the intelligence topic and realizes multiple rounds of interaction. A user interaction information database is established, which records user information for each session. The interaction information is extracted, encoded according to the data type, and preprocessed to obtain the corresponding embedding sequence. The embedding sequence containing the interaction information and the embedding sequence of the preprocessed multimodal data in step S102 are input into the Transformer model, and then steps S103, S104, and S105 are executed until the user no longer makes further requests on the intelligence topic.

[0054] These user requests specifically include user preference information for interaction, including the text's temporal and geographical scope, text subject, topic depth and breadth, semantic level and abstraction, domain specificity, text style, and intent.

[0055] The following describes specific embodiments and... Figure 2 The present invention will be explained in detail.

[0056] In step S102, the three types of multimodal data—audio, text, and image—are preprocessed.

[0057] In one specific embodiment of the present invention, for audio data, the audio data is first converted into a log-Melogram; then, a feature encoder consisting of multiple convolutional layers is used, wherein the stride of the convolutional layers is set to 2 to achieve audio downsampling, until the last convolutional layer outputs an embedding sequence with a specified feature dimension; learnable positional encoding information is superimposed to generate an audio embedding sequence containing positional information. Figure 3 As shown.

[0058] In a specific embodiment of the present invention, text data undergoes text intelligence data preprocessing based on one-hot encoding and positional encoding. For example... Figure 4 As shown. First, all characters are organized into a character table, with V characters and N text information data of length. Each character is converted into a one-hot encoded vector, resulting in an N×V character embedding matrix. Initialized with a random character embedding matrix, the character embedding matrix is ​​learned. After inputting the one-hot encoded vector into the character embedding matrix, the character vector X' is obtained. CE The positional information of each character is added to the character vector, specifically using an absolute positional encoding method composed of sin and cos functions. Formula (1) is the absolute positional information encoding formula, where k represents the position of the character in the sentence, with a value range of (0, N), d represents the dimension of the position vector, and p k,2i p k,2i+1 The position vector X is obtained by representing the 2i-th and 2i+1-th components of the encoded vector at position k. PE .

[0059]

[0060] The word vectors and position vectors obtained after preprocessing the text intelligence data are summed to obtain the text embedding sequence, as shown in formula (2):

[0061] X E =X' CE +X PE (2)

[0062] In one specific embodiment of the present invention, for image data, data preprocessing based on the ViT model is performed. First, the image size is standardized, and the image is cut into small squares of fixed size. All small squares are flattened into one-dimensional vectors and fed into the block embedding layer of the ViT model. A learnable positional encoding method is used to generate positional vectors that provide positional information for each small square. For example... Figure 5 As shown.

[0063] The block embedding sequence X obtained after preprocessing image intelligence data i and position vector X i，PE Feature stitching is performed to obtain image sequence X. i，M Formula (3) is as follows:

[0064] X i,M =Add(X i ,X i,PE (3)

[0065] Here, Add indicates feature addition processing.

[0066] In a specific embodiment of the present invention, step S103 specifically includes:

[0067] Step 1: As Figure 6 As shown, the preprocessed audio embedding sequence, text embedding sequence and image embedding sequence are first fed into the self-attention mechanism to establish the association between different positions of the same modality.

[0068] Step 2: Map the feature sequences of audio, text, and image modalities as the guiding modality M0 to the query feature Q, respectively. Map the feature sequences of the other two modalities as the target modalities M1 and M2 to the fused key-value features K and V. Then, use the cross-modal attention mechanism implemented by SoftMax. Capture consistent semantics where the correlation between modalities is higher than a set value, and generate audio-guided sequence fusion representations, text-guided sequence fusion representations, and image-guided sequence fusion representations; after fusion, the audio-guided sequence fusion representation, text-guided sequence fusion representation, and image-guided sequence fusion representation are concatenated to form a multimodal fusion representation.

[0069] Step 3: After processing the fusion representations of several sequences into the feedforward layer, the fusion representations are concatenated to obtain a fused sequence that incorporates information from three modalities. The fused sequence, having undergone feedforward processing, possesses a higher level of abstraction and representation capabilities.

[0070] Step 4: The multimodal fusion representation input is decoded by a Transformer decoder. The Transformer decoder effectively encodes the feature representation of the input sequence, which is used to identify and extract topic information. The decoded sequence is then passed through a SoftMax layer to calculate word probabilities, and the words with the highest probabilities are output sequentially, thus completing the information feature extraction.

[0071] In a specific embodiment of the present invention, step S104 includes:

[0072] Step 1: Design an adjustable coverage loss function. Considering that the level of attention to a topic may dynamically change during conversational interaction, an adjustable coverage loss function is designed. An adjustable coverage loss β is used as the parameter of the coverage loss function, and the parameter is dynamically adjusted according to the content and context of the current conversation to ensure that the model can effectively capture key topics.

[0073] Let the decoded vector be (x1, x2, ..., xn). n After applying Softmax, the probability is obtained as follows:

[0074]

[0075] Step 2: Calculate the coverage loss for this operation. The loss function used is designed as follows:

[0076]

[0077] Where t is the target class or target topic, x t p is a specific vector for the target topic t, representing the specific topic or category that the model hopes to generate. t Let x be the conditional probability distribution of the target topic, β be the parameter of the coverage loss function, and x be the value of x. i For the decoded sequence (x1, x2, ..., x...) n The i-th vector in ).

[0078] Step 3: Based on this coverage loss, train a multimodal Transformer model and adjust the attention weights to better cover the topics. Simultaneously, during training, to improve convergence speed, a teacher forcing training method is employed. That is, during the decoding process, instead of using the previous step's input as the next step's input, the correct word vectors are used as input, thereby accelerating convergence.

[0079] In a specific embodiment of the present invention, step S105 includes:

[0080] Step 1: After outputting a round of topic information, first determine whether the user has initiated a conversational interaction to further clarify the topic;

[0081] Step 2: Extract user interaction information and save it to the user interaction information database. This database is used to record user information that has been entered multiple times, so as to generate intelligence topic extraction with memory function.

[0082] Step 3: The user interaction information is preprocessed again and input into the cross-fusion layer for the next round of intelligence topic extraction.

[0083] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for extracting unstructured intelligence topics from conversational interactions using a multimodal Transformer model, characterized in that, Includes the following steps: Step S101: Input multimodal data, which includes audio, text, and images; Step S102: Select the respective encoding schemes for the input multimodal data and perform data preprocessing to obtain the embedding sequence of each modality. The audio is encoded using the log-Mel spectrum embedding model; the text is first encoded using one-hot encoding for word vectors, and then positional encoding information is added; the image is processed using the ViT model, and positional encoding information of the image is added. Step S103: Input the embedding sequence of each modality into the Transformer encoder containing the cross-fusion layer to generate a multimodal fusion sequence; output the decoded sequence through the Transformer decoder, use SoftMax to calculate the probability of words containing intelligence topic information, select the word with the highest probability, and iteratively generate the next output character until the complete intelligence topic text is generated; Step S104: Calculate the topic coverage loss of the generated intelligence topic text based on the coverage function with adjustable parameters, wherein the topic coverage loss is: Where, x t p is a specific vector for the target topic t, representing the specific topic or category that the model hopes to generate. t Let x be the conditional probability distribution of the target topic, β be the parameter of the coverage loss function, and x be the value of x. i For the decoded sequence (x1, x2, ..., x...) n The i-th vector in the equation is used to update the parameters of the Transformer model, enabling the Transformer model to output customized intelligence-themed text. Step S105: The user inputs text information to make further requests on the intelligence topic and realizes multiple rounds of interaction. A user interaction information database is established, which records user information for each session. The interaction information is extracted, encoded according to the data type, and preprocessed to obtain the corresponding embedding sequence. The embedding sequence containing the interaction information and the embedding sequence of the preprocessed multimodal data in step S102 are input into the Transformer model, and then steps S103, S104, and S105 are executed until the user no longer makes further requests on the intelligence topic.

2. The method for extracting unstructured intelligence topics from conversational interactions using a multimodal Transformer model according to claim 1, characterized in that, In step S102, the data preprocessing of the audio includes: First, convert the audio data into a log-Mel spectrogram; Then, a feature encoder consisting of multiple convolutional layers is used, where the stride of the convolutional layers is set to 2 to achieve audio downsampling, until the last convolutional layer outputs an embedding sequence with a specified feature dimension; Learnable location coding information is superimposed to generate an audio embedding sequence containing location information.

3. The method for extracting unstructured intelligence topics from conversational interactions using a multimodal Transformer model according to claim 1, characterized in that, In step S102, data preprocessing of the text includes: All words are organized into a word list, using the smallest granularity unit of word position. After the word list is built, the number of word lists is recorded as V, and the length of the recorded text intelligence data is recorded as N. Each word is converted into a one-hot encoded vector to obtain an N×V matrix, which is used as a training sample. Initialize a trainable random word embedding matrix, input one-hot encoding into the trainable random word embedding matrix, and obtain word vectors; The absolute positional encoding method is used to add positional information to each character vector to obtain the text embedding sequence. The positional encoding formula is: Where k is the position of the character in the text data, and the value of k ranges from 1 to k to N. k,2i p k,2i+1 These are the 2i and 2i+1 components in the absolute position encoding vector of position k, where i = 0, 1, ..., d / 2, and d is the dimension of the absolute position encoding vector.

4. The method for extracting unstructured intelligence topics from conversational interactions using a multimodal Transformer model according to claim 1, characterized in that, In step S102, the data preprocessing of the image includes: To unify the image size, the image is cut into multiple small squares of a fixed size. Flatten these small squares into one-dimensional vectors and feed them into the block embedding layer of the ViT model; Learnable location encoding information is added to generate image embedding sequences.

5. A method for extracting unstructured intelligence topics from conversational interactions using a multimodal Transformer model according to any one of claims 1 to 4, characterized in that, In step S103, the Transformer encoder processes the embedding sequences of each input mode, specifically including the following steps: Step S103.1: Input the preprocessed audio embedding sequence, text embedding sequence and image embedding sequence into the self-attention mechanism respectively to establish the association between different positions of the same modality; Step S103.2: Input the self-attention output of the audio, text or image embedded sequence into the cross-modal attention mechanism to capture the semantic association between different modalities and generate audio-guided fusion representation, text-guided fusion representation and image-guided fusion representation; Step S103.3: The audio-guided fusion representation, the text-guided fusion representation, and the image-guided fusion representation are input into the feedforward layer for processing and then the fusion representations are spliced ​​together to obtain a fusion sequence that integrates the three modalities of information. Step S103.4: Input the multimodal fusion representation into the Transformer decoder for decoding, and calculate the word probability of the decoded sequence through the SoftMax layer. Output the words with the highest probability in sequence to complete the information feature extraction.

6. The method for extracting unstructured intelligence topics from conversational interactions using a multimodal Transformer model according to claim 1, characterized in that, In step S105, the user's requirements include the text's temporal and geographical scope, text subject, topic depth and breadth, semantic level and abstraction, domain specificity, text style and intent.

Images