Session interaction unstructured intelligence theme extraction method of multi-mode Transform model

Abstract

The invention discloses a session interaction unstructured intelligence theme extraction method of a multi-modal Transform model. The method comprises the following steps: inputting multi-modal data; carrying out preprocessing on the multi-modal data; inputting the preprocessed multi-modal data into a Transform model, and carrying out cross fusion and decoding to obtain an intelligence theme text; calculating theme coverage loss of the generated intelligence theme text according to the coverage function of the adjustable parameters, and performing parameter updating on the Transform model, so that the Transform model can output a customized intelligence theme text; a user inputs text information to further request an intelligence theme, multiple rounds of interaction are achieved, interaction information is extracted, iterative theme extraction is carried out, and user requirements and context intelligence information are deeply understood. According to the method, the multi-modal intelligence theme can be effectively extracted, the intelligence theme extraction comprehensiveness is improved, meanwhile, the intelligence extraction accuracy is further improved through session interaction, and the user experience is optimized.

Description

Technical Field

[0001] The present invention relates to a method for extracting unstructured intelligence topics, and in particular to a method for extracting unstructured intelligence topics in conversational interaction using a multimodal Transformer model. Background Art

[0002] In today's era of information explosion, massive amounts of unstructured intelligence data such as social media content, online news, and scientific papers are pouring into our field of vision, posing huge challenges to information processing and analysis. As one of the important links in information processing, topic extraction aims to extract representative and meaningful topic information from these data to help users quickly understand and obtain the required information. However, traditional topic extraction methods are often limited to a single type of data, unable to fully utilize the correlation information between different modal data, and lack effective interaction with users, resulting in low accuracy and lack of pertinence in the extraction results.

[0003] At present, there are some shortcomings in the method of combining single-modal text generation methods and multimodal fusion mechanisms. There may be a quantitative imbalance between different modal data, which leads to the neglect or disregard of certain modal information during the fusion process, thus affecting the comprehensiveness and objectivity of topic extraction. The modal fusion strategy is insufficient, and the current multimodal fusion mechanism may have limitations. These methods may not be able to fully explore the intrinsic connections and semantic information between the various modal data, resulting in poor fusion effects and affecting the accuracy and comprehensiveness of topic extraction. Summary of the invention

[0004] The purpose of the present invention is to provide a method for extracting unstructured intelligence topics from conversational interactions of a multimodal Transformer model, thereby helping users to accurately understand and utilize key information in multimodal intelligence through multi-round conversational interactions.

[0005] To achieve the above object, the specific scheme of the present invention is as follows:

[0006] Step S101: inputting multimodal data, wherein the multimodal data includes audio, text and image;

[0007] Step S102: Select respective encoding schemes for data preprocessing according to the input multimodal data to obtain embedding sequences of each modality, wherein the audio is encoded using a logarithmic Mel spectrum embedding model; the text is first encoded using a one-hot encoding for word vector encoding, and then position encoding information is added; the image is processed using a ViT model, and position encoding information of the image is added;

[0008] Step S103: Input the embedding sequence of each modality into the Transformer encoder containing the cross fusion layer to generate a multimodal fusion sequence; output the decoded sequence through the Transformer decoder, use SoftMax to calculate the probability of words containing intelligence topic information, select the word with the highest probability, and iteratively generate the next output character until a complete intelligence topic text is generated;

[0009] Step S104: Calculate the topic coverage loss of the generated intelligence topic text according to the coverage function of the adjustable parameters, and update the parameters of the Transformer model so that the Transformer model can output customized intelligence topic text;

[0010] Step S105: The user inputs text information to make further requirements on the intelligence topic and implements multiple rounds of interaction to establish a user interaction information database, which records the user information of each session, takes out the interaction information, selects the corresponding encoding method according to the data type, and obtains the corresponding embedding sequence through the corresponding preprocessing method; the embedding sequence containing the interaction information is input into the Transformer model together with the embedding sequence of the preprocessed multimodal data in step S102, and then executes steps S103, S104, and S105 until the user no longer makes further requirements on the intelligence topic.

[0011] Furthermore, in step S102, performing data preprocessing on the audio includes:

[0012] First, convert the audio data into a logarithmic Mel spectrogram;

[0013] Then pass through a feature encoder consisting of multiple convolutional layers, where the stride of the convolutional layer is set to 2 to achieve audio downsampling, until the last convolutional layer outputs an embedded sequence of the specified feature dimension;

[0014] The superposition can learn the positional encoding information and generate an audio embedding sequence containing the positional information.

[0015] Furthermore, in step S102, data preprocessing of the text includes:

[0016] All words are organized into a word table, using the smallest granularity unit of word position. After the word table is established, the number of word tables is recorded as V, the length of text intelligence data is recorded as N, and each word is converted into a one-hot encoding vector to obtain a matrix of size N×V as a training sample;

[0017] Initialize a trainable random word embedding matrix, input the one-hot encoding into the trainable random word embedding matrix, and obtain the word vector;

[0018] Use the absolute position encoding method to add position information to each word vector to obtain the text embedding sequence. The position encoding formula is:

[0019]

[0020] Where k is the position of the word in the text data, and the value range of k is 1≤k≤N, p k,2i 、p k,2i+1 They are respectively the 2i-th and 2i+1-th components in the absolute position encoding vector of position k, i=0, 1,…, d / 2, d is the dimension of the absolute position encoding vector.

[0021] Furthermore, in step S102, performing data preprocessing on the image includes:

[0022] Unify the image size and cut the image into multiple small squares of fixed size;

[0023] Flatten these small squares into one-dimensional vector form and feed them into the block embedding layer of the ViT model;

[0024] Add learnable positional encoding information to generate image embedding sequences.

[0025] Furthermore, in step S103, the Transformer encoder processes the input embedding sequence of each modality, specifically including the following steps:

[0026] Step S103.1, respectively input the preprocessed audio embedding sequence, text embedding sequence and image embedding sequence into the self-attention mechanism to establish associations between different positions of the same modality;

[0027] Step S103.2: The self-attention output of the audio, text or image embedding sequence is input into the cross-modal attention mechanism to capture the semantic association between different modalities and generate an audio-guided fusion representation, a text-guided fusion representation and an image-guided fusion representation;

[0028] Step S103.3, input the fusion representation guided by audio, the fusion representation guided by text, and the fusion representation guided by image into the feedforward layer for processing, and then perform fusion representation splicing to obtain a fusion sequence that integrates three modal intelligences;

[0029] Step S103.4: Input the fusion representation under multimodality into the Transformer decoder for decoding, calculate the word probability of the decoded sequence through the SoftMax layer, and output the words with the highest probability in turn, thereby completing the intelligence feature extraction.

[0030] Furthermore, in step S104, the topic coverage loss of the intelligence topic text generated by the coverage function with adjustable parameters is calculated as:

[0031]

[0032] Among them, x t is a specific vector of the target topic t, representing the specific topic or category that the model wants to generate, p t is the conditional probability distribution of the target topic, β is the parameter of the coverage loss function, x i is the decoding sequence (x 1 , x 2 , …, x n ) is the i-th vector in .

[0033] Furthermore, in step S105, the user's requirements include the temporal and geographical scope of the text, the main body of the text, the depth and breadth of the subject, the semantic level and degree of abstraction, domain specificity, and the text style and intention.

[0034] Compared with the existing intelligence topic extraction method, the present invention has the following advantages:

[0035] 1. The method of the present invention uses audio preprocessing based on word embedding model, text preprocessing based on one-hot encoding position encoding, and image preprocessing based on ViT model. When processing multimodal data, the characteristics of multimodal data are comprehensively considered. Preprocessing and alignment are required to ensure that they are comparable in feature space, so as to grasp the content and characteristics of the topic more comprehensively.

[0036] 2. The present invention uses a multimodal Transformer model to cross-fuse multimodal intelligence data. The data of various modalities are input into the Transformer model after feature splicing, and a fusion representation under the guidance of different modal data is generated through a cross-modal mechanism. Then, through a specific cross-fusion strategy, the features of each modal data are fused to generate a more comprehensive and comprehensive multimodal representation. The cross-fusion layer complements and coordinates the information between different modalities, improving the accuracy and comprehensiveness of topic extraction.

[0037] 3. The present invention uses an adjustable coverage loss function to calculate coverage loss. Using the adjustable coverage loss β as a parameter of the coverage loss function, the weight of the coverage loss can be dynamically adjusted according to the content and context of the current session to ensure the accuracy of the intelligence topics generated by the model, while using the coverage loss to train the model to maximize the model performance and the comprehensiveness of the generated results.

[0038] 4. The present invention uses multiple rounds of user interactive intelligence topic extraction. Compared with the traditional static analysis method, the conversational interaction makes the topic extraction process interactive, and the intelligence topic is accurately extracted based on the interactive information input by the user in each round. Secondly, the user interaction information is rebuilt into a database, so that the model can further learn the differences in each round of user interaction information, dynamically adjust the topic extraction strategy and model, and better understand user needs and context information. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a simplified flow chart of the method of the present invention;

[0040] Figure 2 It is a flowchart of the method of the present invention;

[0041] Figure 3 A flowchart of audio data preprocessing provided by a specific embodiment of the present invention;

[0042] Figure 4 A flowchart of text data preprocessing provided by a specific embodiment of the present invention;

[0043] Figure 5 A flowchart of image data preprocessing provided by a specific embodiment of the present invention;

[0044] Figure 6 A block diagram of a multimodal Transformer model provided for a specific embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the features and advantages of the present invention more obvious and understandable, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] like Figure 1 As shown, the method of the present invention comprises:

[0047] Step S101: input multimodal data, where the input multimodal data includes audio, text, and image;

[0048] Step S102: pre-process the multimodal data.

[0049] That is, according to the input multimodal data, respective encoding schemes are selected for data preprocessing to obtain the embedding sequence of each modality. The audio is first converted into a logarithmic Mel-spectrogram, and after the multi-layer convolution encoder extracts the features, the position encoding information is added; the text is first encoded by one-hot encoding for word vector encoding, and then the position encoding information is added; the image is processed by the ViT model, and the position encoding information of the image is added.

[0050] Step S103: Input the preprocessed multimodal data into the Transformer model for cross-fusion and decoding to obtain intelligence subject text.

[0051] The embedding sequences of each modality are input into the Transformer encoder containing a cross-fusion layer to generate a multimodal fusion sequence; the decoded sequence is output through the Transformer decoder, and SoftMax is used to calculate the probability of words containing intelligence topic information, select the word with the highest probability, and iteratively generate the next output character until the complete intelligence topic text is generated.

[0052] Step S104: Calculate the topic coverage loss of the generated intelligence topic text according to the coverage function of the adjustable parameters, and update the parameters of the Transformer model so that the Transformer model can output customized intelligence topic text;

[0053] Step S105: The user inputs text information to make further requirements on the intelligence topic and implements multiple rounds of interaction to establish a user interaction information database, which records the user information of each session, takes out the interaction information, selects the corresponding encoding method according to the data type, and obtains the corresponding embedding sequence through the corresponding preprocessing method; the embedding sequence containing the interaction information is input into the Transformer model together with the embedding sequence of the preprocessed multimodal data in step S102, and then executes steps S103, S104, and S105 until the user no longer makes further requirements on the intelligence topic.

[0054] These user requirements specifically include interactive information of user preferences, including the temporal and geographical scope of the text, the main body of the text, the depth and breadth of the subject, the semantic level and degree of abstraction, domain specificity, and the text style and intent.

[0055] The following is a combination of specific embodiments and Figure 2 , the scheme of the present invention is explained in detail.

[0056] In step S102, the three multimodal data of audio, text and image are preprocessed.

[0057] In a specific embodiment of the present invention, for audio data, the audio data is first converted into a logarithmic Mel-spectrogram; then passed through a feature encoder composed of multiple convolutional layers, wherein the step size of the convolutional layer is set to 2 to achieve audio downsampling, until the last convolutional layer outputs an embedding sequence of a specified feature dimension; and the learnable position encoding information is superimposed to generate an audio embedding sequence containing position information. Figure 3 shown.

[0058] In a specific embodiment of the present invention, for text data, text intelligence data preprocessing based on one-hot encoding and position encoding is performed on the text data. Figure 4 As shown. First, organize all the words into a word table, record the number of word tables as V, record the length of text intelligence data as N, convert each word into a one-hot encoding vector, and get a word embedding matrix of size N×V. Initialize it to a random word embedding matrix, learn the word embedding matrix, and get the word vector X' after inputting the one-hot encoding into the word embedding matrix CE The position information of each word is added to the word vector, specifically using the absolute position encoding method composed of the sin function and the cos function. Formula (1) is the absolute position information encoding formula, k represents the position of the word in the sentence, and the value range is (0, N), d represents the dimension of the position vector, and p k,2i , p k,2i+1 Respectively represent the 2ith and 2i+1th components of the encoding vector at position k, and obtain the position vector X PE .

[0059]

[0060] For the word vectors and position vectors obtained after preprocessing the text intelligence data, feature addition is performed to obtain the text embedding sequence. Formula (2) is as follows:

[0061] X E =X' CE +X PE (2)

[0062] In a specific embodiment of the present invention, for image data, data preprocessing based on the ViT model is performed on the image. First, the image size is unified and the image is cut into small squares of fixed size. All small squares are flattened into a one-dimensional vector form and sent to the block embedding layer of the ViT model. A learnable position encoding method is used to generate a position vector that provides position information for each small block. Figure 5 shown.

[0063] For the block embedding sequence X obtained after image intelligence data preprocessing i and position vector X i，PE , perform feature splicing to obtain the image sequence X i，M , formula (3) is as follows:

[0064] X i,M =Add(X i ,X i,PE ) (3)

[0065] Here, Add represents feature addition processing.

[0066] In a specific embodiment of the present invention, step S103 specifically includes:

[0067] Step 1: If Figure 6 As shown, first, the preprocessed audio embedding sequence, text embedding sequence and image embedding sequence are respectively passed into the self-attention mechanism to establish the association between different positions of the same modality.

[0068] Step 2: Take the feature sequences of audio, text, and image modalities as the guiding modality M respectively 0 Mapped to query feature Q, the feature sequences of the other two modalities are used as target modality M 1 、M 2 Mapped to the fused key-value features K and V, and then implemented through the cross-modal attention mechanism of SoftMax Capture the consistent semantics when the correlation between modalities is higher than the set value, and generate the sequence fusion representation under audio guidance, the sequence fusion representation under text guidance, and the sequence fusion representation under image guidance; after fusion, the sequence fusion representation under audio guidance, the sequence fusion representation under text guidance, and the sequence fusion representation under image guidance are concatenated to form a multi-modal fusion representation.

[0069] Step 3: After the fusion representations of several sequences are input into the feedforward layer for processing, the fusion representations are concatenated to obtain a fusion sequence that integrates the three modal intelligences. The representations of various modal data processed by the feedforward layer are concatenated to obtain a fusion sequence that integrates the three modal intelligences, and after being processed by the feedforward layer, it has a higher level of abstraction and representation capabilities.

[0070] Step 4: The fusion representation input under multimodality is decoded by the Transformer decoder. The Transformer decoder effectively encodes the feature representation of the input sequence, which is used to identify and extract topic information. The decoded sequence is passed through the SoftMax layer to calculate the word probability, and the words with the highest probability are output in turn, thus completing the intelligence feature extraction.

[0071] In a specific embodiment of the present invention, step S104 includes:

[0072] Step 1: Design a coverage loss function with adjustable parameters. Considering that the attention of a topic may change dynamically during the conversation interaction, a coverage loss function with adjustable parameters is designed. Use the adjustable coverage loss β as the parameter of the coverage loss function, and dynamically adjust the parameters according to the content and context of the current conversation to ensure that the model can effectively capture the key topics.

[0073] Let the decoded vector be (x 1 , x 2 , …, x n ), and the probability is obtained after Softmax:

[0074]

[0075] Step 2: Calculate the coverage loss. The loss function used is designed as:

[0076]

[0077] Where t is the target class or target topic, x t is a specific vector of the target topic t, representing the specific topic or category that the model wants to generate, p t is the conditional probability distribution of the target topic, β is the parameter of the coverage loss function, x i is the decoding sequence (x 1 , x 2 , …, x n ) is the i-th vector in .

[0078] Step 3: Based on this coverage loss, the multimodal Transformer model is trained and the attention weight is adjusted to better cover the topic. At the same time, during the training process, in order to improve the convergence speed, the Teacher forcing training method is adopted. That is, during the training process, the input of the previous step is not used as the input of the next step in the process of decoding and obtaining the output, but the correct word vector is used as the input, thereby speeding up the convergence speed.

[0079] In a specific embodiment of the present invention, step S105 includes:

[0080] Step 1: After outputting a round of topic information, first determine whether the user initiates a conversation interaction to further clarify the topic;

[0081] Step 2: extracting user interaction information and saving it in a user interaction information database, which is used to record user information input multiple times to generate intelligence topic extraction with memory function;

[0082] Step 3: Preprocess the user interaction information again and input it into the cross-fusion layer for the next round of intelligence topic extraction.

[0083] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for extracting unstructured intelligence topics from conversational interactions based on a multimodal Transformer model, characterized in that: The following steps are involved: Step S101: inputting multimodal data, wherein the multimodal data includes audio, text and image; Step S102: Select respective encoding schemes for data preprocessing according to the input multimodal data to obtain embedding sequences of each modality, wherein the audio is encoded using a logarithmic Mel spectrum embedding model; the text is first encoded using a one-hot encoding for word vector encoding, and then position encoding information is added; the image is processed using a ViT model, and position encoding information of the image is added; Step S103: Input the embedding sequence of each modality into the Transformer encoder containing the cross fusion layer to generate a multimodal fusion sequence; output the decoded sequence through the Transformer decoder, use SoftMax to calculate the probability of words containing intelligence topic information, select the word with the highest probability, and iteratively generate the next output character until a complete intelligence topic text is generated; Step S104: Calculate the topic coverage loss of the generated intelligence topic text according to the coverage function of the adjustable parameters, and update the parameters of the Transformer model so that the Transformer model can output customized intelligence topic text; Step S105: The user inputs text information to make further requirements on the intelligence topic and implements multiple rounds of interaction to establish a user interaction information database, which records the user information of each session, takes out the interaction information, selects the corresponding encoding method according to the data type, and obtains the corresponding embedding sequence through the corresponding preprocessing method; the embedding sequence containing the interaction information is input into the Transformer model together with the embedding sequence of the preprocessed multimodal data in step S102, and then executes steps S103, S104, and S105 until the user no longer makes further requirements on the intelligence topic.

2. According to the method for extracting unstructured intelligence topics from conversational interactions using a multimodal Transformer model according to claim 1, it is characterized in that: In step S102, performing data preprocessing on the audio includes: First, convert the audio data into a logarithmic Mel spectrogram; Then pass through a feature encoder consisting of multiple convolutional layers, where the stride of the convolutional layer is set to 2 to achieve audio downsampling, until the last convolutional layer outputs an embedded sequence of the specified feature dimension; The superposition can learn the positional encoding information and generate an audio embedding sequence containing positional information.

3. According to claim 1, the method for extracting unstructured intelligence topics from conversational interactions using a multimodal Transformer model is characterized in that: In step S102, data preprocessing of the text includes: All words are organized into a word table, using the smallest granularity unit of word position. After the word table is established, the number of word tables is recorded as V, the length of text intelligence data is recorded as N, and each word is converted into a one-hot encoding vector to obtain a matrix of size N×V as a training sample; Initialize a trainable random word embedding matrix, input the one-hot encoding into the trainable random word embedding matrix, and obtain the word vector; Use the absolute position encoding method to add position information to each word vector to obtain the text embedding sequence. The position encoding formula is: Where k is the position of the word in the text data, and the value range of k is 1≤k≤N, p k,2i 、p k,2i+1 They are respectively the 2i-th and 2i+1-th components in the absolute position encoding vector of position k, i=0, 1,…, d / 2, d is the dimension of the absolute position encoding vector.

4. According to the method for extracting unstructured intelligence topics from conversational interactions using a multimodal Transformer model according to claim 1, it is characterized in that: In step S102, performing data preprocessing on the image includes: Unify the image size and cut the image into multiple small squares of fixed size; Flatten these small squares into one-dimensional vector form and feed them into the block embedding layer of the ViT model; Add learnable positional encoding information to generate image embedding sequences.

5. The method for extracting unstructured intelligence topics from conversational interactions using a multimodal Transformer model according to any one of claims 1 to 4, characterized in that: In step S103, the Transformer encoder processes the input embedding sequence of each modality, specifically including the following steps: Step S103.1, respectively input the preprocessed audio embedding sequence, text embedding sequence and image embedding sequence into the self-attention mechanism to establish associations between different positions of the same modality; Step S103.2: The self-attention output of the audio, text or image embedding sequence is input into the cross-modal attention mechanism to capture the semantic association between different modalities and generate an audio-guided fusion representation, a text-guided fusion representation and an image-guided fusion representation; Step S103.3, input the fusion representation guided by audio, the fusion representation guided by text, and the fusion representation guided by image into the feedforward layer for processing, and then perform fusion representation splicing to obtain a fusion sequence that integrates three modal intelligences; Step S103.4: Input the fusion representation under multimodality into the Transformer decoder for decoding, calculate the word probability of the decoded sequence through the SoftMax layer, and output the words with the highest probability in turn, thereby completing the intelligence feature extraction.

6. According to claim 5, the method for extracting unstructured intelligence topics from conversational interactions using a multimodal Transformer model is characterized in that: In step S104, the topic coverage loss of the intelligence topic text generated by the coverage function with adjustable parameters is calculated as: Among them, x t is a specific vector of the target topic t, representing the specific topic or category that the model wants to generate, p t is the conditional probability distribution of the target topic, β is the parameter of the coverage loss function, x i is the decoding sequence (x1, x2, ..., x n ) is the i-th vector in .

7. According to claim 6, the method for extracting unstructured intelligence topics from conversational interactions using a multimodal Transformer model is characterized in that: In step S105, the user's requirements include the text's temporal and geographical scope, the text's main body, the depth and breadth of the subject, the semantic level and degree of abstraction, domain specificity, and the text's style and intent.

Images