Text summary generation method, electronic device and medium based on multimodal information
Through feature mapping and attention mechanisms, the multimodal information is serialized and interacted with, and the problems of alignment and feature processing in multimodal information processing are solved, and efficient and accurate text summary generation is achieved.
Patent Information
- Application Number
- CN202210104367.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-28
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2042-01-28
AI Technical Summary
The prior art has problems in the multimodal information processing that cannot be effectively aligned and treats different features equally, resulting in insufficient efficiency and accuracy of text summary generation of multimodal inputs.
The serialized features of multimodal information are obtained through feature mapping, and the attention mechanism and the cross-attention mechanism are used for feature enhancement and fusion. The text summary is generated in combination with the feedforward neural network, and the self-attention and cross-attention mechanism are used for information interaction and Feature splicing, masking mask matrix is serialized and generated.
提高了多模态数据的表达效率和准确性,能够更好地捕捉融合特征,生成高效和准确的文本摘要。
Smart Images
Figure CN114491006B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a text summary generation technology, and in particular to a text summary generation method, electronic equipment and medium with reference to multimodal information. Background Art
[0002] With the abundance of multimedia information on the Internet, people's demand for short and effective information is increasing. Many cross-media platforms rely on text related to video or audio, such as news headlines. Therefore, the development of abstract sentence summarization technology with multimodal input can avoid the waste of resources, reduce repetitive work, and promote social development.
[0003] What the present invention needs to solve is to automatically generate text summaries based on multi-modal input information. Existing summary research mainly focuses on single-modality (either text summaries or video summaries). Some researchers have summarized multi-modal inputs into text or multi-modal outputs. Although some progress has been made, there are still two major problems:
[0004] (1) Multimodal information cannot be effectively aligned;
[0005] (2) All features are treated equally, and the divide-and-conquer approach is not adopted to treat different parts of the features. Summary of the invention
[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a text summary generation method, electronic device and medium with reference to multimodal information, which are highly efficient and accurate.
[0007] The purpose of the present invention can be achieved by the following technical solutions:
[0008] A method for generating a text summary with reference to multimodal information, comprising an encoding step and a decoding step;
[0009] The encoding step comprises:
[0010] Acquiring serialized features of multimodal information through feature mapping, wherein the multimodal information includes text information, audio information, and image information;
[0011] The serialized features of multimodal information are enhanced through the attention mechanism to obtain enhanced features of multimodal information;
[0012] Classify the enhanced features of multimodal information to obtain several types of enhanced feature sets;
[0013] Input several types of enhanced feature sets into several feedforward neural networks one by one, and obtain several groups of fusion features accordingly;
[0014] Concatenate several groups of fused features into a feature fusion vector;
[0015] The decoding step comprises:
[0016] Through the cross-attention mechanism, the feature fusion vector is used as the hidden state input, the enhanced features of the text information are fused with the feature fusion vector to generate a text summary.
[0017] Furthermore, the serialization feature of the text information is a word embedding feature;
[0018] The serialized features of the audio information are MFCC features;
[0019] The process of acquiring the serialized features of the image information includes:
[0020] The image is segmented and sliced, and the feature dimension of the segmented image is the same as the feature dimension of the word embedding feature, and the data length of the segmented image is the same as the data length of the MFCC feature.
[0021] Furthermore, the enhanced features of the multimodal information are classified by a classifier, and the expression of the classifier is:
[0022]
[0023] Where n is the number of feedforward neural networks, R is the input information, and R i is the i-th enhanced feature set after classification, For text information, L T is the length of the text message, D T is the characteristic dimension of text information, is the audio information, L I is the length of the audio information, D I is the characteristic dimension of audio information, is the image information, L A is the length of the image information, D A is the characteristic dimension of image information.
[0024] Furthermore, the expression of the fusion feature is:
[0025]
[0026] Among them, FFN i is the i-th feed-forward neural network.
[0027] Furthermore, the process of enhancing the serialization features of multimodal information through the attention mechanism includes:
[0028] The text information is enhanced through the self-attention mechanism, and the audio and image information are enhanced through the cross-attention mechanism using the text information.
[0029] Furthermore, the self-attention mechanism is implemented through a Transformer model.
[0030] Furthermore, the cross-attention mechanism is implemented through a Transformer model.
[0031] Furthermore, the mask matrix in the attention mechanism is masked.
[0032] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor calls the program instructions to execute the text summary generation method.
[0033] A computer-readable storage medium includes a computer program, wherein the computer program can be executed by a processor to implement the text summary generation method.
[0034] Compared with the prior art, the present invention has the following beneficial effects:
[0035] (1) After feature mapping, the ternary multimodal data groups of text, image and audio in the present invention are all converted into serialized data structures, and the feature expression of text information is enhanced by using the self-attention mechanism. The information that is beneficial to text expression is learned from the image information and the audio information by using the cross-attention mechanism, and the enhanced features of the ternary multimodal data are obtained. After the enhanced features are classified, they are input into several feedforward neural networks one by one, and several groups of fusion features are obtained accordingly. The several groups of fusion features are spliced into feature fusion vectors, and the feature fusion vectors are used as latent vectors. The enhanced features of the text information are used as inputs, and the cross-attention mechanism is used for interaction to generate the final text summary prediction result. The present invention uses cross-modal interaction to fuse information, and then classifies the enhanced features. The corresponding feedforward neural network is used to fuse the enhanced features of different modalities belonging to the same category, so that the different information of the fusion features can be better captured, making the final expression more efficient and accurate. The feature fusion vector is used to react to the text summary generation process, which can better fuse multimodal information;
[0036] (2) In order to enable the entire text summary to follow the serialized generation strategy during the generation process, the present invention adopts a masking form to perform the operation, that is, to mask the mask matrices in the cross-attention mechanism and the self-attention mechanism. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 Schematic diagram of the multimodal hierarchical selection framework structure based on the Transformer model. DETAILED DESCRIPTION
[0038] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.
[0039] Example 1
[0040] A method for generating a text summary with reference to multimodal information, comprising an encoding step and a decoding step;
[0041] The encoding steps include:
[0042] Obtain serialized features of multimodal information through feature mapping, where the multimodal information includes text information, audio information, and image information;
[0043] The serialized features of multimodal information are enhanced through the attention mechanism to obtain enhanced features of multimodal information;
[0044] Classify the enhanced features of multimodal information to obtain several types of enhanced feature sets;
[0045] Input several types of enhanced feature sets into several feedforward neural networks one by one, and obtain several groups of fusion features accordingly;
[0046] Concatenate several groups of fused features into a feature fusion vector;
[0047] The decoding steps include:
[0048] Through the cross-attention mechanism, the feature fusion vector is used as the hidden state input, the enhanced features of the text information are fused with the feature fusion vector to generate a text summary.
[0049] Feature representation (feature mapping) mainly maps the features of data of different modalities to make them have a serialized data structure. The serialized features of text information are word embedding features, and the serialized features of audio information are MFCC features. Image information needs to be segmented because the image itself does not have serialized data characteristics. The process of obtaining the serialized features of image information includes:
[0050] The original image is a three-dimensional matrix of length × width × channels (H×W×C). The image is then cut into pieces of size P×P (the value P is set in advance, usually 8 pixels). The dimension of the image after segmentation can be expressed as P^2×C (corresponding to the feature dimension of the word embedding feature), and the length of the image is HW / P^2 (corresponding to the data length of the MFCC feature).
[0051] The enhanced features of multimodal information are classified by a classifier, and the expression of the classifier is:
[0052]
[0053] Where n is the number of feedforward neural networks, R is the input information, and R i is the i-th enhanced feature set after classification, For text information, L T is the length of the text message, D T is the characteristic dimension of text information, is the audio information, L I is the length of the audio information, D I is the characteristic dimension of audio information, is the image information, L A is the length of the image information, D A is the characteristic dimension of image information.
[0054] The expression of fusion features is:
[0055]
[0056] Among them, FFN i is the i-th feed-forward neural network.
[0057] Since the text summary generation method proposed in this embodiment is mainly aimed at text summarization, the text data feature expression process only needs to focus on itself without paying attention to the features of other modalities. The process of enhancing the serialization features of multimodal information through the attention mechanism includes:
[0058] The text information is enhanced through the self-attention mechanism, and the audio and image information are enhanced through the cross-attention mechanism using the text information.
[0059] like Figure 1 As shown, the text summary generation method with reference to multimodal information proposed in this embodiment is implemented by a multimodal hierarchical selection framework based on the Transformer model, which includes a cross-modal encoder and a sequence mask decoder;
[0060] The cross-modal encoder includes a feature representation module, a low-dimensional cross-modal interaction module, and a high-dimensional selective routing module;
[0061] The feature representation module mainly performs feature mapping on data of different modalities to make them have a serialized data structure (because the Transformer model is parallel processing, the data must have serialization characteristics) and performs feature mapping on the input information (text-image-audio triplet) respectively.
[0062] The low-dimensional cross-modal interaction module enhances the serialized features of each modality to make it more suitable for this task. Simply put, it interacts with different data in low dimensions (relative to other operations, it is low-dimensional because it is performed immediately after feature representation). The existing multimodal triplets are all serialized data after feature mapping, that is, their dimensions are all L×D, where L represents the data length and D represents the dimension of feature expression. The text is then input into the self-attention module (a part of the Transformer model) for feature expression enhancement (because the text summary generation method proposed in this embodiment is aimed at text summarization, so the text data feature expression process only needs to pay attention to itself without paying attention to the features of other modalities); images and audio are used as supplementary information of text information, and are aligned to the text through the cross-attention mechanism (a mechanism between the encoder Encoder and the decoder Decoder in the Transformer model).
[0063] The high-dimensional selective routing module adopts a divide-and-conquer strategy to treat different parts of the features (such as dividing the data between different modalities into several parts, taking three parts as an example, namely, strong semantic correlation, weak semantic correlation and semantic irrelevant, and inputting the different modal data belonging to these three parts into their corresponding downstream sub-modules for processing). Therefore, after feature enhancement and supplementation of other modal information, the data of the three different modalities are feature spliced (fused) to better capture the different information of the fused features.
[0064] The sequence mask decoder takes the enhanced features of text information as input, interacts with the fused features as hidden state input, and finally generates the required text summary.
[0065] In order to make the entire text summary follow the serialized generation strategy during the generation process, the operation is performed in the form of masking, that is, the mask matrices in the cross-attention mechanism and the self-attention mechanism are masked (masking the diagonal and upper triangular matrices).
[0066] Example 2
[0067] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor calls the program instructions to execute the text summary generation method of embodiment 1.
[0068] Example 3
[0069] A computer-readable storage medium includes a computer program, wherein the computer program can be executed by a processor to implement the text summary generation method of embodiment 1.
[0070] Embodiment 1, embodiment 2 and embodiment 3 propose a method, electronic device and medium for generating text summaries with reference to multimodal information, pay more attention to how to better integrate and express multimodal information, utilize cross-modal interaction to integrate information at a low level, make the final expression more efficient through routing selection at a high level, and utilize this fused information to react to text summary generation at the decoder end, which can better integrate multimodal information and has many application scenarios in real life, such as news tags, summary generation and automatic generation of video conference minutes.
[0071] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.
Claims
1. A method for generating a text summary with reference to multimodal information, characterized in that: It includes encoding step and decoding step; The encoding step comprises: Acquiring serialized features of multimodal information through feature mapping, wherein the multimodal information includes text information, audio information, and image information; The serialized features of multimodal information are enhanced through the attention mechanism to obtain enhanced features of multimodal information; Classify the enhanced features of multimodal information to obtain several types of enhanced feature sets; Input several types of enhanced feature sets into several feedforward neural networks one by one, and obtain several groups of fusion features accordingly; Concatenate several groups of fused features into a feature fusion vector; The decoding step comprises: Through the cross-attention mechanism, the feature fusion vector is used as the hidden state input, the enhanced features of the text information are fused with the feature fusion vector to generate a text summary.
2. The method for generating a text summary with reference to multimodal information according to claim 1, characterized in that: The serialization feature of the text information is a word embedding feature; The serialized features of the audio information are MFCC features; The process of acquiring the serialized features of the image information includes: The image is segmented and sliced, and the feature dimension of the segmented image is the same as the feature dimension of the word embedding feature, and the data length of the segmented image is the same as the data length of the MFCC feature.
3. The method for generating a text summary with reference to multimodal information according to claim 1, characterized in that: The enhanced features of the multimodal information are classified by a classifier, and the expression of the classifier is: Where n is the number of feedforward neural networks, R is the input information, and R i is the i-th enhanced feature set after classification, For text information, L T is the length of the text message, D T is the characteristic dimension of text information, is the audio information, L I is the length of the audio information, D I is the characteristic dimension of audio information, is the image information, L A is the length of the image information, D A is the characteristic dimension of image information.
4. The method for generating a text summary with reference to multimodal information according to claim 3, characterized in that: The expression of the fusion feature is: Among them, FFN i is the i-th feed-forward neural network.
5. The method for generating a text summary with reference to multimodal information according to claim 1, characterized in that: The process of enhancing the serialized features of multimodal information through the attention mechanism includes: The text information is enhanced through the self-attention mechanism, and the audio and image information are enhanced through the cross-attention mechanism using the text information.
6. The method for generating a text summary with reference to multimodal information according to claim 5, characterized in that: The self-attention mechanism is implemented through the Transformer model.
7. The method for generating a text summary with reference to multimodal information according to claim 5, characterized in that: The cross-attention mechanism is implemented through the Transformer model.
8. The method for generating a text summary with reference to multimodal information according to claim 1, characterized in that: Mask the mask matrix in the attention mechanism.
9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor calls the program instructions to execute the text summary generation method according to any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that: The method comprises a computer program, wherein the computer program can be executed by a processor to implement the text summary generation method according to any one of claims 1 to 8.