Abstract generation method and device based on multi-modal information, equipment and storage medium

Abstract

The invention belongs to the field of artificial intelligence, and relates to a multimodal information-based abstract generation method, which comprises the following steps of: inputting target audio data into a speech recognition model for processing to obtain a speech recognition text, and performing text recognition on target image data to obtain an image recognition text; performing confidence coefficient filtering on the speech recognition text to obtain a filtered recognition text; extracting a target text corresponding to the target image data from the filtered recognition text; and inputting the target text, the target image data and the image recognition text into an abstract generation model to generate a text abstract. The invention further provides an abstract generation device and equipment based on the multi-modal information and a storage medium. In addition, the invention also relates to a block chain technology, and the target audio data and the target image data can be stored in a block chain. The digest generation method and device can increase digest diversity and improve digest generation efficiency and accuracy.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence, finance and digital medical technology, and in particular to a summary generation method, apparatus, device and storage medium based on multimodal information. Background Art

[0002] With the rapid development of communication technology, online conference tools have become an emerging mode of communication and collaboration across enterprises and regions. Online conferences have the unique advantage of not being restricted by time and space, allowing people to hold conferences and discussions at any time and any place, greatly improving the efficiency and convenience of online communication. Online conferences also generate a large amount of conference data, and users are usually surrounded by lengthy or fragmented information. In order to help users quickly locate the core content from the complex conference information, summary technology came into being.

[0003] The existing intelligent conference summarization technology in the industry usually converts the conference audio into text through the speech recognition (ASR) system, then processes the text and uses natural language processing technology to generate a summary based on the text level. However, due to the noisy and chaotic environment of multi-person conferences and the limitations of the ASR system, problems such as inaccurate audio recognition results and limited text information will occur, which will further lead to inaccurate summaries.

[0004] On the other hand, in smart meetings, meetings usually generate image materials related to the meeting, such as presentation slides, etc. This additional information is not utilized in general meeting summary generation methods, resulting in the generated summary being not comprehensive enough. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose a summary generation method, device, equipment and storage medium based on multimodal information to solve the technical problem that the summaries generated in existing meetings are of low accuracy and insufficient comprehensiveness.

[0006] In order to solve the above technical problems, the embodiment of the present application provides a summary generation method based on multimodal information, which adopts the following technical solution:

[0007] Acquire target audio data and corresponding target image data;

[0008] Input the target audio data into a trained speech recognition model for processing to obtain speech recognition text, and perform text recognition on the target image data to obtain image recognition text;

[0009] Performing confidence filtering on the speech recognition text to obtain filtered recognition text;

[0010] Extracting target text corresponding to the target image data from the filtered recognition text according to the image recognition text;

[0011] The target text, the target image data and the image recognition text are input into a trained summary generation model to generate a text summary.

[0012] In order to solve the above technical problems, the embodiment of the present application further provides a summary generation device based on multimodal information, which adopts the following technical solution:

[0013] An acquisition module, used for acquiring target audio data and target image data;

[0014] A text recognition module is used to input the target audio data into a trained speech recognition model for processing to obtain speech recognition text, and to perform text recognition on the target image data to obtain image recognition text;

[0015] A filtering module, used for performing confidence filtering on the speech recognition text to obtain a filtered recognition text;

[0016] A matching module, used for extracting a target text corresponding to the target image data from the filtered recognition text according to the image recognition text;

[0017] The generation module is used to input the target text, the target image data and the image recognition text into a trained summary generation model to generate a text summary.

[0018] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0019] The computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned method for generating a summary based on multimodal information when executing the computer-readable instructions.

[0020] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0021] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the summary generation method based on multimodal information as described above are implemented.

[0022] Compared with the prior art, this application has the following beneficial effects:

[0023] The present application provides a summary generation method based on multimodal information, which recognizes the acquired target audio data through a speech recognition model to obtain speech recognition text, and performs text recognition on the acquired target image data to obtain image recognition text; then, the speech recognition text is confidence filtered to remove inaccurate and unsmooth text fragments, thereby improving the accuracy of the text; according to the image recognition text, the target text corresponding to the target image data in the filtered recognition text is extracted, and the target text, target image data and image recognition text are input into a trained summary generation model to generate a text summary, which can enrich the content source of the summary, increase the diversity of the summary, make the generated summary more comprehensive, and further improve the efficiency and accuracy of summary generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0026] Figure 2 is a flowchart of an embodiment of a summary generation method based on multimodal information according to the present application;

[0027] Figure 3 yes Figure 2 A flowchart of a specific implementation of step S205;

[0028] Figure 4 is a structural schematic diagram of an embodiment of a summary generation device based on multimodal information according to the present application;

[0029] Figure 5 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0031] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0032] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0033] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0034] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0035] This application provides a summary generation method based on multimodal information, which can be applied to Figure 1 In the system architecture 100 shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0036] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0037] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0038] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0039] It should be noted that the summary generation method based on multimodal information provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the summary generation device based on multimodal information is generally set in the server / terminal device.

[0040] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0041] Continue to refer Figure 2 , shows a flowchart of an embodiment of a method for generating a summary based on multimodal information according to the present application, comprising the following steps:

[0042] Step S201, acquiring target audio data and target image data.

[0043] The target audio data and the target image data correspond to the voice signals of the conference participants in the intelligent conference for which a conference summary needs to be generated and the visual contents such as documents, slides, and charts that may be shared by the conference participants.

[0044] In this embodiment, the electronic device (eg, Figure 1 The server / terminal device shown in the figure can obtain the target audio data and the target image data through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0045] It should be emphasized that in order to further ensure the privacy and security of the target audio data and the target image data, the above target audio data and the target image data can also be stored in a node of a blockchain.

[0046] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0047] Step S202: Input the target audio data into the trained speech recognition model for processing to obtain speech recognition text, and perform text recognition on the target image data to obtain image recognition text.

[0048] In this embodiment, speech recognition is performed on the target audio data and image recognition is performed on the target image data, respectively, so as to enrich the content source of the text and obtain a more comprehensive text.

[0049] The trained speech recognition model is an end-to-end speech recognition model based on deep learning, which can improve the efficiency and accuracy of speech recognition. The text recognition of the image can adopt the existing OCR technology or the trained image text recognition model constructed by the user.

[0050] In some optional embodiments, the speech recognition model includes a recognition segmentation layer, an acoustic feature extraction layer, an acoustic unit recognition layer and a speech recognition layer.

[0051] The step of inputting the target audio data into the trained speech recognition model for processing to obtain the speech recognition text comprises:

[0052] Input the target audio data into the recognition segmentation layer, determine the speech segmentation endpoint based on the current segmentation parameter, segment the target audio data according to the speech segmentation endpoint, and generate a speech frame sequence;

[0053] Input the speech frame sequence into the acoustic feature extraction layer for feature extraction to obtain acoustic features;

[0054] The acoustic features are identified through the acoustic unit identification layer to obtain an acoustic unit sequence;

[0055] The acoustic unit sequence is processed through the speech recognition layer to obtain the speech recognition text.

[0056] In this embodiment, the target audio data may include multiple frames of sound signals, and the target audio data may be processed by framing. Specifically, the target audio data is input into the recognition segmentation layer, and the framing is performed according to the audio data to obtain speech frames, and the speech frames are windowed, and the speech frames after windowing are classified. The types of speech frames include but are not limited to unvoiced, voiced, noise and silence, etc., wherein unvoiced and voiced are valid speech frames, and valid speech frames are the parts that need to be recognized by speech. The recognition segmentation layer determines the probability that the speech features are divided into unvoiced, voiced, noise and silence according to the classification results of each speech frame, determines the probability that the speech frame is a valid speech frame according to the probability, and then determines the starting point and end point of the speech segment in the audio data, that is, the speech segmentation endpoint, and segments the audio data to be processed at the current moment according to the speech segmentation endpoint and the current segmentation parameter to generate a speech frame sequence to be recognized.

[0057] The current segmentation parameter is determined according to the speech rate of the current audio data, and the speech rate is the ratio of the number of words contained in the current audio data to the duration of the current audio data.

[0058] The acoustic features of each speech frame are extracted through the acoustic feature extraction layer. Commonly used acoustic feature representations include Fbank (FilterBank) feature vectors and MFCC (Mel-frequency cepstral coefficients) feature vectors.

[0059] Speech is a typical time series signal. In this embodiment, the acoustic feature extraction layer can use an LSTM network, specifically, a bidirectional Bi-LSTM layer. For the input audio data, sequential and reverse recurrent neural networks are used to obtain two independent hidden layer representations, and then the two hidden layer representations are calculated (concatenated or added) to obtain a final hidden layer representation, i.e., acoustic features, and the acoustic features are output for subsequent calculations. The acoustic features represented by this hidden layer contain speech information from both the previous moment and the next moment.

[0060] In this embodiment, the acoustic unit recognition layer is used to convert the characteristic speech frame sequence output by the recognition segmentation layer into acoustic units (such as phonemes or letters), that is, to obtain an acoustic unit sequence, which is a sequence composed of at least one acoustic unit. Among them, the phoneme is the smallest speech unit, such as "hello", which consists of two syllables and corresponding two tones, and can be decomposed into 7 phonemes "n, i, 3, h, a, o, 3".

[0061] In this embodiment, the speech recognition layer is used to convert the acoustic units in the acoustic unit sequence into natural language text. The acoustic unit sequence is processed by the speech recognition layer to obtain the candidate probability of at least one candidate recognition text, and the candidate recognition text corresponding to the maximum candidate probability is determined as the speech recognition text determined by the acoustic unit sequence. The speech recognition layer can be implemented using a Transformer decoding network.

[0062] The recognition segmentation layer can dynamically modify the speech segmentation endpoints according to the user's speaking speed, thereby improving the accuracy of speech segmentation endpoint detection. The subsequent acoustic feature extraction layer, acoustic unit recognition layer and speech recognition layer are processed in sequence to improve the efficiency and accuracy of speech recognition.

[0063] In some optional implementations, the image text recognition model includes a text detection layer, an image feature extraction layer and a feature recognition layer. The text detection layer mainly uses a text detection algorithm to locate the text area and obtain a text area layout feature map containing text position information; the image feature extraction layer segments the text area layout feature map and extracts character features to obtain single character features; the feature recognition layer performs text recognition on the single character features to obtain image recognized text.

[0064] Step S203, confidence filtering is performed on the speech recognition text to obtain filtered recognition text.

[0065] In this embodiment, the transcribed filtered recognition text is scored by the trained language model, that is, the probability of the sentence. After the transcribed filtered recognition text is scored, an appropriate confidence threshold is set, which can be adjusted according to the performance of the speech recognition system, the summary quality requirements and the actual application scenario. The text with a score less than the threshold is filtered, and only the high-quality text with a score greater than or equal to the threshold is retained.

[0066] It should be understood that the trained language model is obtained by fine-tuning the pre-trained language model with relevant text data for specific field meeting scenarios. For example, for meetings in the financial field, the pre-trained language model can be fine-tuned with text data in the financial field, and a smaller learning rate can be set to allow the language model to learn the text knowledge in the field, so that the language model performs better in the financial field; the same is true for the medical field.

[0067] In some optional implementations, the step of performing confidence filtering on the speech recognition text to obtain the filtered recognition text includes:

[0068] Inputting the speech recognition text into a trained language model, wherein the language model includes an embedding layer, a plurality of encoding layers, and an output layer;

[0069] Vector encoding is performed on the speech recognition text through the embedding layer to obtain a text encoding vector;

[0070] Attention fusion is performed on the text encoding vector through multiple encoding layers to obtain text fusion features;

[0071] The output layer classifies and predicts the text fusion features and outputs the probability distribution of each word in the speech recognition text;

[0072] Calculate the sentence probability of each sentence in the speech recognition text based on the probability distribution;

[0073] All sentences whose sentence probabilities are greater than or equal to the preset confidence threshold are output to obtain the final filtered recognition text.

[0074] In this embodiment, the embedding layer is used to perform position embedding, word embedding and character embedding on the speech recognition text, wherein position embedding provides information about the position of words in the sentence, which helps to understand the grammatical structure and contextual relationship of the sentence; word embedding captures the semantic information of the vocabulary, so that the model can understand the meaning of the vocabulary in the context; character embedding captures character-level information, which helps the model process unknown vocabulary or spelling variants. By incorporating character-level information into the model, the model's ability to process complex data is improved.

[0075] Multiple encoding layers are stacked, for example, 12 layers. Each encoding layer includes a multi-head attention sublayer and a feedforward connection layer. The output of each encoding layer is used as the input of the next layer. Text information of different layers is transmitted between encoding layers to more comprehensively understand the input text. Specifically, the process of multi-scale attention calculation in each multi-head attention sublayer is as follows:

[0076]

[0077] Among them, Q i =H i-1 W q ; K i =H i-1 W k ; V i =H i-1 W v ;Q i represents the query vector of the i-th head, K i represents the key vector of the i-th head, V i H represents the value vector of the i-th head; i represents the mask-self-attention output of the i-th head; d k Indicates the dimension of the key vector.

[0078] Merging the self-attention, the calculation formula for multi-head attention is as follows:

[0079] MultiHead=Concat(head1,head2,…,head A )W0;

[0080] Among them, Concat represents the matrix concatenation function; W0 represents the parameter matrix when compressing each self-attention.

[0081] The feedforward connection layer is used to connect and enhance the text fusion features to prevent insufficient fitting. Residual connection and layer normalization are performed after each multi-head attention sub-layer and feedforward connection layer to help gradient flow and reduce degradation problems during training.

[0082] In this embodiment, the output layer includes a fully connected layer and a softmax layer. After the text fusion features are weighted summed and nonlinearly transformed by the fully connected layer, they are converted into probability distribution by the softmax layer, and the probability distribution of each word in the sound recognition text is output.

[0083] The probability of each word is added or the probability is weighted and summed according to the weight of each word to obtain the sentence probability. The sentence probability is compared with the preset confidence threshold. When the sentence probability is greater than or equal to the preset confidence threshold, the sentence is retained. When the sentence probability is less than the preset confidence threshold, the sentence is removed, and finally the filtered recognition text is obtained.

[0084] By introducing the confidence filtering method, inaccurate and unsmooth text fragments are removed, thereby improving the accuracy and effect of the text. Secondly, by filtering low-confidence text, the impact of speech recognition errors on summary generation can be reduced, thereby improving the accuracy of the summary.

[0085] Step S204, extracting target text corresponding to the target image data from the filtered recognition text according to the image recognition text.

[0086] Specifically, a text matching algorithm is used to determine a text segment in the filtered recognition text that corresponds to the image recognition text; and the text segment is output as a target text corresponding to the target image data.

[0087] Text matching algorithms include string matching, edit distance, word vector similarity, substring search, regular expression matching, etc. The text matching algorithm is used to match the text fragments in the filtered recognition text that correspond to the image recognition text, so that the text fragments are associated with the target image data, making the text content more comprehensive and rich.

[0088] Step S205: input the target text, target image data and image recognition text into the trained summary generation model to generate a text summary.

[0089] The target text, target image data and image recognition text are input into the trained summary generation model. The summary generation model performs text prediction based on the features of the target text, target image data and image recognition text after multiple modal information is fused to obtain a text summary.

[0090] In some optional implementations, the summary generation model includes an image feature extraction layer, a text feature extraction layer, a feature fusion layer, an encoder layer, and a decoder layer. The target text, target image data, and image recognition text are processed by the image feature extraction layer, the text feature extraction layer, the feature fusion layer, the encoder layer, and the decoder layer to generate a corresponding text summary.

[0091] In this embodiment, see Figure 3 As shown, the target text, target image data and image recognition text are input into the trained summary generation model, and the steps of generating a text summary include:

[0092] Step S301, input the target text and the image recognition text into the text feature extraction layer to obtain text features.

[0093] The target text and the image recognition text may be feature extracted through the same text feature extraction layer, or different sublayers may be set in the text feature extraction layer to perform feature extraction on the target text and the image recognition text respectively.

[0094] For example, the text feature extraction layer includes a first feature extraction sublayer and a second feature extraction sublayer. The first feature extraction sublayer and the second feature extraction sublayer respectively use Transformer encoders to perform feature extraction to obtain a first text feature and a second text feature. The first text feature and the second text feature are calculated through an attention mechanism, and the first text feature and the second text feature are concatenated to obtain a final text feature.

[0095] When the text feature extraction layer is a single layer, the extracted text features are directly output.

[0096] Step S302, inputting the target image data into the image feature extraction layer to obtain image features.

[0097] Among them, the image feature extraction layer can be performed using the common VGG neural network, and can also be implemented through a pre-set neural network.

[0098] In some optional implementations, the image feature extraction layer includes an image segmentation network, a residual network resnet and an attention network attention. The target image data is segmented by the image segmentation network to obtain a segmented image; hierarchical feature extraction is performed on the segmented image by resnet to obtain image features of different scales; and image features of different scales are fused by attention through attention to obtain the final image features.

[0099] Step S303: fuse the text features and the image features through a feature fusion layer to obtain a multimodal fusion feature.

[0100] Specifically, a shared representation method is used to fuse text features and image features. For example, a fully connected layer can be used as a mapping function to map text features and image features to a shared space, respectively, to obtain a shared representation of the text t1. g '(feature representation corresponding to the target text), t2 g '(the feature representation corresponding to the image recognition text), the shared representation of the image g', using the splicing method, to obtain a higher-dimensional fusion representation S = [t1 g ', t2 g ',g'].

[0101] Step S304, performing attention calculation on the multimodal fusion features through the encoder layer to obtain semantic enhancement features.

[0102] The encoder layer adopts the Transformer encoder architecture, including an encoding embedding layer and multiple stacked encoders, each of which includes a multi-head attention sublayer and a feedforward network sublayer.

[0103] In this embodiment, the step of performing attention calculation on the multimodal fusion features through the encoder layer to obtain semantic enhancement features includes:

[0104] The multimodal fusion features are vector-embedded through the encoding embedding sublayer to obtain a fusion feature vector;

[0105] The fused feature vector is self-attentioned through the multi-head attention sub-layer to obtain the weighted fused feature;

[0106] Perform residual connection and layer normalization processing on the weighted fusion features to obtain normalized fusion features;

[0107] Input the normalized fusion features into the feed-forward network sublayer to obtain enhanced fusion features;

[0108] The enhanced fusion features are processed with residual connection and layer normalization to obtain semantically enhanced features.

[0109] Among them, the encoding embedding layer is used to convert the multimodal fusion features into vectors; the stacked multiple encoders are 6 layers, and the multi-head attention allows attention to different positions of the fusion feature vector to enhance the ability to capture features at different positions, and obtain weighted fusion features containing contextual information. Through the residual connection and layer normalization of the weighted fusion features, it helps to solve the gradient disappearance problem in deep networks. After the feedforward network sublayer uses the weight matrix for linear transformation, a nonlinear activation function is used to increase the nonlinear ability of the model, allowing the model to learn more complex feature representations and obtain enhanced fusion features. The enhanced fusion features are then processed by residual connection and layer normalization to obtain semantic enhancement features.

[0110] The encoder layer can effectively process long sequences and cross-modal information. The self-attention mechanism can capture long-distance dependencies in the sequence. The multi-head attention mechanism can learn the characteristics of the data from different angles and provide richer feature representations. Residual connections and layer normalization help to learn deeper feature representations and improve the generalization ability of the model.

[0111] Step S305, generating text from the semantic enhancement features through the decoder layer and outputting a text summary.

[0112] The decoder layer adopts the Transformer decoder architecture, including a position embedding layer, multiple stacked decoders and an output layer. Each decoder includes a masked multi-head attention sublayer, a multi-head attention sublayer, and a feedforward network sublayer. The semantically enhanced features are decoded through the decoder layer and combined with the attention mechanism to gradually generate a text summary.

[0113] In some optional implementations, the decoder layer includes a position embedding layer, a masked multi-head attention sublayer, a multi-head attention sublayer, a feedforward network sublayer and an output layer, and the semantically enhanced features pass through the masked multi-head attention sublayer, the multi-head attention sublayer, the feedforward network sublayer and the output layer in sequence to gradually generate a text summary.

[0114] In this embodiment, the step of generating text from the semantic enhancement features through the decoder layer and outputting the text summary includes:

[0115] Input the semantic enhancement features into the position embedding layer to obtain the position encoding vector;

[0116] The position encoding vector and semantic enhancement features are input into the masked multi-head attention sublayer to obtain the local attention features;

[0117] Through the multi-head attention sub-layer, the local attention features are calculated by multi-head attention to obtain the global attention features;

[0118] The global attention feature is input into the feedforward network sublayer for processing to obtain the feedforward enhanced feature;

[0119] The feedforward enhanced features are input into the output layer for prediction, and the predicted text probability distribution is output;

[0120] The beam search algorithm is used to search according to the predicted text probability distribution and output the final text summary.

[0121] Among them, the position embedding layer is used to provide the position information of words or image blocks in the semantic enhancement feature sequence, and needs to be combined with the mask (i.e., the masked multi-head attention sublayer) to ensure that at each step of the generated sequence, the decoder can only see its previous state, but cannot "see the future". The masked multi-head attention sublayer cannot capture the dependency relationship of words in the following positions because the decoder cannot know future information when decoding, that is, the words in the following positions have not been generated yet. Usually, a Mask matrix is ​​used to mark which positions cannot calculate the attention vector. When calculating the attention vector, the value of the position marked by the Mask matrix will be set to 0; the multi-head attention sublayer can capture the dependency relationship between any two words and capture contextual information when calculating the attention vector; the feedforward network sublayer is used to connect and enhance the features output by the multi-head attention sublayer to prevent insufficient fitting.

[0122] It should be noted that both the multi-head attention sub-layer and the forward feedback sub-layer contain a residual connection structure, which adds the sub-layer output to the sub-layer input and then normalizes it as the final output of the sub-layer.

[0123] The output layer includes a linear layer and a softmax layer. Since the final output of the decoder layer is a text sequence, the output text sequence is mapped into a longer sequence through the linear layer. The text at each position represents the score of the corresponding text. The score is converted into probabilities through the softmax layer, and a preset number of words with the highest probability are output to obtain the probability distribution of each text.

[0124] The beam search algorithm is used to expand the search according to the probability distribution to obtain the best text summary.

[0125] Through the decoder layer, the entire sequence can be processed in parallel, greatly speeding up summary generation; through the self-attention mechanism, the decoder is able to capture long-distance dependencies in the sequence and generate coherent and contextually relevant text; by using masking operations, the decoder cannot see future words when generating each word, which helps avoid error propagation in sequence generation and further improves the accuracy of generated summaries.

[0126] It should be noted that the speech recognition model, language model, summary generation model, etc. of the present application can be trained separately or jointly.

[0127] Taking joint training as an example, a historical meeting data set is obtained, the historical meeting data set carries a real meeting summary label and the historical meeting data set includes several historical audio data sets and corresponding historical image data sets; the historical audio data set is divided into a first training set and a first test set, the historical image data corresponding to the first training set is used as the second training set, and the historical image data corresponding to the first test set is used as the second test set; the first training set is input into a pre-built speech recognition model to obtain predicted speech recognition text, and text recognition is performed on the second training set to obtain historical image recognition text; the predicted speech recognition text is confidence filtered through the pre-trained language model, Obtain filtered predicted text; extract predicted text corresponding to the historical image data set from the filtered predicted text according to the historical image recognition text; input the predicted text, the second training set and the predicted speech recognition text into a pre-built summary generation model to generate a predicted summary; calculate the loss value of the predicted summary and the real meeting summary label based on a preset loss function, adjust the model parameters based on the loss value, continue iterative training until the model converges, and obtain the speech recognition model to be verified, the language model to be verified and the summary generation model to be verified; use the first test set and the second test set to verify the speech recognition model to be verified, the language model to be verified and the summary generation model to be verified, and output a final model that meets the conditions.

[0128] Among them, the speech recognition model can use the CTC loss function L1, the language model can use the perceptual loss function L2, and the summary generation model can use the cross entropy loss L3, and the final loss value is obtained by weighted summation.

[0129] The historical audio data in the historical audio dataset can be enhanced, including time stretching, pitch shifting, volume adjustment, noise injection, etc.

[0130] The summary generation model fuses multimodal information to generate text summaries, improve the coherence and readability of the summaries, and provide more comprehensive, accurate and rich conference summaries.

[0131] This application can generate more accurate summaries and reduce information loss due to speech recognition errors by performing confidence filtering on speech recognition text; by generating text summaries based on multimodal information of target text, target image data and image recognition text, the coherence and readability of the summaries can be improved, making them easier to understand and digest. At the same time, the multimodal information fusion method can enable the conference summary algorithm to obtain, analyze and integrate information from different angles, providing more comprehensive, accurate and rich conference summaries to meet users' needs for in-depth understanding and summary of conference content.

[0132] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0133] The summary generation method based on multimodal information provided in this application can be applied to conference scenarios in the medical and financial fields. For example, a remote consultation meeting is conducted through an intelligent conference system, audio data and image data of the remote consultation meeting are obtained, and the above steps S202 to S205 are executed to obtain a meeting summary of the remote consultation meeting.

[0134] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0135] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0136] Further references Figure 4 , as a response to the above Figure 2The present application provides an embodiment of a summary generation device based on multimodal information, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0137] like Figure 4 As shown, the summary generation device 400 based on multimodal information described in this embodiment includes: an acquisition module 401, a text recognition module 402, a filtering module 403 and a display module 404. Among them:

[0138] The acquisition module 401 is used to acquire target audio data and target image data;

[0139] The text recognition module 402 is used to input the target audio data into a trained speech recognition model for processing to obtain speech recognition text, and perform text recognition on the target image data to obtain image recognition text;

[0140] The filtering module 403 is used to perform confidence filtering on the speech recognition text to obtain a filtered recognition text;

[0141] The matching module 404 is used to extract the target text corresponding to the target image data from the filtered recognition text according to the image recognition text;

[0142] The generation module 405 is used to input the target text, the target image data and the image recognition text into the trained summary generation model to generate a text summary.

[0143] It should be emphasized that in order to further ensure the privacy and security of the target audio data and the target image data, the above target audio data and the target image data can also be stored in a node of a blockchain.

[0144] Based on the above-mentioned summary generation device based on multimodal information, by performing confidence filtering on the speech recognition text, a more accurate summary can be generated, reducing information loss caused by speech recognition errors; by generating a text summary based on the multimodal information of the target text, target image data and image recognition text, the coherence and readability of the summary are improved, making it easier to understand and digest. At the same time, the multimodal information fusion method can enable the conference summary algorithm to obtain, analyze and integrate information from different angles, providing a more comprehensive, accurate and rich conference summary, and meeting the user's in-depth understanding and summary needs of the conference content.

[0145] In some optional implementations, the speech recognition model includes a recognition segmentation layer, an acoustic feature extraction layer, an acoustic unit recognition layer, and a speech recognition layer, and the text recognition module 402 further includes:

[0146] A segmentation submodule, used for inputting the target audio data into the recognition segmentation layer, determining a speech segmentation endpoint based on a current segmentation parameter, segmenting the target audio data according to the speech segmentation endpoint, and generating a speech frame sequence;

[0147] An acoustic feature extraction submodule, used for inputting the speech frame sequence into the acoustic feature extraction layer to extract features and obtain acoustic features;

[0148] An acoustic recognition submodule, used for recognizing the acoustic features through the acoustic unit recognition layer to obtain an acoustic unit sequence;

[0149] The speech recognition submodule is used to process the acoustic unit sequence through the speech recognition layer to obtain speech recognition text.

[0150] The recognition segmentation layer can dynamically modify the speech segmentation endpoints according to the user's speaking speed, thereby improving the accuracy of speech segmentation endpoint detection. The subsequent acoustic feature extraction layer, acoustic unit recognition layer and speech recognition layer are processed in sequence to improve the efficiency and accuracy of speech recognition.

[0151] In some optional implementations, the filtering module 403 includes:

[0152] An input submodule, used for inputting the speech recognition text into a trained language model, wherein the language model includes an embedding layer, a plurality of encoding layers and an output layer;

[0153] A first embedding submodule, configured to perform vector encoding on the speech recognition text through the embedding layer to obtain a text encoding vector;

[0154] A first encoding submodule, configured to perform attention fusion on the text encoding vector through a plurality of encoding layers to obtain a text fusion feature;

[0155] A classification prediction submodule, used for performing classification prediction on the text fusion feature through the output layer, and outputting the probability distribution of each word in the speech recognition text;

[0156] A calculation submodule, used for calculating the sentence probability of each sentence in the speech recognition text based on the probability distribution;

[0157] The filtering submodule is used to output all the sentences whose sentence probabilities are greater than or equal to a preset confidence threshold to obtain a final filtered recognition text.

[0158] By introducing the confidence filtering method, inaccurate and unsmooth text fragments are removed, thereby improving the accuracy and effect of the text. Secondly, by filtering low-confidence text, the impact of speech recognition errors on summary generation can be reduced, thereby improving the accuracy of the summary.

[0159] In some optional implementations, the matching module 404 includes:

[0160] A matching submodule, used for determining a text segment in the filtered recognition text corresponding to the image recognition text by using a text matching algorithm;

[0161] The output submodule is used to output the text segment as the target text corresponding to the target image data.

[0162] The text matching algorithm is used to match the text fragments in the filtered recognition text that correspond to the image recognition text, so that the text fragments are associated with the target image data, making the text content more comprehensive and rich.

[0163] In some optional implementations, the summary generation model includes an image feature extraction layer, a text feature extraction layer, a feature fusion layer, an encoder layer, and a decoder layer; the generation module 405 includes:

[0164] A text feature extraction submodule, used for inputting the target text and the image recognition text into the text feature extraction layer to obtain text features;

[0165] An image feature extraction submodule, used for inputting the target image data into the image feature extraction layer to obtain image features;

[0166] A fusion submodule, used for fusing the text feature and the image feature through the feature fusion layer to obtain a multimodal fusion feature;

[0167] An encoder submodule, configured to perform attention calculation on the multimodal fusion features through the encoder layer to obtain semantic enhancement features;

[0168] A generation submodule is used to generate text from the semantic enhancement features through the decoder layer and output a text summary.

[0169] The summary generation model fuses multimodal information to generate text summaries, improve the coherence and readability of the summaries, and provide more comprehensive, accurate and rich conference summaries.

[0170] In some optional implementations of this embodiment, the encoder layer includes a coding embedding sublayer, a multi-head attention sublayer and a feedforward network sublayer; the encoder submodule includes:

[0171] A second embedding submodule, used for performing vector embedding on the multimodal fusion feature through the encoding embedding sublayer to obtain a fusion feature vector;

[0172] A multi-head attention submodule, used for performing self-attention calculation on the fused feature vector through the multi-head attention sublayer to obtain a weighted fused feature;

[0173] A first residual normalization submodule, used for performing residual connection and layer normalization processing on the weighted fusion features to obtain normalized fusion features;

[0174] An enhancement submodule, used for inputting the normalized fusion features into the feedforward network sublayer to obtain enhanced fusion features;

[0175] The second residual normalization submodule is used to perform residual connection and layer normalization processing on the enhanced fusion features to obtain semantic enhancement features.

[0176] The encoder layer can effectively process long sequences and cross-modal information. The self-attention mechanism can capture long-distance dependencies in the sequence. The multi-head attention mechanism can learn the features of the data from different angles and provide richer feature representations. Residual connections and layer normalization help to learn deeper feature representations and improve the generalization ability of the model.

[0177] In some optional implementations of this embodiment, the decoder layer includes a position embedding layer, a masked multi-head attention sublayer, a multi-head attention sublayer, a feedforward network sublayer and an output layer; the generation submodule includes:

[0178] A position embedding submodule, used for inputting the semantic enhancement feature into the position embedding layer to obtain a position encoding vector;

[0179] A mask submodule, used for inputting the position encoding vector and the semantic enhancement feature into the mask multi-head attention sublayer to obtain a local attention feature;

[0180] A decoding multi-head attention submodule, used for performing multi-head attention calculation on the local attention feature through the multi-head attention sublayer to obtain a global attention feature;

[0181] A feedforward enhancement submodule, used for inputting the global attention feature into the feedforward network sublayer for processing to obtain a feedforward enhancement feature;

[0182] An output submodule, used for inputting the feedforward enhanced features into the output layer for prediction, and outputting a predicted text probability distribution;

[0183] The search generation submodule is used to use a beam search algorithm to search according to the predicted text probability distribution and output a final text summary.

[0184] Through the decoder layer, the entire sequence can be processed in parallel, greatly speeding up summary generation; through the self-attention mechanism, the decoder is able to capture long-distance dependencies in the sequence and generate coherent and contextually relevant text; by using masking operations, the decoder cannot see future words when generating each word, which helps avoid error propagation in sequence generation and further improves the accuracy of generated summaries.

[0185] To solve the above technical problems, the present application also provides a computer device. Figure 5 , Figure 5 This is a basic structural block diagram of the computer device in this embodiment.

[0186] The computer device 5 includes a memory 51, a processor 52, and a network interface 53 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 5 with components 51-53, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0187] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0188] The memory 51 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 51 can be an internal storage unit of the computer device 5, such as a hard disk or memory of the computer device 5. In other embodiments, the memory 51 can also be an external storage device of the computer device 5, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 5. Of course, the memory 51 can also include both the internal storage unit of the computer device 5 and its external storage device. In this embodiment, the memory 51 is generally used to store the operating system and various application software installed on the computer device 5, such as computer-readable instructions of the summary generation method based on multimodal information, etc. In addition, the memory 51 can also be used to temporarily store various types of data that have been output or are to be output.

[0189] The processor 52 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 52 is generally used to control the overall operation of the computer device 5. In this embodiment, the processor 52 is used to run the computer-readable instructions stored in the memory 51 or process data, such as computer-readable instructions for running the summary generation method based on multimodal information.

[0190] The network interface 53 may include a wireless network interface or a wired network interface. The network interface 53 is generally used to establish a communication connection between the computer device 5 and other electronic devices.

[0191] This embodiment implements the steps of the summary generation method based on multimodal information in the above-mentioned embodiment when the processor executes computer-readable instructions stored in the memory. By performing confidence filtering on the speech recognition text, a more accurate summary can be generated, reducing information loss due to speech recognition errors; by generating a text summary based on the multimodal information of the target text, target image data and image recognition text, the coherence and readability of the summary are improved, making it easier to understand and digest. At the same time, the method of multimodal information fusion can enable the conference summary algorithm to obtain, analyze and integrate information from different angles, provide a more comprehensive, accurate and rich conference summary, and meet the user's in-depth understanding and summary needs of the conference content.

[0192] The present application also provides another implementation mode, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor so that the at least one processor executes the steps of the summary generation method based on multimodal information as described above, and by performing confidence filtering on the speech recognition text, a more accurate summary can be generated, and information loss caused by speech recognition errors can be reduced; by generating a text summary based on multimodal information of the target text, target image data and image recognition text, the coherence and readability of the summary are improved, making it easier to understand and digest, and at the same time, the method of multimodal information fusion can enable the conference summary algorithm to obtain, analyze and integrate information from different angles, and provide a more comprehensive, accurate and rich conference summary to meet the user's in-depth understanding and summary needs of the conference content.

[0193] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0194] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A summary generation method based on multimodal information, characterized in that: The steps include: Acquire target audio data and corresponding target image data; Input the target audio data into a trained speech recognition model for processing to obtain speech recognition text, and perform text recognition on the target image data to obtain image recognition text; Performing confidence filtering on the speech recognition text to obtain filtered recognition text; Extracting target text corresponding to the target image data from the filtered recognition text according to the image recognition text; The target text, the target image data and the image recognition text are input into a trained summary generation model to generate a text summary.

2. The method for generating a summary based on multimodal information according to claim 1, characterized in that: The speech recognition model includes a recognition segmentation layer, an acoustic feature extraction layer, an acoustic unit recognition layer and a speech recognition layer; the step of inputting the target audio data into the trained speech recognition model for processing to obtain the speech recognition text includes: Inputting the target audio data into the recognition segmentation layer, determining the speech segmentation endpoints based on the current segmentation parameters, segmenting the target audio data according to the speech segmentation endpoints, and generating a speech frame sequence; Inputting the speech frame sequence into the acoustic feature extraction layer to extract features to obtain acoustic features; The acoustic feature is identified by the acoustic unit identification layer to obtain an acoustic unit sequence; The acoustic unit sequence is processed by the speech recognition layer to obtain a speech recognition text.

3. The method for generating a summary based on multimodal information according to claim 1, characterized in that: The step of performing confidence filtering on the speech recognition text to obtain the filtered recognition text comprises: Inputting the speech recognition text into a trained language model, wherein the language model includes an embedding layer, a plurality of encoding layers and an output layer; Performing vector encoding on the speech recognition text through the embedding layer to obtain a text encoding vector; Attention fusion is performed on the text encoding vector through multiple encoding layers to obtain text fusion features; Classifying and predicting the text fusion features through the output layer, and outputting the probability distribution of each word in the speech recognition text; Calculate the sentence probability of each sentence in the speech recognition text based on the probability distribution; All the sentences whose sentence probabilities are greater than or equal to a preset confidence threshold are output to obtain a final filtered recognition text.

4. The method for generating a summary based on multimodal information according to claim 1, characterized in that: The step of extracting the target text corresponding to the target image data from the filtered recognition text according to the image recognition text comprises: Using a text matching algorithm to determine a text segment in the filtered recognized text that corresponds to the image recognized text; The text segment is output as the target text corresponding to the target image data.

5. The method for generating a summary based on multimodal information according to claim 1, characterized in that: The summary generation model includes an image feature extraction layer, a text feature extraction layer, a feature fusion layer, an encoder layer and a decoder layer; the step of inputting the target text, the target image data and the image recognition text into the trained summary generation model to generate a text summary includes: Inputting the target text and the image recognition text into the text feature extraction layer to obtain text features; Inputting the target image data into the image feature extraction layer to obtain image features; The text feature and the image feature are fused through the feature fusion layer to obtain a multimodal fusion feature; Performing attention calculation on the multimodal fusion features through the encoder layer to obtain semantic enhancement features; The semantic enhancement features are subjected to text generation through the decoder layer, and a text summary is output.

6. The method for generating a summary based on multimodal information according to claim 5, characterized in that: The encoder layer includes a coding embedding sublayer, a multi-head attention sublayer and a feedforward network sublayer; the step of performing attention calculation on the multimodal fusion feature through the encoder layer to obtain a semantic enhancement feature includes: Performing vector embedding on the multimodal fusion feature through the encoding embedding sublayer to obtain a fusion feature vector; Performing self-attention calculation on the fused feature vector through the multi-head attention sublayer to obtain a weighted fused feature; Performing residual connection and layer normalization processing on the weighted fusion features to obtain normalized fusion features; Inputting the normalized fusion features into the feedforward network sublayer to obtain enhanced fusion features; The enhanced fusion features are subjected to residual connection and layer normalization processing to obtain semantic enhancement features.

7. The method for generating a summary based on multimodal information according to claim 5, characterized in that: The decoder layer includes a position embedding layer, a masked multi-head attention sublayer, a multi-head attention sublayer, a feedforward network sublayer and an output layer; The step of generating text from the semantic enhancement feature through the decoder layer and outputting a text summary comprises: Inputting the semantic enhancement feature into the position embedding layer to obtain a position encoding vector; Inputting the position encoding vector and the semantic enhancement feature into the masked multi-head attention sublayer to obtain a local attention feature; Performing multi-head attention calculation on the local attention feature through the multi-head attention sublayer to obtain a global attention feature; Inputting the global attention feature into the feedforward network sublayer for processing to obtain a feedforward enhanced feature; Inputting the feedforward enhanced features into the output layer for prediction, and outputting predicted text probability distribution; A beam search algorithm is used to search according to the predicted text probability distribution, and a final text summary is output.

8. A summary generation device based on multimodal information, characterized in that: include: An acquisition module, used for acquiring target audio data and target image data; A text recognition module is used to input the target audio data into a trained speech recognition model for processing to obtain speech recognition text, and to perform text recognition on the target image data to obtain image recognition text; A filtering module, used for performing confidence filtering on the speech recognition text to obtain a filtered recognition text; A matching module, used for extracting a target text corresponding to the target image data from the filtered recognition text according to the image recognition text; The generation module is used to input the target text, the target image data and the image recognition text into a trained summary generation model to generate a text summary.

9. A computer device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the method for generating a summary based on multimodal information according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the method for generating a summary based on multimodal information are implemented as claimed in any one of claims 1 to 7.

Images