Message summarization method and device based on multiple modes, equipment and storage medium

Abstract

The embodiment of the invention relates to the technical field of artificial intelligence, and discloses a message summarization method and device based on multiple modes, equipment and a storage medium. The method comprises the following steps: acquiring a multi-modal message to be summarized, and performing message importance identification on the message to be summarized in each modal; according to the message importance identification result and the modal importance of the to-be-summarized message, generating modal feature weights of the to-be-summarized message of each modal; performing feature extraction on the to-be-summarized message of each modal, and generating a multi-modal feature fusion result according to a feature extraction result and a corresponding modal feature weight; and inputting the feature fusion result into an abstract extraction model generated by pre-training to obtain a multi-modal message abstract of the to-be-summarized message. According to the method, feature empowerment based on message importance is carried out on the multi-modal messages, feature fusion is carried out on the multi-modal messages, then message digest generation is carried out, and the message summary quality and comprehensiveness can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology and can be applied to the field of financial technology, and in particular to a message summarization method, device, equipment and storage medium based on multimodality. Background Art

[0002] With the rapid development of the Internet and mobile communication technologies, social software and instant messaging tools are becoming more and more popular in our lives. When communicating through social software and instant messaging tools, a large amount of message data will be generated.

[0003] Due to the huge amount of message data, users face many inconveniences when searching for specific information. Therefore, message summarization comes into being to summarize message content for users and improve information retrieval efficiency. However, existing message summarization mainly processes text data and ignores information in other modes. As a result, the comprehensiveness and accuracy of message summarization are limited. Summary of the invention

[0004] The present invention provides a message summarization method, device, equipment and storage medium based on multimodality to improve the quality and comprehensiveness of message summarization.

[0005] According to one aspect of the present invention, a message summarization method based on multimodality is provided, the method comprising:

[0006] Acquire multimodal messages to be summarized, and identify the importance of the messages to be summarized in each mode;

[0007] Generating modal feature weights of the messages to be summarized in each modality according to the message importance identification result and the modal importance of the messages to be summarized;

[0008] Extracting features of the messages to be summarized in each modality, and generating a multi-modal feature fusion result according to the feature extraction result and the corresponding modality feature weight;

[0009] The feature fusion result is input into a summary extraction model generated by pre-training to obtain a message summary of the multimodal message to be summarized.

[0010] According to another aspect of the present invention, a multimodal message summarization device is provided, the device comprising:

[0011] A message importance identification module, used to obtain multi-modal messages to be summarized, and identify the message importance of the messages to be summarized in each mode;

[0012] A modal feature weight generation module, used to generate modal feature weights of the messages to be summarized in each modality according to the message importance recognition result and the modal importance of the messages to be summarized;

[0013] A feature fusion result generating module is used to extract features of the messages to be summarized in each modality, and generate a multi-modal feature fusion result according to the feature extraction results and the corresponding modality feature weights;

[0014] The message summary determination module is used to input the feature fusion result into a summary extraction model generated by pre-training to obtain a message summary of the multimodal message to be summarized.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the multimodal-based message summarization method described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the multimodal-based message summarization method described in any embodiment of the present invention when executed.

[0020] According to another aspect of the present invention, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the multimodal-based message summarization method according to any embodiment of the present invention is implemented.

[0021] The technical solution of the embodiment of the present invention obtains multimodal messages to be summarized, identifies the message importance of each modal message to be summarized; generates modal feature weights of each modal message to be summarized according to the message importance identification result and the modal importance of the message to be summarized; extracts features of each modal message to be summarized, and generates a multimodal feature fusion result according to the feature extraction result and the corresponding modal feature weight; inputs the feature fusion result into a summary extraction model generated by pre-training to obtain a message summary of the multimodal message to be summarized, solves the problem that the message summary does not fuse multimodal data, and improves the quality and comprehensiveness of the message summary by weighting the features of the multimodal messages based on the message importance, fusion of the features of the multimodal messages, and then generating the message summary.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flowchart of a message summarization method based on multimodality provided according to Embodiment 1 of the present invention;

[0025] Figure 2 is a flowchart of a message summarization method based on multimodality provided according to Embodiment 2 of the present invention;

[0026] Figure 3 is a structural diagram of a multi-modal message summarization device provided according to Embodiment 3 of the present invention;

[0027] Figure 4 It is a structural schematic diagram of an electronic device for implementing the multimodal-based message summarization method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Embodiment 1

[0031] Figure 1 This is a flowchart of a multimodal message summarization method provided according to the first embodiment of the present invention. This embodiment is applicable to the situation where messages are summarized based on the multimodal content in the communication when communicating through social software and instant messaging tools. The method can be executed by a multimodal message summarization device, which can be implemented in the form of hardware and / or software. The multimodal message summarization device can be configured in an electronic device, which can be a mobile device such as a mobile phone, a tablet computer (PAD), a wearable device, or a personal computer (PC, Personal Computer). Figure 1 As shown, the method includes:

[0032] Step 110: Obtain multi-modal messages to be summarized, and identify the importance of the messages to be summarized in each mode.

[0033] The multimodal messages to be summarized may include messages in one or any of the following states: text, document, picture, video, and audio. The messages to be summarized may be messages in social software or instant messaging tools. Information summaries can be generated through message summarization to facilitate information retrieval by users. The acquisition of the messages to be summarized is only after authorization by the user, and the acquisition method is reasonable and legal.

[0034] There are many ways to identify the message importance of the messages to be summarized in each modality. For example, the message importance identification results of the messages to be summarized in each modality can be determined by combining prior knowledge with expert opinions. In another example, the message importance of the messages to be summarized in each modality can be identified by combining cross-validation with a machine learning model. In another example, the messages in the messages to be summarized in each modality can be quantified by one-dimensional or multi-dimensional importance indicators to obtain the message importance identification results.

[0035] In an optional implementation of an embodiment of the present invention, message importance identification is performed on messages to be summarized in each modality, including: identifying at least one of a message body, a message intent, a message category, and message keywords in the messages to be summarized in each modality; and determining a message importance identification result of the messages to be summarized based on at least one of the message body, the message intent, the message category, and the message keywords.

[0036] Among them, the message body can indicate the source of the message. The message body includes, but is not limited to: one or more of: superiors, suppliers, partners, peers, subordinates, and user entities. The message intent can be determined by intent recognition. For example, semantic recognition is performed on the message to be identified to obtain the message intent. In an embodiment of the present invention, the message intent can correspond to different business scenarios. The message category can indicate whether the message to be identified is a summary message, emotional communication, or a key issue that needs attention. Message keywords can be keywords in the message to be identified. Message keywords can be determined by comparing the message to be identified with a keyword library. Alternatively, the message keywords can be determined by judging the importance of the words in the message to be identified in the message.

[0037] When determining the message importance identification result of the message to be summarized according to at least one of the message body, message intent, message category, and message keywords, different subjects in the message body can be mapped to different weight values, the business corresponding to the message intent can be mapped to different weight values, each category in the message category can be mapped to different weight values, and different message keywords can be mapped to different weight values. Thus, according to the weight value corresponding to at least one of the message body, message intent, message category, and message keywords, the message importance identification result of the message to be summarized is determined. For example, the weight value of at least one of the message body, message intent, message category, and message keywords is averaged, or the maximum weight value is selected, or the median value is selected, etc., to obtain the message importance identification result of the message to be summarized. By quantifying the importance index of the messages in the messages to be summarized of each modality in one dimension or multiple dimensions, the message importance identification result can be obtained, which can make the message importance closely related to the message content, and reflect the importance of the message content, so as to give reasonable weight values ​​to the features under the modality according to the importance of the message content.

[0038] Step 120: Generate modal feature weights of the messages to be summarized in each modality according to the message importance recognition results and the modal importance of the messages to be summarized.

[0039] In an embodiment of the present invention, two factors can be considered when generating modal feature weights. One is the importance of the message itself, and the other is the importance of the modality. By generating modal feature weights based on the message importance recognition result and the modal importance, the reliability of the modal feature weights can be improved. Among them, the modal feature weight can be the product of the message importance recognition result and the modal importance. The modal importance can be that the text weight is greater than the image weight, and the image weight is greater than the voice weight.

[0040] Step 130: extract features of the messages to be summarized in each modality, and generate a multi-modal feature fusion result based on the feature extraction result and the corresponding modality feature weight.

[0041] There are many ways to extract features. Different feature extraction methods can be used for messages to be summarized in different modes to improve the accuracy of feature extraction. Feature extraction methods include but are not limited to: importance feature extraction, feature extraction through machine learning models, and audio feature extraction.

[0042] The multimodal feature fusion result can be obtained by weighted fusion of feature extraction results under different modalities and modal feature weights. Alternatively, the feature extraction results and the corresponding modal feature weights can be fused through the attention mechanism to obtain the multimodal feature fusion result. Through multimodal feature fusion, the features of different modalities can be comprehensively considered and the most important information can be retained.

[0043] Step 140: Input the feature fusion result into a summary extraction model generated by pre-training to obtain a message summary of the multimodal message to be summarized.

[0044] The summary extraction model can be generated by training based on the basic summary extraction model combined with sample data. Among them, the basic summary extraction model can be a neural network model for natural language processing. For example, the basic summary extraction model can be a model for extracting semantic features (Bidirectional Encoder Representations from Transformers, BERT). In order to ensure the reliability of the summary extraction model, clean and accurately labeled sample data is required when generating sample data. For example, the sample data may include positive samples and negative samples. The sample data may include the format, degree of refinement, length of the summary, language style of the summary, summary display format (such as whether to bold, highlight, and focus mark, etc.), to-do mark, and message body of the summary. The positive sample can be a correctly labeled summary, and the negative sample can be an incorrectly labeled summary. By combining positive samples and negative samples for training to generate a summary extraction model, the recognition accuracy of the summary extraction model can be improved.

[0045] Therefore, when the feature fusion result is input into the summary extraction model, the message summary of the multimodal message to be summarized can be obtained, which ensures the reliability, comprehensiveness and accuracy of the message summary.

[0046] The technical solution of this embodiment obtains multimodal messages to be summarized, identifies the message importance of each modal message to be summarized; generates modal feature weights of each modal message to be summarized according to the message importance identification results and the modal importance of the messages to be summarized; extracts features of each modal message to be summarized, and generates a multimodal feature fusion result according to the feature extraction results and the corresponding modal feature weights; inputs the feature fusion result into a summary extraction model generated by pre-training to obtain a message summary of the multimodal message to be summarized, solves the problem that the message summary does not fuse multimodal data, and improves the quality and comprehensiveness of the message summary by weighting the features of the multimodal messages based on the message importance, fusion of the features of the multimodal messages, and then generating a message summary.

[0047] Embodiment 2

[0048] Figure 2 1 is a flowchart of a message summarization method based on multimodality provided according to Embodiment 2 of the present invention. This embodiment is a further refinement of the above technical solution. The technical solution in this embodiment can be combined with each optional solution in one or more of the above embodiments. Figure 2 As shown, the method includes:

[0049] Step 210: Acquire multimodal original messages to be summarized, and preprocess the original messages to be summarized in each mode to obtain multimodal messages to be summarized.

[0050] Among them, by preprocessing the original message to be summarized, noise interference in the original message to be summarized can be eliminated, and the reliability of summary extraction of the multimodal message to be summarized can be improved. There are many ways of preprocessing. For example, the preprocessing method may include: filtering, irrelevant information elimination, and noise reduction.

[0051] In an optional implementation of an embodiment of the present invention, the original messages to be summarized in each mode are preprocessed to obtain multimodal messages to be summarized, including at least one of the following: segmenting the original messages to be summarized in a text state and removing information irrelevant to the message summary to obtain messages to be summarized corresponding to the text state; performing denoising processing on the original messages to be summarized in a voice state through a low-pass filter and a high-pass filter, and performing a text conversion operation to obtain messages to be summarized corresponding to the voice state; performing denoising processing on the original messages to be summarized in an image state through a mean filter, and performing a picture feature extraction operation to obtain messages to be summarized corresponding to the image state.

[0052] Among them, standard word segmentation methods can be used for word segmentation of the original information to be recognized in the text state. For example, the entire text can be divided into the smallest granularity of words. Removing irrelevant information from the original information to be recognized in the text state can include various situations. For example, removing all non-alphabetic and non-numeric characters. Another example is to剔除 unimportant words such as prepositions "de", "de", "di" according to information such as word frequency and part of speech. Another example is to剔除 words with little semantic contribution according to dependency syntactic analysis.

[0053] The original message to be summarized in the voice state can be denoised and converted to text. Among them, voice denoising can remove high-frequency noise and low-frequency noise in the voice through a low-pass filter and a high-pass filter. The operation of converting to text can use natural language processing to convert voice to text. For example, combining a statistical language model (N-gram) and feature extraction to obtain the text corresponding to the voice.

[0054] The original message to be summarized in the image state can be denoised and image feature extraction operations can be performed. In the application scenarios in the financial field, image information is generally clear. Therefore, image denoising can be performed using a mean filter. Specifically, image denoising can be achieved by calculating the average value of the pixels around a pixel and replacing the pixel with this average value. The image feature extraction operation can be to extract features using one or more methods among color feature extraction, texture feature extraction, shape feature extraction, and geometric feature extraction, etc.

[0055] By performing different preprocessing on the original information to be recognized in each modality, the messages in each modality can be reasonably processed, noise can be removed, thereby reducing the computational amount in summary extraction and improving the accuracy of summary extraction, and avoiding noise interference.

[0056] Step 220: Identify at least one of the message body, message intention, message category, and message keywords in the message to be summarized in each modality.

[0057] Step 230: Determine the message importance recognition result of the message to be summarized according to at least one of the message body, message intention, message category, and message keywords.

[0058] Step 240: Generate the modality feature weights of the messages to be summarized in each modality according to the message importance recognition result and the modality importance of the message to be summarized.

[0059] Step 250: Extract features from the messages to be summarized in each modality, and generate a multi-modal feature fusion result according to the feature extraction result and the corresponding modality feature weights.

[0060] In an optional implementation of the embodiments of the present invention, feature extraction is performed on the messages to be summarized in each modality, including at least one of the following: for the messages to be summarized corresponding to the text state, the importance of words to the text is evaluated according to the text weight algorithm, and the features corresponding to the text state are obtained; for the messages to be summarized corresponding to the voice state, voice signal features are extracted by voice signal processing to obtain features corresponding to the voice state; for the messages to be summarized corresponding to the image state, image features are extracted by a pre-trained convolutional neural network to obtain features corresponding to the image state.

[0061] Among them, the text weight algorithm can be a weight algorithm (TermFrequency-Inverse Document Frequency, TF-IDF) used for text mining and information retrieval. TF-IDF can be used to evaluate the importance of a word to a document set or a document in a corpus. TF is term frequency, which refers to the number of times a word appears in a document. The more frequently a word appears in a document, the higher its weight. IDF is inverse document frequency, which measures the uniqueness of a word. If a word appears in many documents, its IDF value will be relatively low; conversely, if a word only appears in a few documents, its IDF value will be relatively high. The IDF calculation method is to divide the number of all documents by the number of documents containing the word, and then take the logarithm. The product of TF and IDF is the importance of the word to the text.

[0062] The speech signal processing may be Mel Frequency Cepstral Coefficients (MFCC). MFCC may include pre-emphasis, framing, windowing, Fourier transform, Mel filter bank, logarithm, discrete cosine transform, and selection coefficient processing. Through MFCC, speech signal features may be extracted from the message to be summarized corresponding to the speech state, including but not limited to: frequency, intensity, and duration.

[0063] Extracting image features through pre-trained convolutional neural networks can extract deep-level features of images, which is very effective for image classification and recognition.

[0064] The features of different modalities obtained can be fused. Specifically, the features of each modality can be weighted and added according to the aforementioned modal feature weights. The embodiments of the present invention can improve the comprehensiveness and reliability of feature extraction by performing different types of feature extraction operations on information of different modalities, thereby improving the accuracy of message summarization and fusing multi-modal information.

[0065] Step 260: Input the feature fusion result into a summary extraction model generated by pre-training to obtain a message summary of the multimodal message to be summarized.

[0066] Step 270: Perform text analysis on the message digest, remove redundant data, and generate a first message digest that meets word order requirements.

[0067] After the message digest is obtained, the message digest may be post-processed, wherein the post-processing includes but is not limited to removing redundancy from the message digest, adjusting the word order, avoiding the introduction of content not in the original information, and adjusting according to user feedback.

[0068] Removing redundant data from a message summary may involve using text analysis tools or rules to identify and remove redundant words or sentences from the message summary, thereby improving the conciseness of the summary.

[0069] When generating the first message digest that meets the word order requirement, the word order of the message digest can be adjusted in a variety of ways to make the order of appearance of key information logical and make the message digest clearer and easier to understand. Ways to adjust the word order include but are not limited to manual, semi-automatic, and fully automatic. For example, the word order of the message digest can be adjusted through a word order adjustment network model.

[0070] Step 280: Compare the first message digest with the feature extraction result, merge the contents in the first message digest that do not belong to the message to be summarized, and obtain a second message digest.

[0071] By comparing the first message digest with the feature extraction result, it can be determined whether the content in the first message digest belongs to the feature extraction result (that is, whether it belongs to the message to be summarized). Data merging can be performed for the first message digest that does not belong to the message to be summarized. For example, the first message digest can be summarized, and the first message digest that does not belong to the message to be summarized can be merged into other message digests based on semantics. Alternatively, the first message digest that does not belong to the message to be summarized can be directly deleted. In this way, the obtained second message digest can maintain contextual consistency with the original information, making it easier for users to recall or search for relevant information based on the message digest.

[0072] Based on the above implementation, optionally, after inputting the feature fusion result into a pre-trained summary extraction model to obtain a message summary of the multimodal message to be summarized, it also includes: obtaining user feedback data on the message summary, and optimizing the summary extraction model according to the feedback data.

[0073] By optimizing the summary extraction model through user feedback data, a feedback loop mechanism can be established to help improve the quality of the model's message summarization based on user feedback information.

[0074] The technical solution of the embodiment of the present invention obtains multimodal original messages to be summarized and preprocesses the original messages to be summarized of each modality to obtain multimodal messages to be summarized; identifies the message body, message intent, message category, and at least one of message keywords in the messages to be summarized of each modality; determines the message importance recognition result of the messages to be summarized according to the message body, message intent, message category, and at least one of message keywords; generates the modal feature weights of the messages to be summarized of each modality according to the message importance recognition result and the modal importance of the messages to be summarized; extracts features of the messages to be summarized of each modality, and generates the multimodal feature fusion result according to the feature extraction result and the corresponding modal feature weights; The feature fusion result is input into the summary extraction model generated by pre-training to obtain the message summary of the multimodal message to be summarized; the message summary is subjected to text analysis to remove redundant data and generate a first message summary that meets the word order requirements; the first message summary is compared with the feature extraction result, and the content in the first message summary that does not belong to the message to be summarized is merged to obtain the second message summary, which solves the problem that the message summary does not integrate multimodal data. By weighting the features of the multimodal messages based on the importance of the messages, the features of the multimodal messages are fused, and then the message summary is generated, the reliance of the message summary on single modal data can be avoided, the semantic messages under multimodality can be deeply captured, and the quality and comprehensiveness of the message summary can be improved.

[0075] Embodiment 3

[0076] Figure 3 1 is a schematic diagram of a structure of a message summarization device based on multi-modality provided according to Embodiment 3 of the present invention. Figure 3 As shown, the device includes: a message importance identification module 310, a modal feature weight generation module 320, a feature fusion result generation module 330 and a message summary determination module 340. Among them:

[0077] The message importance identification module 310 is used to obtain multi-modal messages to be summarized and identify the message importance of the messages to be summarized in each mode;

[0078] A modality feature weight generation module 320, for generating modality feature weights of the messages to be summarized in each modality according to the message importance recognition result and the modality importance of the messages to be summarized;

[0079] The feature fusion result generating module 330 is used to extract features of the messages to be summarized in each modality, and generate a multi-modal feature fusion result according to the feature extraction results and the corresponding modality feature weights;

[0080] The message summary determination module 340 is used to input the feature fusion result into the summary extraction model generated by pre-training to obtain the message summary of the multi-modal message to be summarized.

[0081] Optionally, the message importance identification module 310 includes:

[0082] An importance element identification unit, used to identify at least one of a message body, a message intent, a message category, and a message keyword in the messages to be summarized in each modality;

[0083] The message importance identification unit is used to determine the message importance identification result of the message to be summarized according to at least one of the message body, message intent, message category, and message keywords.

[0084] Optionally, the message importance identification module 310 includes:

[0085] The message preprocessing unit is used to obtain the original multi-modal messages to be summarized, and pre-process the original messages to be summarized in each mode to obtain the multi-modal messages to be summarized.

[0086] Optionally, the message preprocessing unit includes at least one of the following:

[0087] A text message preprocessing unit, used to segment the original message to be summarized in the text state, remove information irrelevant to the message summary, and obtain the message to be summarized corresponding to the text state;

[0088] A voice message preprocessing unit, used to perform denoising processing on the original message to be summarized in the voice state through a low-pass filter and a high-pass filter, and perform text conversion operation to obtain the message to be summarized corresponding to the voice state;

[0089] The image message preprocessing unit is used to perform denoising processing of the original message to be summarized of the image state by using a mean filter and image feature extraction operation to obtain the message to be summarized corresponding to the image state.

[0090] Optionally, the feature fusion result generating module 330 includes at least one of the following:

[0091] A text feature extraction unit, for evaluating the importance of words to the text according to a text weight algorithm for the message to be summarized corresponding to the text state, and obtaining features corresponding to the text state;

[0092] A speech feature extraction unit, for extracting speech signal features of the message to be summarized corresponding to the speech state by using speech signal processing to obtain features corresponding to the speech state;

[0093] The image feature extraction unit is used to extract image features of the to-be-summarized message corresponding to the image state by using a pre-trained convolutional neural network to obtain features corresponding to the image state.

[0094] Optionally, the device further includes:

[0095] A first message summary generation module is used to input the feature fusion result into a pre-trained summary extraction model to obtain a message summary of the multimodal message to be summarized, perform text analysis on the message summary, remove redundant data, and generate a first message summary that meets the word order requirements;

[0096] The second message summary generating module is used to compare the first message summary with the feature extraction result, merge the contents in the first message summary that do not belong to the message to be summarized, and obtain the second message summary.

[0097] Optionally, the device further includes:

[0098] The summary extraction model optimization module is used to input the feature fusion result into the pre-trained summary extraction model to obtain the message summary of the multimodal message to be summarized, obtain the user's feedback data on the message summary, and optimize the summary extraction model according to the feedback data.

[0099] The multimodal message summarization device provided in the embodiment of the present invention can execute the multimodal message summarization method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0100] In the technical solution of the embodiment of the present invention, the collected information is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0101] When conducting big data analysis based on information and using artificial intelligence technology for automated decision-making, the corresponding operation entrance is provided to users for them to choose to agree or reject the automated decision results; if the user chooses to reject, the expert decision-making process will be entered.

[0102] Embodiment 4

[0103] Figure 4A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0104] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0105] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0106] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as a message summarization method based on multimodality.

[0107] In some embodiments, the message summarization method based on multimodality can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the message summarization method based on multimodality described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to execute the message summarization method based on multimodality in any other appropriate manner (for example, by means of firmware).

[0108] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0109] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0110] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0111] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0112] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0113] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0114] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0115] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A message summarization method based on multimodality, characterized in that: include: Acquire multimodal messages to be summarized, and identify the importance of the messages to be summarized in each mode; Generating modal feature weights of the messages to be summarized in each modality according to the message importance recognition result and the modal importance of the messages to be summarized; Extracting features of the messages to be summarized in each modality, and generating a multi-modal feature fusion result according to the feature extraction result and the corresponding modality feature weight; The feature fusion result is input into a summary extraction model generated by pre-training to obtain a message summary of the multimodal message to be summarized.

2. The method according to claim 1, characterized in that The message importance is identified for the messages to be summarized in each mode, including: Identify at least one of a message body, a message intent, a message category, and a message keyword in the to-be-summarized message of each modality; According to at least one of the message body, the message intent, the message category, and the message keywords, a message importance identification result of the message to be summarized is determined.

3. The method according to claim 1, characterized in that Get multi-modal messages to be summarized, including: The multimodal original messages to be summarized are obtained, and the original messages to be summarized of each mode are preprocessed to obtain the multimodal messages to be summarized.

4. The method according to claim 3, characterized in that Preprocessing the original messages to be summarized in each mode to obtain multi-modal messages to be summarized, including at least one of the following: Segmenting the original message to be summarized in the text state, removing information irrelevant to the message summary, and obtaining the message to be summarized corresponding to the text state; The original message to be summarized in the voice state is subjected to denoising processing through a low-pass filter and a high-pass filter, and is converted into text, so as to obtain the message to be summarized corresponding to the voice state; The original message to be summarized of the image state is subjected to denoising processing using a mean filter and a picture feature extraction operation to obtain the message to be summarized corresponding to the image state.

5. The method according to claim 1, characterized in that Feature extraction is performed on the messages to be summarized in each mode, including at least one of the following: For the message to be summarized corresponding to the text state, evaluating the importance of words to the text according to a text weight algorithm to obtain features corresponding to the text state; For the message to be summarized corresponding to the voice state, extracting voice signal features by voice signal processing to obtain features corresponding to the voice state; For the message to be summarized corresponding to the image state, a pre-trained convolutional neural network is used to extract image features to obtain features corresponding to the image state.

6. The method according to claim 1, characterized in that After the feature fusion result is input into a pre-trained summary extraction model to obtain a message summary of the multimodal message to be summarized, the method further includes: Performing text analysis on the message digest, removing redundant data, and generating a first message digest that meets word order requirements; The first message digest is compared with the feature extraction result, and the content in the first message digest that does not belong to the message to be summarized is merged to obtain a second message digest.

7. The method according to claim 1, characterized in that After the feature fusion result is input into a pre-trained summary extraction model to obtain a message summary of the multimodal message to be summarized, the method further includes: Acquire user feedback data on the message summary, and optimize the summary extraction model according to the feedback data.

8. A message summarizing device based on multimodality, characterized in that: include: A message importance identification module, used to obtain multi-modal messages to be summarized, and identify the message importance of the messages to be summarized in each mode; A modal feature weight generation module, used to generate modal feature weights of the messages to be summarized in each modality according to the message importance recognition result and the modal importance of the messages to be summarized; A feature fusion result generating module is used to extract features of the messages to be summarized in each modality, and generate a multi-modal feature fusion result according to the feature extraction results and the corresponding modality feature weights; The message summary determination module is used to input the feature fusion result into a summary extraction model generated by pre-training to obtain a message summary of the multimodal message to be summarized.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the multimodal-based message summarization method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the multimodal message summarization method according to any one of claims 1 to 7 when executed.

Images