Conference summary generation method based on sublingual acoustic features

Abstract

The invention discloses a conference summary generation method based on sublingual acoustic features. The method comprises the following steps: converting a conference audio into a text, and extracting key information of a text level by using a deep learning model; key information of an acoustic level is extracted from the conference audio, the content of significant speech features meeting a set standard is obtained, and the significant speech features are determined based on intonation, speech speed and pause duration; an open question is designed with the aim of mining clues of the conference, so that a large language model is guided to answer, and answer information is obtained; and integrating the key information of the text level, the key information of the acoustic level, the transcriptional text, the open question and the answer information, inputting the integrated information into a large language model, and generating a conference summary. According to the invention, the conference summary generation quality and efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of acoustic feature recognition, and more specifically, to a conference summary generation method based on paralinguistic acoustic features. Background Art

[0002] Meetings are the main way to exchange information in the workplace. During meetings, speakers' conversations are usually unstructured or unedited, and are often discussed in real time without prior preparation. Therefore, the density of important information in meetings is usually low, and the wording used may appear scattered, repetitive, and incoherent. In other words, the speeches in meetings may contain a lot of "noise". In this case, meeting summaries become very important, which not only provides a basis for participants to review and trace the history of the meeting, but also enables people who did not attend the meeting to quickly understand the important content and conclusions of the meeting. However, it is a challenging task for people to organize meeting summaries. The recorder needs to quickly record the key information in the meeting. In this process, they face many difficulties such as different speaking speeds of participants, understanding of professional knowledge, and handling of emergencies. After the meeting, a summary needs to be written based on the notes taken during the meeting and the understanding and recollection of the meeting content.

[0003] In the prior art, automatic summary generation schemes mainly include two categories: extractive summaries and generative summaries. In the early days, some studies focused on extractive conference summaries, for example, extracting certain parts of the text to form a summary. However, the conference summaries obtained by this extraction method have poor readability and are difficult for third-party readers to understand. The conference summaries obtained based on the generative method are clear and concise, and the readability is much higher. In the early years, the emergence of Transformer-based pre-trained models, such as BERT, BART, and T5, brought significant improvements to natural language generation (NLG) tasks. Although they have achieved great success in generation tasks, there is still a disadvantage that a complete corpus is required. Current work on conference summaries mainly focuses on supervised methods, which makes the training model inevitably rely on a large corpus.

[0004] One potential way to address the problem of limited data is to turn to conference summarization techniques that rely on weaker supervision. In recent years, the popularity of large-scale language models (LLMs) and their strong generalization capabilities have brought new opportunities for conference summarization. LLMs are able to understand world knowledge and context, and conference summaries can be obtained directly using LLMs without any additional training. However, if the conference record text is directly handed over to LLM for processing, the generated result is completely based on the text modality, which means that the key information expressed by other modalities may be ignored, resulting in the final generated summary being unfocused and of poor quality.

[0005] In summary, there is a need to improve the existing technology to further improve the quality and readability of automatic summary generation. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method for generating conference summaries based on paralinguistic acoustic features. The method comprises the following steps:

[0007] Convert conference audio to text, obtain transcripts, and use deep learning models to extract key information at the text level;

[0008] Extracting key information at the acoustic level from the conference audio to obtain content with significant speech features that meet set standards, where the significant speech features are determined based on intonation, speech speed, and pause duration;

[0009] With the goal of mining clues from the meeting, we designed open-ended questions to guide the large language model to answer and obtain answer information;

[0010] The key information at the text level, the key information at the acoustic level, the transcribed text, the open questions and the answer information are integrated and input into the large language model to generate a meeting summary.

[0011] Compared with the prior art, the advantage of the present invention is that the conference summary generation method based on paralinguistic acoustic features first uses a deep learning method to extract keyword information at the text level, then extracts acoustic feature information such as intonation, speech rate and pauses from the conference audio, and identifies speech segments with significant features; then designs open questions to guide the large language model to mine key information and clues in the meeting; then integrates text keywords, acoustic key information, transcribed text and the answers of the large language model to generate the final conference summary. The present invention integrates paralinguistic acoustic feature information into the conference summary generation process, making full use of the emotion, attitude and importance information contained in the voice information, and effectively improving the accuracy and readability of the conference summary.

[0012] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0014] Figure 1 is a process diagram of a method for generating conference summaries based on paralinguistic acoustic features according to an embodiment of the present invention;

[0015] Figure 2 is a flow chart of a method for generating conference summaries based on paralinguistic acoustic features according to an embodiment of the present invention;

[0016] Figure 3 is a schematic diagram comparing changes in fundamental frequency in a normal tone and an emphasized tone according to an embodiment of the present invention;

[0017] In the attached figure, Speech Keyphase-speech key segment; Intonation Recognition-intonation recognition; Long Stop and High Speed ​​Recognition-long stop and fast speech recognition; Keyphase Extraction-key segment extraction. DETAILED DESCRIPTION

[0018] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0019] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0020] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0021] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0022] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0023] Existing meeting summary generation models focus almost entirely on textual information in speech transcriptions, while ignoring other paralinguistic information during the meeting, such as intonation, body language, and facial expressions. However, these paralinguistic cues can convey richer information and help understand context and perceive key information. This invention explores for the first time the use of acoustic modal information to assist in the generation of meeting summaries, and uses a highly interpretable machine learning method to extract key information at the speech level from meeting records, such as emphasis on spoken content with higher intonation, appropriate pauses, and faster speech, which can be well adapted to different speakers.

[0024] In general, see Figure 1 As shown in the figure, the provided conference summary generation method based on paralinguistic acoustic features first transcribes a conference recording (A) into text (B), and then processes the conference recording and the transcribed text (C) to extract language acoustic features. After processing, a text with key phrase annotations (including importance) is obtained, and then processed by a large model (D) to output the final conference summary (E).

[0025] Specifically, combined Figure 1 and Figure 2 As shown, the provided conference summary generation method based on paralinguistic acoustic features includes the following steps:

[0026] Step S1, convert the conference audio into text, and use the deep learning model to extract key information at the text level.

[0027] Key phrases are the core of a text segment and are crucial to the overall semantics of the text. Studies have shown that keywords are the core part of a sentence and can be used as an indicator for selecting important sentences. In order to minimize the waste of resources caused by training models and achieve efficient conference summary generation, an unsupervised model can be used to extract key phrases.

[0028] In one embodiment, extracting key information from text includes: firstly transcribing the audio into text, and using an unsupervised keyword extraction model to obtain key information. For example, the keyword extraction model uses the MDERank model (a keyword extraction method based on BERT), which ranks candidate keywords by using the BERT embedding similarity between the source document and the mask document, and has stronger performance and generalization ability. In this paper, for each speaker, K i A set of text keywords representing the speaker.

[0029] It should be noted that other deep learning models can also be used to extract key information of the text, such as KeyBert, TextRank, etc.

[0030] Step S2, extracting key information at the acoustic level from the conference audio, focusing on text content with large changes in intonation and speed and obvious pauses.

[0031] For example, the fundamental frequency f0 information of speech is extracted from conference recordings, and the speaking speed of each sentence is calculated. The clustering algorithm is used to select the parts of each speaker where the intonation and speaking speed change greatly and the pauses are obvious as key information at the acoustic level.

[0032] like Figure 3 As shown, the left picture is normal intonation, and the right picture is emphasized intonation. It can be clearly seen that the fundamental frequency f0 of the right picture is higher and more volatile overall.

[0033] In one embodiment, extracting key information at the acoustic level includes the following steps:

[0034] Step S201, extracting the baseband.

[0035] The fundamental frequency (f0) of speech is a key indicator for describing intonation changes. The fundamental frequency reflects the lowest frequency component of the sound and is used to analyze the rise and fall of intonation. For example, for all speech clips of each speaker First, the fundamental frequency value of each segment is extracted by the fundamental frequency extraction algorithm pYIN, and then the average fundamental frequency of each segment is calculated. Used for subsequent intonation analysis and processing.

[0036] Step S202, calculating the speaking rate.

[0037] Speech rate is an important indicator for measuring language fluency and comprehensibility. For example, speech rate is calculated by the ratio of the number of words in each speech segment to the speech duration. Specifically, for each speech segment First, count the number of words in the transcribed text Then calculate its duration by The final speaking speed

[0038] Step S203, calculating the pause time.

[0039] Pauses play an important role in speech and can effectively enhance the expression of key points. For example, when processing pauses, a pause time calculation method based on word units is used. For each speech segment, First obtain its transcript Then calculate the pause time before each word based on the start and end time of each word

[0040] Step S204: extracting speech segments with significant features based on intonation, speaking speed and pause duration.

[0041] In order to automatically identify which speech features (such as high intonation, fast speech rate, and long pauses) meet the criteria of "high", "fast", and "long", a clustering method in unsupervised learning can be used. Considering that the fixed threshold method cannot be universally applicable due to the natural differences between each speaker, in one embodiment, a K-Means clustering algorithm is used to automatically identify and extract speech segments with significant features. For example, for each speaker S i , we get the following three feature sets: a high-intonation segment set, a fast-speech segment set, and a long-pause segment set.

[0042] In summary, by focusing on acoustic pattern information, multiple important acoustic features are extracted. These features help understand the different emotions, attitudes, and key information of meeting participants. This is because when people want to emphasize a point or highlight specific information, they usually raise their intonation, making the key words or key phrases more prominent in the speech, making them more important in the entire sentence or paragraph. This change in intonation can help the audience better understand and remember the information conveyed.

[0043] Step S3, by designing open questions, guiding the large language model to answer and explore clues and important information of the meeting.

[0044] To leverage the large language model to generate more professional conference summaries, the design thinking chain optimizes the prompts input to the large language model.

[0045] For example, by guiding LLM to answer multiple questions to mine clues and important information of the meeting, it is found through analysis that the information of open questions is more effective than that of structured questions. Therefore, in one embodiment, four open questions (Q) are designed to help LLM mine key information and clues in the meeting. The four questions designed are as follows:

[0046] Q1: Who are the participants in this meeting?

[0047] Q2: What topics were discussed at this meeting?

[0048] Q3: What are the key events related to the topic during the meeting?

[0049] Q4: What consensus was reached on relevant topics at the meeting?

[0050] By designing open-ended questions, the summaries generated subsequently are more advantageous in terms of cross-content applicability and effectiveness.

[0051] Step S4, integrating the key information at the text level, the key information at the acoustic level, the transcribed text, and the guiding questions to generate a summary.

[0052] In this step, the final summary is generated by integrating the key information of the text, the key information of the voice, the transcribed text, and the guiding questions. For example, the LLM is provided with text keywords, large fluctuations in intonation and speech speed, and long pauses as key information, and different weights are assigned to these key information. Subsequently, the key information is weighted by overlapping the time dimension to obtain the final importance of the text and embedded in the prompt word.

[0053] Specifically, LLM's answers to the four questions (R) are used for further information integration. Next, the key information (both textual and acoustic), the meeting transcript, the guiding questions, and LLM's answers are concatenated to form a new input form [K, D, Q, R], which is then fed into LLM to generate the final summary. It should be noted that different weights are assigned to the key information at the textual and acoustic levels.

[0054] In summary, in order to improve the adaptability of LLM to different conference contents and reduce the training cost to meet the original intention of using LLM, a weighted method is used to integrate text key information and voice key information, and guide questions related to conference elements are embedded to constrain and guide LLM to generate higher quality conference summaries. It should be noted that this constraint method prior to LLM can seamlessly transition to a more advanced LLM.

[0055] Furthermore, the entire conference summary generation process can be encapsulated and an application can be developed based on the concept of human-computer collaboration, or Paralinguistic-SUM application. During the entire summary generation process, users can participate and interact with the system.

[0056] Since artificial intelligence (AI) may have errors and each user's interpretation of the results is subjective, users may not be completely satisfied with the annotation results given by AI. If users are not given the opportunity to adjust the annotation results, it may directly affect the quality of the generated conference abstracts and user satisfaction. For the above reasons, the annotation results are kept open and allowed to be adjusted by users. By providing users with the ability to adjust the annotation results, the quality of the conference abstracts and the high satisfaction of users are guaranteed.

[0057] In order to reduce the interaction burden between users and the system and improve ease of use, the designed Paralinguistic-SUM application interface includes: audio playback and transcript synchronization, key phrase annotation adjustment, and answer modification functions. For example, interface A provides a browsing function for conference audio, supports variable speed playback, and includes synchronized transcripts. Interfaces B and C allow users to adjust key phrases and the responses given by the large language model to guiding questions, respectively.

[0058] Specifically, for the function of synchronous display of audio playback and transcript text, the application allows users to browse the meeting audio and supports variable speed playback. At the same time, the audio transcript text is synchronized with the audio playback and adds a timestamp. For the key phrase annotation adjustment function, users can modify the key phrases generated by AI. The annotated key phrases are distinguished by different background colors, such as red for the most important, yellow for medium importance, and green for the least important. Users can modify the level of the phrase in the part that needs to be adjusted, add new key phrases, or cancel the existing key phrase annotations. For the answer modification function, users can modify the AI's answers to guiding questions to ensure that the generated summary is more in line with the actual content. The application adds labels to the AI-generated annotations to help distinguish the AI ​​annotations from the user's modifications.

[0059] In summary, since the present invention is a tool for collaboration with humans, errors may occur. By allowing users to participate in the conference summary generation process, their subjective initiative can be brought into play and the quality of summary generation can be improved. The designed application can be embedded in smartphones or other smart devices, which is conducive to users injecting their own understanding and knowledge, providing users with a channel to modify all content, and providing friendly interaction.

[0060] In order to further verify the effect of the present invention, the results of the generation after using the paralinguistic acoustic features and the results of the generation without the paralinguistic acoustic features are compared for the same text. After analysis and comparison, the summary generated based on the paralinguistic acoustic features can better convey the speaker's intention because it takes into account acoustic features such as speaking speed and intonation.

[0061] In summary, the present invention provides a method for enhancing the conference summary capability based on a large language model based on paralinguistic acoustic features. By incorporating paralinguistic acoustic feature information into the conference summary generation process, the emotion, attitude, and importance information contained in the speech information are fully utilized. The use of acoustic modal information to assist in the generation of conference summaries effectively improves the accuracy and readability of conference summaries.

[0062] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0063] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0064] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0065] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0066] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0067] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0068] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0069] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0070] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A method for generating conference summaries based on paralinguistic acoustic features, comprising the following steps: Convert conference audio to text, obtain transcripts, and use deep learning models to extract key information at the text level; Extracting key information at the acoustic level from the conference audio to obtain content with significant speech features that meet set standards, where the significant speech features are determined based on intonation, speech speed, and pause duration; With the goal of mining clues from the meeting, we designed open-ended questions to guide the large language model to answer and obtain answer information; The key information at the text level, the key information at the acoustic level, the transcribed text, the open questions and the answer information are integrated and input into the large language model to generate a meeting summary.

2. The method according to claim 1, wherein: The following steps are used to obtain the content of the significant speech features that meet the set criteria: For each speech segment of the speaker Extract the fundamental frequency value of each speech segment and compare it with the average fundamental frequency to obtain the degree of change of intonation, where the subscript i is the speaker index and the superscripts 1 to m are the speech segment indexes; For each speech segment Count the words in the transcript And calculate its duration by Get speech speed For each speech segment Get its transcript And calculate the pause time before each word based on the start time and end time of each word Based on the acquired information on the degree of intonation change, speaking speed and pause time, the content of significant speech features that meet the set standards is extracted, including a set of high-intonation segments, a set of fast-speaking segment sets, and a set of long pause segment sets.

3. The method according to claim 2, wherein: The content of the significant speech features that meet the set standard is obtained by using K-Means clustering.

4. The method according to claim 1, wherein: The open questions include: participants in the meeting, topics discussed in the meeting, key events related to the topics, and consensus reached on related topics.

5. The method according to claim 1, wherein: The key information at the text level is extracted using the MDERank model, including: sorting candidate keywords based on the embedding similarity between the source document and the mask document; and selecting a set number of keyword combinations as the key information at the text level according to the sorting result.

6. The method according to claim 1, wherein: The input form input to the large language model is expressed as [K, D, Q, R], where K represents the key information at the text level and the key information at the acoustic level, D represents the transcribed text, Q represents the open question, and R represents the answer information, and the key information at the text level and the key information at the acoustic level are assigned different set weights.

7. The method according to claim 1, wherein: The process of generating the conference summary also includes providing an interface for users to modify.

8. The method according to claim 1, wherein: The key information at the text level, the key information at the acoustic level, the transcription text, the open questions and the answer information are weighted and integrated in an overlapping manner in a time dimension.

9. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images