Conference summary generation method and device

Abstract

The invention provides a conference summary generation method and device, and the method comprises the steps: obtaining an audio and video file of a target conference, and the audio and video file comprises an audio file and / or a video file; performing voice activity detection processing, text conversion processing and speaker recognition processing on the audio and video file to obtain a target dialogue text; constructing prompt words based on the application scene of the target conference and the target dialogue text; and inputting the cue word into a conference summary generation large model, and generating a conference summary corresponding to the target conference. According to the invention, the problem of low conference summary generation efficiency is solved, and more time is saved; the generation result of the conference summary generation large model is optimized by adopting a prompt word engineering mode, so that the accuracy of generating the conference summary can be further improved; the conference summary is generated by adopting the conference summary generation large model, compared with a traditional neural language programming technology, feature engineering construction is omitted, and the performance is better.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular, to a method and device for generating meeting minutes. Background Art

[0002] In daily meetings, it is usually necessary to manually sort out and summarize the content of the meeting to obtain meeting minutes. To solve the problems of low efficiency and time-consuming of manually generating meeting minutes, the meeting minutes generation system has developed rapidly.

[0003] In the existing meeting minutes generation system, business users log in to the business side through the system interface, select an existing meeting minutes generation template, customize the meeting minutes template according to requirements, upload the key information records in the meeting in the form of a text file, preview the generated meeting minutes and download them, which improves the generation efficiency of meeting minutes.

[0004] However, at present, many meetings are mainly recorded by screen recording or audio recording, that is, the meeting content is an audio-visual file. And the existing meeting minutes generation system cannot achieve one-key generation from audio-visual to meeting minutes, resulting in only manual generation of meeting minutes for audio-visual meetings, with a low success rate of generating meeting minutes and limited application scope. Summary of the Invention

[0005] In view of the problems existing in the prior art, an embodiment of the present invention provides a method and device for generating meeting minutes.

[0006] The present invention provides a method for generating meeting minutes, including the following: Obtain an audio-visual file of a target meeting, where the audio-visual file includes an audio file and / or a video file; Perform voice activity detection processing, text conversion processing, and speaker recognition processing on the audio-visual file to obtain a target dialogue text; Construct a prompt word based on the application scenario of the target meeting and the target dialogue text; Input the prompt word into a meeting minutes generation large model to generate meeting minutes corresponding to the target meeting.

[0007] According to the method for generating meeting minutes provided by the present invention, the performing voice activity detection processing, text conversion processing, and speaker recognition processing on the audio-visual file to obtain a target dialogue text includes: Detect the voice activity part of the audio-visual file, generate timestamp information, and mark at least one voice segment in the audio-visual file based on the timestamp information; Perform text conversion processing on each of the speech segments to obtain text fragments corresponding to each of the speech segments, merge the text fragments to generate an initial dialogue text; and perform speaker identification processing on each of the speech segments to obtain speaker labels corresponding to each of the speech segments. Based on the speaker labels corresponding to each of the speech segments, add speaker labels to the corresponding text fragments in the initial dialogue text to obtain a target dialogue text.

[0008] According to a meeting minutes generation method provided by the present invention, the constructing a prompt based on the application scenario of the target meeting and the target dialogue text includes: Obtain the meeting data of the target meeting, and based on the meeting data, determine the application scenario of the target meeting, where the meeting data includes meeting attributes and / or meeting content materials; According to the application scenario of the target meeting, determine the meeting minutes objective, where the meeting minutes objective represents the standard to be achieved by the generated meeting minutes; Construct a prompt based on the meeting minutes objective and the target dialogue text.

[0009] According to a meeting minutes generation method provided by the present invention, the elements of the prompt at least include role, task, background, and output format; The constructing a prompt based on the meeting minutes objective and the target dialogue text includes: Based on the meeting minutes objective, determine the role, background, and output format of the prompt; Based on the meeting minutes objective and the target dialogue text, determine the task of the prompt.

[0010] According to a meeting minutes generation method provided by the present invention, the determining the task of the prompt based on the meeting minutes objective and the target dialogue text includes: Obtain the associated content of the audio-visual file, where the associated content at least includes relevant meeting materials; Search for historical meeting materials associated with the relevant meeting materials from a pre-constructed meeting database, and based on the meeting minutes objective, the target dialogue text, the associated content, and the historical meeting materials, determine the task of the prompt.

[0011] According to a meeting minutes generation method provided by the present invention, it further includes: Add the relevant meeting materials to the meeting database, and update the index of the meeting database based on the location information of the relevant meeting materials in the meeting database.

[0012] A method for generating meeting minutes provided by the present invention, determining the application scenario of the target meeting based on the meeting data includes: Analyze the meeting content materials in the meeting data to determine the application scenario of the target meeting, where the meeting content materials include at least one of the audio-visual file, the target dialogue text, relevant meeting materials, and other voice text files.

[0013] A method for generating meeting minutes provided by the present invention, before performing voice activity detection processing, text conversion processing, and speaker recognition processing on the audio-visual file to obtain the target dialogue text, further includes: Preprocess the audio signal in the audio-visual file, and the preprocessing includes at least one of denoising processing, accent recognition and annotation processing, and language recognition and annotation processing.

[0014] The present invention also provides a meeting minutes generation device, including: An acquisition module configured to acquire the audio-visual file of the target meeting, where the audio-visual file includes an audio file and / or a video file; A processing module configured to perform voice activity detection processing, text conversion processing, and speaker recognition processing on the audio-visual file to obtain the target dialogue text; A construction module configured to construct a prompt word based on the application scenario of the target meeting and the target dialogue text; A generation module configured to input the prompt word into a meeting minutes generation large model to generate the meeting minutes corresponding to the target meeting.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, it implements the meeting minutes generation method as described in any one of the above.

[0016] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the meeting minutes generation method as described in any one of the above.

[0017] The present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the meeting minutes generation method as described in any one of the above.

[0018] The meeting minutes generation method and device provided by the present invention obtain an audio-video file of a target meeting, where the audio-video file includes an audio file and / or a video file; perform voice activity detection processing, text conversion processing, and speaker recognition processing on the audio-video file to obtain a target dialogue text; construct a prompt based on the application scenario of the target meeting and the target dialogue text; input the prompt into a meeting minutes generation large model to generate the meeting minutes corresponding to the target meeting. It realizes the conversion from audio-visual to meeting minutes, which is more in line with the current meeting habits; generates meeting minutes through a meeting minutes generation large model, solves the problem of low efficiency in generating meeting minutes, and saves more time; optimizes the generation result of the meeting minutes generation large model by using the method of prompt engineering, which can further improve the accuracy of generating meeting minutes; adopts the meeting minutes generation large model generation technology instead of the traditional neuro-linguistic programming technology, eliminating the construction of feature engineering and having better performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the implementation examples or the description of the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0020] Figure 1 It is one of the schematic flowcharts of the meeting minutes generation method provided by the present invention.

[0021] Figure 2 It is the second schematic flowchart of the meeting minutes generation method provided by the present invention.

[0022] Figure 3 It is the third schematic flowchart of the meeting minutes generation method provided by the present invention.

[0023] Figure 4 It is the schematic structural diagram of the meeting minutes generation device provided by the present invention.

[0024] Figure 5 It is the schematic structural diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention with reference to the drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0026] First, a brief description of the relevant content related to the present invention will be given.

[0027] At the present stage, most of the large model meeting minutes generation systems are based on large model technology, use Python language as the basic programming language, and are built in combination with the modern global wide area network (World Wide Web, Web) development framework Next.js.

[0028] Business users can log in to the business side through the interface of the large model meeting minutes generation system, select the existing meeting minutes generation template, customize the meeting minutes template according to needs, upload the key information records in the meeting in the form of a text file, such as an editable text file (word file), a portable document format (PDF, Portable Document Format) file, preview the generated meeting minutes and download them. This large model meeting minutes generation system can quickly generate meeting minutes.

[0029] However, the problems faced in the existing actual work process are that various meetings are mainly based on screen recording or recording meetings, and various local dialects are often mixed in audio and video meetings; if only the function of processing text data by the large model is supported, additional processing of audio and video to text data is required; at the same time, it is also necessary to consider how to make full use of the large model technology to improve the final effect.

[0030] It can be seen that the existing technology has the following technical problems: it cannot support the one-key generation of text minutes from audio, and it is necessary to extract text data from audio data separately; a large number of feature engineering processes are required; the model performance results need to be optimized.

[0031] The following combines Figures 1-5 Describe the meeting minutes generation method and device of the present invention Figure 1 is one of the flow diagrams of the meeting minutes generation method provided by the present invention. As Figure 1 shown, the method includes steps 101 to 104.

[0032] Step 101: Obtain the audio and video file of the target meeting, and the audio and video file includes an audio file and / or a video file.

[0033] The execution subject of this embodiment can be a meeting minutes generation system or a functional unit, or an intelligent device loaded with this meeting minutes generation system or functional unit, such as a smart phone, a smart computer, etc.

[0034] Specifically, the target meeting refers to the meeting for which meeting minutes need to be generated; the audio and video file refers to the file obtained by performing at least one operation such as recording the meeting, taking images, and screen recording.

[0035] In practical applications, users can upload the audio and video files of the target meeting through the upload interface. Correspondingly, the execution entity receives the audio and video files of the target meeting; or, the execution entity searches for the audio and video files of the target meeting from the preset audio and video file storage area based on the identifier of the target meeting.

[0036] Step 102: Perform voice activity detection processing, text conversion processing, and speaker recognition processing on the audio and video files to obtain the target dialogue text.

[0037] Specifically, the target dialogue text refers to the dialogue text carrying the speaker identifier.

[0038] In practical applications, it is necessary to perform speech-to-text processing on the audio and video files, which mainly includes three processing processes: voice activity detection processing, text conversion processing (automatic speech recognition processing), and speaker recognition processing (speaker segmentation processing).

[0039] The voice activity part in the audio and video files can be detected first, where the voice activity part refers to the part of the audio and video files where there is the speech content of the participants.

[0040] Then, perform text conversion and speaker recognition on the voice activity part, so as to convert the voice activity part into the dialogue text carrying the speaker identifier, that is, the target dialogue text.

[0041] It should be noted that since the video file is composed of video frames, video data, and audio data, when the audio and video file contains a video file, the audio data in the video file can be extracted first to obtain the audio file corresponding to the video file, and then speech-to-text processing is performed on the audio file corresponding to the video file.

[0042] Step 103: Construct a prompt based on the application scenario of the target meeting and the target dialogue text.

[0043] Specifically, the prompt refers to the text input when interacting with the meeting minutes generation large model, which is used to guide the meeting minutes generation large model to generate specific outputs. Among them, the meeting minutes generation large model, that is, the artificial intelligence large model, refers to the "large parameter" model trained using large-scale data and powerful computing capabilities, which has high generality and generalization capabilities and can be applied to fields such as natural language processing, image recognition, and speech recognition. It can be divided into large language models, vision large models, multi-modal large models, and basic large models, etc. The meeting minutes generation large model in the embodiments of the present invention can be a large language model.

[0044] In practical applications, the task and output format of the prompt can be determined according to the application scenario of the target meeting and the target dialogue text, so as to construct the prompt.

[0045] Exemplarily, select the meeting minutes format corresponding to the application scenario as the output format of the prompt. Based on the target dialogue text and the meeting minutes format, select an example of the prompt. For the task of generating a prompt according to the target dialogue text, such as generating meeting minutes based on the target dialogue text, so as to obtain the prompt. Among them, since the task of the prompt is based on the target dialogue text, the task of the prompt contains the target dialogue text.

[0046] Exemplarily, parse the application scenario and the target dialogue text to determine the required meeting minutes format. Based on this meeting minutes format, select a prompt from the meeting minutes prompt library. Among them, the output format of the selected prompt is the meeting minutes format, and the task of the meeting minutes prompts in the meeting minutes prompt library is to generate meeting minutes. Among them, since the task of the prompt is unified, the task of the prompt does not contain the target dialogue text.

[0047] Step 104: Input the prompt into the meeting minutes generation large model to generate the meeting minutes corresponding to the target meeting.

[0048] In practical applications, when the task of the prompt contains the target dialogue text, the prompt can be directly input into the meeting minutes generation large model. The meeting minutes generation large model completes the task of generating meeting minutes under the guidance of the prompt, that is, outputs the meeting minutes in the target format corresponding to the target meeting.

[0049] When the task of the prompt does not contain the target dialogue text, it is necessary to input both the target dialogue text and the prompt into the meeting minutes generation large model. The meeting minutes generation large model processes the target dialogue text under the guidance of the prompt, so as to output the meeting minutes in the target format indicated by the prompt, that is, the meeting minutes in the target format corresponding to the target meeting.

[0050] The meeting minutes generation method provided by the present invention obtains the audio-video file of the target meeting, and the audio-video file includes an audio file and / or a video file; performs voice activity detection processing, text conversion processing, and speaker recognition processing on the audio-video file to obtain the target dialogue text; constructs a prompt based on the application scenario of the target meeting and the target dialogue text; inputs the prompt into the meeting minutes generation large model to generate the meeting minutes corresponding to the target meeting. It realizes the conversion from audio-visual to meeting minutes, which is more in line with the current meeting habits; generates meeting minutes through the meeting minutes generation large model, solves the problem of low efficiency in generating meeting minutes, and saves more time; optimizes the generation result of the meeting minutes generation large model by adopting the method of prompt engineering, which can further improve the accuracy of generating meeting minutes; adopts the meeting minutes generation large model generation technology instead of the traditional neural language programming technology, eliminating the construction of feature engineering and having better performance.

[0051] In one or more alternative embodiments of the present invention, the performing speech activity detection processing, text conversion processing, and speaker recognition processing on the audio-video file to obtain a target dialogue text includes: Detecting a speech activity portion of the audio-video file, generating timestamp information, and marking at least one speech segment in the audio-video file based on the timestamp information; Performing text conversion processing on each of the speech segments to obtain a text fragment corresponding to each of the speech segments, merging the text fragments to generate an initial dialogue text; and performing speaker recognition processing on each of the speech segments to obtain a speaker label corresponding to each of the speech segments; Based on the speaker labels corresponding to the speech segments, adding speaker labels to the corresponding text fragments in the initial dialogue text to obtain a target dialogue text.

[0052] In practical applications, the purpose of speech activity detection processing is to detect the speech activity portion in an audio-video file and filter out silent segments and noise. Specifically, the input audio-video file can be preprocessed, and speech activity detection technology can be used to detect the speech activity portion, generate timestamp information, and mark each speech segment in the audio-video file.

[0053] Exemplarily, a speech activity detection method based on a Voice Activity Detection (VAD) model can be used to detect the speech activity portion.

[0054] Among them, the speech activity detection model is mainly used to determine whether the input audio signal contains speech and to determine the start and end positions of the speech. The speech activity detection model can distinguish speech segments from non-speech segments (such as silence, background noise, etc.). For example, in a video conferencing software, the speech activity detection model can automatically detect the speech period of the speaker, thereby realizing the voice activation function and saving network bandwidth and computing resources.

[0055] The speech activity detection model detects based on the energy of the audio signal. Speech signals usually have higher energy than background noise. By setting an energy threshold, when the energy of the audio signal exceeds the threshold, it is preliminarily determined that speech may exist. However, this method is easily affected by noise. For example, in a noisy environment, the background noise energy may be high, resulting in misjudgment. Therefore, in addition to energy detection, the speech activity detection model also performs spectral analysis on the audio signal. Since there are certain differences in the spectral distributions of speech signals and noise, speech usually has a more obvious harmonic structure, and the speech activity detection model can utilize these differences to more accurately determine the presence or absence of speech by analyzing spectral features (such as spectral centroid, bandwidth, etc.).

[0056] A voice activity detection method based on a voice activity detection model, that is, the optimized voice activity detection model will use machine learning or deep learning methods for voice activity detection. For example, the Hidden Markov Model (HMM) or deep neural network is used to learn the feature patterns of speech and non-speech. These models can be trained on a large amount of labeled data (including audio samples of speech and non-speech), thereby improving the detection accuracy.

[0057] Furthermore, each speech segment can be extracted from the audio-visual file based on the timestamp information, and text conversion processing and speaker recognition processing are respectively performed on each speech segment.

[0058] The purpose of performing text conversion processing on each speech segment is to convert the detected speech segment into text. Specifically, automatic speech recognition technology is used to convert each speech segment into a corresponding text fragment, and then all text fragments are merged in chronological order to form a continuous dialogue text, that is, the initial dialogue text.

[0059] Exemplarily, an Automatic Speech Recognition (ASR) model can be used to perform text conversion processing on each speech segment.

[0060] The automatic speech recognition model can convert the lexical content in human speech into a computer-readable text format. For example, when using a voice input method to input text on a mobile phone, the automatic speech recognition model is used for recognition. The automatic speech recognition model can identify the speech units (such as phonemes, syllables, words, etc.) in the speech signal and combine these units into meaningful text according to certain language rules.

[0061] The automatic speech recognition model consists of a feature extraction unit, an acoustic model, a language model, and a decoding unit.

[0062] The feature extraction unit preprocesses the input speech signal and extracts its acoustic features. Commonly used acoustic features include Mel-Frequency Cepstral Coefficients (MFCC), etc. These features can better reflect the essential features of the speech signal and facilitate subsequent model processing. For example, MFCC can simulate the human ear's perception of sounds at different frequencies and highlight the important frequency components in speech.

[0063] The acoustic model is one of the core components of an automatic speech recognition model, which is used to learn the mapping relationship between speech features and phonemes. It is generally constructed based on deep neural networks (such as long short-term memory networks, convolutional neural networks, etc.). The acoustic model is trained with a large amount of speech data to learn the phoneme probability distributions corresponding to different speech features. For example, for a specific speech feature vector, the acoustic model can output the probabilities of the phonemes it may correspond to.

[0064] The language model is mainly used to process information such as grammar, semantics, and word order of text. The language model is trained based on a large-scale text corpus and can predict the probability of words appearing in a sentence. In an automatic speech recognition model, the language model can help correct possible recognition errors of the acoustic model and make the recognition result more in line with language habits. For example, according to the knowledge of the language model, the probability of "I go to school" is much higher than that of "I go to xuexiao".

[0065] Decoding unit: Finally, the decoding algorithm of the decoding unit combines the outputs of the acoustic model and the language model to find the most likely text sequence as the recognition result. Common decoding algorithms include the Viterbi algorithm, etc., which can find the path with the highest probability among all possible text paths.

[0066] The purpose of performing speaker recognition processing on each speech segment is to identify and segment different speakers and add speaker labels to each speech segment (text segment). Specifically, speaker recognition technology is used to label each speech segment with a speaker label, such as a speaker serial number (Identity Document, ID).

[0067] Exemplarily, a speaker recognition (Speaker Recognition, SPK) model can be used to perform speaker recognition processing on each speech segment.

[0068] The speaker recognition model, also known as the voiceprint recognition model, is used to identify or verify the identity of a speaker. The speaker recognition model determines which specific person is speaking based on the speaker's speech features, mainly including two tasks: speaker identification (identifying who the current speaker is from multiple known speakers) and speaker verification (verifying whether the current speaker is the person claimed). For example, in a security access control system, it is determined whether the person is an authorized person through voice.

[0069] The speaker recognition model mainly includes two parts: feature extraction and model training and matching.

[0070] Similar to the automatic speech recognition model, the speaker recognition model first extracts information from the speech signal that can represent the speaker's characteristics. These characteristics include acoustic features such as fundamental frequency, formant frequency, and vocal tract length, as well as suprasegmental features such as speech rate, intonation, and prosody. These features can be combined to form the speaker's voiceprint features.

[0071] Model training and matching: Commonly used models include Gaussian Mixture Model - Universal Background Model (GMM - UBM) and deep neural networks (such as deep neural network embeddings), etc. Taking GMM - UBM as an example, first use a large amount of speech data from different speakers to train a universal background model, and then for each target speaker, adapt the universal background model with their speech data to obtain the model for each speaker. In the recognition and matching stage, match the speech features to be recognized with the models of each speaker, calculate the similarity score, and judge the speaker's identity based on the score.

[0072] It should be noted that the text conversion process and the speaker recognition process for the speech segment can be carried out simultaneously or in a certain order, and there is no specific limitation on this.

[0073] Furthermore, after the text conversion process and the speaker recognition process for the speech segment, generate a structured dialogue text, that is, the target dialogue text, from the processed initial dialogue text and speaker labels, and output it.

[0074] In the embodiments of the present invention, by detecting the timestamp information of the speech activity part of the audio - video file and extracting the speech segment in the audio - video file based on the timestamp information, the rapidity and accuracy of the speech segment can be improved. By performing text conversion processing and merging on each speech segment to generate the initial dialogue text, the reliability of the initial dialogue text can be improved. By performing speaker recognition processing on each speech segment to obtain the speaker label corresponding to each speech segment, and adding the speaker label corresponding to each speech segment to the corresponding text segment in the initial dialogue text to obtain the target dialogue text, the target dialogue text can be made more accurate.

[0075] In one or more alternative embodiments of the present invention, constructing the prompt word based on the application scenario of the target meeting and the target dialogue text includes: Obtain the meeting data of the target meeting, and based on the meeting data, determine the application scenario of the target meeting, where the meeting data includes meeting attributes and / or meeting content materials; According to the application scenario of the target meeting, determine the meeting minutes objective, where the meeting minutes objective represents the standard that the generated meeting minutes should achieve; Construct a prompt based on the meeting minutes objective and the objective dialogue text.

[0076] In practical applications, the meeting data of the target meeting can be obtained first, and then the meeting data can be analyzed to determine the application scenario of the target meeting. Then, based on the application scenario query, the corresponding meeting minutes objective can be selected. Further, a prompt for guiding the meeting minutes generation large model can be constructed based on the meeting minutes objective and the objective dialogue text.

[0077] In the embodiments of the present invention, by determining the application scenario of the meeting through the meeting data, and then determining the meeting minutes objective, and constructing a prompt based on the meeting minutes objective and the objective dialogue text, when the meeting minutes generation large model generates meeting minutes based on the prompt, it can be more targeted, which is beneficial to the flexible setting of different meetings and can be applied to different requirements for meeting minutes generation of different meetings.

[0078] In one or more alternative embodiments of the present invention, determining the application scenario of the target meeting based on the meeting data includes: Analyze the meeting attributes in the meeting data to determine the application scenario of the target meeting, where the meeting attributes include at least one of meeting type, meeting participants, meeting importance, and meeting tasks. In practical applications, meetings of different types, meetings with different parameters of people (such as different leaders supporting the meeting), meetings of different importance levels, and meetings of different people will all affect the application scenario of the meeting, and thus lead to different meeting minutes.

[0079] Exemplarily, for periodic meetings (such as daily meetings, weekly meetings, monthly meetings, and annual meetings), breakfast meetings, individual meetings, urgent and important meetings, meetings held by leaders of different levels, and group task meetings, etc., their corresponding meeting minutes objectives may be different. For example, for a daily meeting, the meeting minutes objective can be to generate meeting minutes in the form of a diary; for another example, for an urgent and important meeting, the meeting minutes objective can be to generate meeting minutes in the form of a notice.

[0080] Therefore, the attribute characteristics of at least one of the meeting type, meeting participants, meeting importance, and meeting tasks of the target meeting can be analyzed to determine the application scenario of the target meeting, and then the meeting minutes objective can be determined based on the application scenario.

[0081] In one or more alternative embodiments of the present invention, the elements of the prompt at least include role, task, background, and output format; Constructing the prompt based on the meeting minutes objective and the objective dialogue text includes: Based on the meeting minutes objective, determine the role, background, and output format of the prompt word; Based on the meeting minutes objective and the target dialogue text, determine the task of the prompt word.

[0082] In practical applications, the role of the prompt word can be determined based on the meeting minutes objective, such as the speaker and the expert; the background (i.e., context) of the prompt word can be determined based on the meeting minutes objective to help the large model for generating meeting minutes understand the task; the output format of the prompt word can be determined based on the meeting minutes objective, such as extracting the meeting minutes format to be generated from the meeting minutes objective and using this meeting minutes format as the output format prompted.

[0083] In addition, the task of the prompt word can also be determined based on the meeting minutes objective and the target dialogue text. Since the meeting minutes objective is to generate meeting minutes, combining this objective with the target dialogue text results in the task of generating meeting minutes based on the dialogue text, which is used as the task of the prompt word.

[0084] In the embodiments of the present invention, by constructing the prompt word in a fine-grained manner from four aspects: role, task, background, and output format, the comprehensiveness of the prompt word can be improved, which can help the large model for generating meeting minutes generate meeting minutes faster and more accurately.

[0085] In one or more alternative embodiments of the present invention, the determining the task of the prompt word based on the meeting minutes objective and the target dialogue text includes: Obtain the associated content of the audio-visual file, where the associated content at least includes relevant meeting materials; Based on the meeting minutes objective, the target dialogue text, and the associated content, determine the task of the prompt word.

[0086] Specifically, the relevant meeting materials may include paper documents used in the meeting, presentation slides used in the meeting, etc.

[0087] In practical applications, when the relevant materials of the target meeting only contain audio-visual files, the output format (target format) can be determined according to the meeting minutes objective, and then the task of "generating meeting minutes in the target format based on the target dialogue text" is generated. In this way, the speed of generating meeting minutes is guaranteed to a certain extent.

[0088] When the relevant materials of the target meeting also contain the associated content of the audio-visual file, the output format (target format) can be determined according to the meeting minutes objective, and then the task of "generating meeting minutes in the target format based on the target dialogue text and the associated content" is generated. In this way, the integrity and accuracy of the meeting minutes are guaranteed to a certain extent.

[0089] In one or more alternative embodiments of the present invention, the task of determining the prompt word based on the meeting minutes objective, the target dialogue text, and the associated content includes: Search for the historical meeting materials associated with the relevant meeting materials from a pre-constructed meeting database, and determine the task of the prompt word based on the meeting minutes objective, the target dialogue text, the associated content, and the historical meeting materials.

[0090] Specifically, the meeting database is the knowledge base.

[0091] In practical applications, the target meeting may be related to historical meetings. For example, a part of the content of this meeting is used to solve the remaining tasks of the previous meeting. At this time, the historical meeting materials associated with the relevant meeting materials can be searched based on the relevant meeting materials. Further, the output format (target format) is determined according to the meeting minutes objective, and then the task of "generating the meeting minutes in the target format based on the target dialogue text, the associated content, and the historical meeting materials" is generated. In this way, the integrity and accuracy of the meeting minutes are greatly guaranteed.

[0092] In the embodiments of the present invention, the output of the adjustment model is adjusted based on the Retrieval-augmented Generation (RAG) method. Among them, RAG mainly uses meeting materials to establish a knowledge base, quickly and accurately finds the knowledge related to the user's request through the knowledge base, and combines the retrieved knowledge with the user's request content to generate a high-quality answer.

[0093] In one or more alternative embodiments of the present invention, the method further includes: adding the relevant meeting materials to the meeting database, and updating the index of the meeting database based on the position information of the relevant meeting materials in the meeting database.

[0094] In practical applications, since the target meeting may also be related to the next meeting or future meetings to be held, the relevant meeting materials of the target meeting can be added to the meeting database. Since the content in the meeting database has changed, the index of the meeting database also needs to be updated, that is, the position information of the relevant meeting materials in the meeting database is generated as a sub-index and added to the index of the meeting database for quick and accurate query in the next use.

[0095] In one or more alternative embodiments of the present invention, determining the application scenario of the target meeting based on the meeting data includes: Analyze the meeting content materials in the meeting data to determine the application scenario of the target meeting, where the meeting content materials include at least one of the audio-visual file, the target dialogue text, relevant meeting materials, and other voice text files.

[0096] In practical applications, the application scenario can be determined not only based on the meeting attributes, that is, based on at least one of the meeting type, meeting participants, meeting importance, meeting tasks, and meeting material data, etc., to confirm the application scenario of the target meeting, but also based on the meeting content materials of the target meeting. That is, analyze the data characteristics of at least one of the audio-visual file of the target meeting, the target dialogue text, relevant meeting materials, and other voice text files, so as to determine the application scenario of the target meeting. In this way, the accuracy of identifying the application scenario can be guaranteed, and further, the accuracy of the meeting minutes target can be improved, thus improving the accuracy of the target meeting minutes.

[0097] In one or more alternative embodiments of the present invention, before performing voice activity detection processing, text conversion processing, and speaker recognition processing on the audio-visual file to obtain the target dialogue text, it further includes: Preprocess the audio signal in the audio-visual file, and the preprocessing includes at least one of denoising processing, accent recognition and annotation processing, and language recognition and annotation processing.

[0098] To ensure the speed and accuracy of voice activity detection processing, text conversion processing, and speaker recognition processing on the audio-visual file: the audio signal in the audio-visual file can be denoised, for example, using a denoising model to remove the voice noise of making a call; the audio signal in the audio-visual file can be subjected to accent recognition and annotation processing, for example, if the meeting speaker has an accent, the voice part of the meeting speaker can be subjected to accent recognition and annotation; the audio signal in the audio-visual file can be subjected to language recognition and annotation processing, for example, if multiple people in the meeting communicate in different languages (such as Chinese, English, German, etc.), the voice parts of different languages can be subjected to language recognition and annotation.

[0099] In one or more alternative embodiments of the present invention, the method further includes: updating the meeting minutes generation large model according to the audio-visual file and the meeting minutes, and / or updating the voice activity detection model, automatic speech recognition, and speaker recognition model according to the audio-visual file and the target dialogue text. In this way, it is possible to use audio or video and perform model iteration on the basis of each meeting to predict orders in a rolling manner.

[0100] In addition, the models involved in the present invention can be processed based on semantic understanding and pre-training, fine-tuning techniques of the attention mechanism.

[0101] Exemplarily, the attention mechanism enables the model to automatically focus on different parts of the text, dynamically allocate weights according to the context, and assign higher weights to the parts where key information is located, thereby better understanding the text semantics and capturing key information. Transformer is a commonly used architecture for large language models. When processing text, the self-attention mechanism in it calculates the degree of association between each word and other words to determine the importance weight of each word.

[0102] The meeting minutes generation large model is pre-trained on a large-scale text corpus and learns rich language knowledge and general semantic representations. When dealing with specific key information extraction tasks, a small amount of labeled data can be used for fine-tuning to enable the model to better adapt to the specific task requirements and thus more accurately extract key information. During the pre-training process, the model learns knowledge such as word co-occurrence relationships, grammatical structures, and semantic roles, and fine-tuning further optimizes the model's performance on specific tasks.

[0103] Exemplarily, refer to Figure 2 , Figure 2 FIG.

[0104] Next, in conjunction with Figure 3 , the meeting minutes generation method provided by the present invention will be further described. Refer to Figure 3 , Figure 3 FIG.

[0105] Step 1. Construction of the audio recording to text workflow.

[0106] (1) Input audio or video file: The application receives the audio or video file uploaded by the user, that is, the meeting audio and video.

[0107] (2) Voice activity detection: Perform voice activity detection on the audio and video data to remove noise and silent segments.

[0108] (3) Perform speaker recognition / splitting on the audio and video data.

[0109] (4) Convert the detected speech segments into text to generate a structured dialogue text file.

[0110] Step 2. The meeting minutes generation large model generates meeting minutes based on the text content.

[0111] Take the target dialogue text as the input of the meeting minutes generation large model. If there are relevant meeting materials, select the retrieval algorithm and tool according to the data characteristics and application scenarios (meeting minutes target), and at the same time create an index and optimize the established knowledge base document. That is, for the knowledge base such as relevant meeting materials, then construct prompt words, and input other speech text data, target dialogue text and prompt words into the meeting minutes generation large model to obtain the meeting minutes. Among them, the four core elements of constructing the prompt word engineering are: persona / role (Persona), task (Task), context (Context) and format (Format).

[0112] Exemplarily, obtain the audio or video from the meeting system. If there are relevant meeting materials, such as meeting pdf and other materials, also include them for parsing: First, process the audio or video, that is, convert the audio or video into text format. First, consider preprocessing the input speech signal to extract its acoustic features; through voice activity detection, remove the non-speech part and establish the start and end positions of the speech. Finally, through speaker recognition, first train different speaker models to obtain different speaker models. In the recognition stage, calculate the similarity score and judge the speaker according to the score. Second, process the text data converted from the audio or video, mainly based on the capabilities of the meeting minutes generation large model, rather than traditional NLP techniques. Use the method of prompt engineering (PE) and retrieval-augmented generation (RAG) to adjust the model output. Among them, RAG mainly uses the meeting materials to establish a knowledge base, quickly and accurately find the knowledge related to the user's request through the knowledge base, and combine the retrieved knowledge with the user's request content to generate a high-quality answer. Prompt engineering is used to guide the behavior of the meeting minutes generation large model and improve the generation quality.

[0113] The meeting minutes generation method provided by the present invention starts directly from the audio or video, which is more in line with the end-to-end algorithm idea. Through the meeting minutes generation large model technology, it solves the problem of low efficiency of the meeting minutes generated by manual recording, and saves more time; uses the method of prompt engineering and retrieval generation to optimize the generation result of the meeting minutes generation large model, which can further improve the accuracy of the generated meeting minutes; uses the meeting minutes generation large model generation technology instead of the traditional neuro-linguistic programming (NLP) technology, eliminating the construction of feature engineering and having better performance.

[0114] Next, the meeting minutes generation device provided by the present invention will be described. The meeting minutes generation device described below can be mutually referred to the meeting minutes generation method described above.

[0115] Figure 4 This is a schematic structural diagram of the meeting minutes generation device provided by the present invention. As Figure 4 shown, the device includes: An acquisition module 401, configured to acquire the audio-visual file of the target meeting, where the audio-visual file includes an audio file and / or a video file; A processing module 402, configured to perform voice activity detection processing, text conversion processing, and speaker recognition processing on the audio-visual file to obtain a target dialogue text; A construction module 403, configured to construct a prompt word based on the application scenario of the target meeting and the target dialogue text; A generation module 404, configured to input the prompt word into a meeting minutes generation large model to generate the meeting minutes corresponding to the target meeting.

[0116] Optionally, the processing module 402 is specifically configured to: Detect the voice activity part of the audio-visual file, generate timestamp information, and mark at least one voice segment in the audio-visual file based on the timestamp information; Perform text conversion processing on each voice segment to obtain a text fragment corresponding to each voice segment, merge the text fragments to generate an initial dialogue text; and perform speaker recognition processing on each voice segment to obtain a speaker label corresponding to each voice segment; Based on the speaker labels corresponding to each voice segment, add speaker labels to the corresponding text fragments in the initial dialogue text to obtain a target dialogue text.

[0117] Optionally, the construction module 403 is specifically configured to: Acquire the meeting data of the target meeting, and based on the meeting data, determine the application scenario of the target meeting, where the meeting data includes meeting attributes and / or meeting content materials; Determine the meeting minutes target according to the application scenario of the target meeting, where the meeting minutes target represents the standard to be achieved by the generated meeting minutes; Construct a prompt word based on the meeting minutes target and the target dialogue text.

[0118] Optionally, the elements of the prompt word at least include role, task, background, and output format; The construction module 403 is specifically configured to: Based on the meeting minutes target, determine the role, background, and output format of the prompt word; [[ID=A]]Based on the meeting minutes target and the target dialogue text, determine the task of the prompt word.

[0119] Optionally, the building block 403 is specifically configured to: Obtain the associated content of the audio-visual file, where the associated content includes at least relevant meeting materials; Search for historical meeting materials associated with the relevant meeting materials in a pre-constructed meeting database, and determine the task of the prompt word based on the meeting minutes objective, the target dialogue text, the associated content, and the historical meeting materials.

[0120] Optionally, the device further includes an update module, configured to: Add the relevant meeting materials to the meeting database, and update the index of the meeting database based on the location information of the relevant meeting materials in the meeting database.

[0121] Optionally, the device further includes a determination module, configured to: Analyze the meeting content materials of the target meeting to determine the application scenario of the target meeting, where the meeting content materials include at least one of the audio-visual file, the target dialogue text, relevant meeting materials, and other speech text files.

[0122] Optionally, the device further includes a preprocessing module, configured to: Preprocess the audio signal in the audio-visual file, where the preprocessing includes at least one of denoising processing, accent recognition and annotation processing, and language recognition and annotation processing.

[0123] Figure 5 Illustrates a schematic physical structure diagram of an electronic device, as Figure 5 shown. The electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540. Among them, the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logical instructions in the memory 530 to execute a meeting minutes generation method, which includes: obtaining an audio-visual file of a target meeting, where the audio-visual file includes an audio file and / or a video file; performing voice activity detection processing, text conversion processing, and speaker recognition processing on the audio-visual file to obtain a target dialogue text; constructing a prompt word based on the application scenario of the target meeting and the target dialogue text; and inputting the prompt word into a meeting minutes generation large model to generate the meeting minutes corresponding to the target meeting.

[0124] In addition, when the logical instructions in the above-mentioned memory 530 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0125] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the meeting minutes generation method provided by the above-mentioned various methods. The method includes: obtaining an audio-video file of a target meeting, where the audio-video file includes an audio file and / or a video file; performing voice activity detection processing, text conversion processing, and speaker recognition processing on the audio-video file to obtain a target dialogue text; constructing a prompt word based on the application scenario of the target meeting and the target dialogue text; inputting the prompt word into a meeting minutes generation large model to generate the meeting minutes corresponding to the target meeting.

[0126] On another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the meeting minutes generation method provided by the above-mentioned various methods. The method includes: obtaining an audio-video file of a target meeting, where the audio-video file includes an audio file and / or a video file; performing voice activity detection processing, text conversion processing, and speaker recognition processing on the audio-video file to obtain a target dialogue text; constructing a prompt word based on the application scenario of the target meeting and the target dialogue text; inputting the prompt word into a meeting minutes generation large model to generate the meeting minutes corresponding to the target meeting.

[0127] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative work.

[0128] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments or equivalently replace some of the technical features. However, these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating meeting minutes, characterized in that, Including: Obtain the audio-visual file of the target meeting, where the audio-visual file includes an audio file and / or a video file; Perform voice activity detection processing, text conversion processing, and speaker recognition processing on the audio-visual file to obtain the target dialogue text; Construct a prompt based on the application scenario of the target meeting and the target dialogue text; Input the prompt into the meeting minutes generation large model to generate the meeting minutes corresponding to the target meeting.

2. The method for generating a meeting minutes according to claim 1, wherein, The performing voice activity detection processing, text conversion processing, and speaker recognition processing on the audio-visual file to obtain the target dialogue text includes: Detect the voice activity part of the audio-visual file, generate timestamp information, and mark at least one voice segment in the audio-visual file based on the timestamp information; Perform text conversion processing on each voice segment to obtain the text fragment corresponding to each voice segment, merge the text fragments to generate an initial dialogue text; and perform speaker recognition processing on each voice segment to obtain the speaker label corresponding to each voice segment; Based on the speaker labels corresponding to each voice segment, add speaker labels to the corresponding text fragments in the initial dialogue text to obtain the target dialogue text.

3. The method for generating a meeting minutes according to claim 1, wherein The constructing a prompt based on the application scenario of the target meeting and the target dialogue text includes: Obtain the meeting data of the target meeting, and based on the meeting data, determine the application scenario of the target meeting, where the meeting data includes meeting attributes and / or meeting content materials; Determine the meeting minutes objective according to the application scenario of the target meeting, where the meeting minutes objective represents the standard to be achieved by the generated meeting minutes; Construct a prompt based on the meeting minutes objective and the target dialogue text.

4. The method for generating meeting minutes according to claim 3, wherein The elements of the prompt at least include role, task, background, and output format; The constructing a prompt based on the meeting minutes objective and the target dialogue text includes: Based on the meeting minutes objective, determine the role, background, and output format of the prompt; Based on the meeting minutes objective and the target dialogue text, determine the task of the prompt.

5. The method for generating a meeting minutes according to claim 4, wherein The determining the task of the prompt based on the meeting minutes objective and the target dialogue text includes: Obtain the associated content of the audio-visual file, where the associated content at least includes relevant meeting materials; Search for the historical meeting materials associated with the relevant meeting materials in the pre-constructed meeting database, and based on the meeting minutes objective, the target dialogue text, the associated content, and the historical meeting materials, determine the task of the prompt.

6. The method for generating meeting minutes according to claim 5, wherein, Also including: Add the relevant meeting materials to the meeting database, and update the index of the meeting database based on the location information of the relevant meeting materials in the meeting database.

7. The method for generating a meeting minutes according to claim 3, wherein The determining the application scenario of the target meeting based on the meeting data includes: Analyze the meeting content materials in the meeting data to determine the application scenario of the target meeting, where the meeting content materials include at least one of the audio-visual file, the target dialogue text, relevant meeting materials, and other voice text files.

8. The method for generating a meeting minutes according to any one of claims 1-7, characterized in that, Before performing voice activity detection processing, text conversion processing, and speaker recognition processing on the audio-visual file to obtain the target dialogue text, it further includes: Preprocessing the audio signal in the audio-visual file, where the preprocessing includes at least one of denoising processing, accent recognition and annotation processing, and language recognition and annotation processing.

9. A meeting minutes generation device, characterized in that, It includes: An acquisition module configured to acquire an audio-visual file of a target meeting, where the audio-visual file includes an audio file and / or a video file; A processing module configured to perform voice activity detection processing, text conversion processing, and speaker recognition processing on the audio-visual file to obtain a target dialogue text; A construction module configured to construct a prompt word based on the application scenario of the target meeting and the target dialogue text; A generation module configured to input the prompt word into a meeting minutes generation large model to generate meeting minutes corresponding to the target meeting.

10. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the meeting minutes generation method according to any one of claims 1 to 8.

Images