Meeting recording method and device based on language large model, medium and equipment

Abstract

The invention discloses a conference recording method and device based on a language large model, a medium and equipment, and the method comprises the steps: carrying out the text preprocessing of original conference audio data, and obtaining conference text data; performing syntactic analysis on the conference text data to obtain a basic SRL structure; performing semantic analysis on the conference text data through a language large model to obtain a written rewriting result, and performing content reconstruction on the written rewriting result based on the basic SRL structure to obtain reconstructed text data; normalizing the reconstructed text data to obtain normalized text data; and filling the normalized text data into the structured slot template to generate a structured conference summary text. According to the method, the SRL model and the large language model are combined, functional complementation of the SRL model and the large language model is achieved, and the problems of information omission, unclear expression or logic jump, context ambiguity, understanding omission, unstable output structure and insufficient generation credibility are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a conference recording method and device based on a language large model, a medium and equipment. BACKGROUND

[0002] With the popularity of remote work, online meetings and multi-team collaboration, meeting minutes play an increasingly important role in enterprise management, project advancement and scientific research discussion. High-quality meeting minutes not only record key information and distribute to-do lists, but also serve as an important basis for subsequent decision-making and responsibility tracing. However, in the past, manually written conference content has the problems of low efficiency, strong subjectivity and inconsistent content, so it needs the support of automatic generation technology. In recent years, relying on the progress of natural language processing (NLP) and speech recognition (ASR), automatic conference recording has become a research hotspot in intelligent office.

[0003] In recent years, language large models (such as GPT-3.5 / 4, Claude, Gemini) have shown strong text understanding and generation capabilities. The existing method is to directly input the complete conference text into the large model, and prompt the large model to generate meeting minutes through prompt words, which has achieved good results.

[0004] However, language large models lack deep modeling of semantic structures at the sentence level and even the chapter level, and cannot accurately capture the "action-target-participant" relationship in multi-party dialogue. The existing large model directly generates conference minutes, which has the problems of missing structural information and shallow semantic understanding.

[0005] Therefore, there is an urgent need for a new conference recording method based on a language large model. SUMMARY

[0006] In view of the above problems, the present application is proposed in order to provide a conference recording method based on a language large model, device, medium and equipment which overcome the above problems or at least partially solve the above problems.

[0007] Other characteristics and advantages of the present application will become apparent from the following detailed description, or will be learned by practice of the present application.

[0008] According to a first aspect of the embodiments of the present application, a conference recording method based on a language large model is provided, which comprises: obtaining original conference audio data, performing text preprocessing on the original conference audio data to obtain conference text data; performing syntax analysis on the conference text data, identifying predicates in the conference text data, calling a SRL model to perform role labeling on each predicate to obtain a basic SRL structure; performing semantic analysis on the conference text data by a language large model, extracting semantic elements, performing written rewriting based on the semantic elements to obtain a written rewriting result, and performing content reconstruction on the written rewriting result based on the basic SRL structure to obtain reconstructed text data; performing normalization processing on the reconstructed text data to obtain normalized text data; obtaining a predefined structured slot template, filling the normalized text data into the structured slot template, and generating a structured conference minutes text.

[0009] In some embodiments of the present application, the text preprocessing on the original conference audio data to obtain conference text data includes: performing frame cutting processing on the original conference audio data to obtain a slice sequence; performing clustering on the slice sequence by a DBSCAN density clustering algorithm to obtain clustered segments; extracting speaker audio representations in the clustered segments by a voiceprint recognition model, and combining temporal relationships and similarities to perform identity labeling on speakers to obtain a set of speaker-labeled speech paragraphs; performing speech recognition processing on each set of speech paragraphs to generate corresponding speech text information; performing text normalization processing on the speech text information, and splicing the text-normalized speech text information according to temporal relationships to obtain the conference text data.

[0010] In some embodiments of the present application, the clustering of the slice sequence by the DBSCAN density clustering algorithm to obtain clustered segments includes: sorting the slice sequence according to temporal relationships, and calculating average signal intensity of a plurality of continuous slices in the sorted slice sequence by a sliding window, the average signal intensity being the average signal intensity of all slices in the sliding window; determining whether the average signal intensity is greater than a preset intensity threshold, and if the average signal intensity is greater than the preset intensity threshold, regarding all slices in the sliding window as a voice activity segment; collecting all voice activity segments in the slice sequence, and merging to obtain the clustered segments with timestamps.

[0011] In some embodiments of the present application, the speaker audio feature in the clustering segment is extracted by the voiceprint recognition model, and the speaker is labeled with an identity based on the time sequence relationship and the similarity, to obtain a set of speech paragraphs with speaker labels, which includes: The speaker audio feature in the clustering segment is extracted by the voiceprint recognition model, and the similarity of the speaker audio features of multiple continuous interval segments is calculated based on the time sequence relationship using a sliding window, and the multiple interval segments are located within the sliding window. It is confirmed whether the similarity is greater than a preset similarity threshold, and if the similarity is greater than the preset similarity threshold, the speaker audio features of multiple continuous interval segments are defined as the same speaker, and the speaker is labeled with an identity. After all the speakers have completed the identity labeling, the set of speech paragraphs with speaker labels is obtained.

[0012] In some embodiments of the present application, the normalization processing of the reconstructed text data to obtain normalized text data includes: The reconstructed text data is taken as the input of the SRL model, and the SRL normalization processing of the basic SRL structure is performed by the SRL model to obtain a normalized SRL structure. The normalized SRL structure and the reconstructed text data are jointly normalized by a language large model to obtain normalized text data.

[0013] In some embodiments of the present application, the pre-defined structured slot template is obtained, and the normalized text data is filled into the structured slot template, which includes: A pre-defined structured slot template is obtained, and the slots in the structured slot template include at least topics, speakers, key points, task arrangements, and meeting resolutions. Based on the normalized SRL structure, the structured information corresponding to the slots in the normalized text data is extracted and filled into the corresponding slots in the structured slot template.

[0014] In some embodiments of the present application, the structured meeting minutes text is generated, which includes: The topics filled into the slots are collected, and the language large model is used to classify and merge the topics according to the preset merging rules. The corresponding paragraph texts of the substantially identical topics in the normalized text data are confirmed according to the classification and merging results, and the corresponding structured text segments of the substantially identical topics are obtained by merging and reconstructing the corresponding paragraph texts of the substantially identical topics by the language large model. The structured text segments are converted into the structured meeting minutes text according to the preset meeting minutes template format by the language large model.

[0015] According to a second aspect of the embodiments of the present application, a conference recording device based on a language large model is provided, and the conference recording device based on the language large model comprises: An audio preprocessing module is configured to acquire original conference audio data, perform text preprocessing on the original conference audio data, and obtain conference text data. A structured processing module is configured to perform syntax analysis on the conference text data, identify predicates in the conference text data, call a SRL model to perform role labeling on each predicate, and obtain a basic SRL structure. A text reconstruction module is configured to perform semantic analysis on the conference text data by using a language large model, extract semantic elements, perform written rewriting based on the semantic elements, obtain a written rewriting result, perform content reconstruction on the written rewriting result based on the basic SRL structure, and obtain reconstructed text data. A text normalization module is configured to perform normalization processing on the reconstructed text data to obtain normalized text data. A conference minutes generation module is configured to acquire a predefined structured slot template, fill the normalized text data into the structured slot template, and generate structured conference minutes text.

[0016] According to a third aspect of the embodiments of the present application, a computer readable storage medium is provided, and the computer readable storage medium stores computer program instructions. The computer program instructions are loaded and executed by a processor to implement operations performed by the method according to any one of the above aspects.

[0017] According to a fourth aspect of the embodiments of the present application, an electronic device is provided, which comprises a processor and a memory. The memory stores computer program instructions executable by the processor. When the processor executes the computer program instructions, instructions of the method according to any one of the above aspects are implemented.

[0018] The technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: The conference recording method, device, medium and equipment based on a language large model provided by the embodiment of the present application, the method extracts a basic SRL structure by performing syntax analysis on the conference text data through an SRL model, and extracts semantic elements by performing semantic analysis on the conference text data through a language large model, the SRL model and the language large model are combined, the SRL model provides clear semantic slots, so that the minutes content has the characteristics of traceable source, controllable content and verifiable structure, avoids the problem that the generated sentence of the traditional language large model minutes is smooth but lacks factual support, makes the generated result natural and has structure traceability, realizes the complementation of the functions of the SRL model and the language large model, and solves the problems of information omission, unclear expression or logical jump, context ambiguity and omission understanding, unstable output structure and insufficient generation credibility.

[0019] The above description is only a summary of the technical solutions of the present application, in order to more clearly understand the technical means of the present application, the content of the specification can be implemented, and in order to make the above and other purposes, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described below. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] Figure 1 The flowchart of the conference recording method based on a language large model provided by the embodiment of the present application is shown in the figure. Figure 2 The principle structure diagram of the conference recording device based on a language large model provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0022] The exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0023] In the drawings, various structural schematic diagrams according to the embodiments of the present disclosure are shown. These drawings are not drawn to scale, in which certain details are enlarged for the purpose of clear expression, and certain details can be omitted. The shapes of various regions, layers shown in the drawings and their relative size, position relationship are only exemplary, and in actuality, there can be deviations due to manufacturing tolerances or technical limitations, and the skilled in the art can additionally design regions / layers with different shapes, sizes, relative positions according to actual needs.

[0024] In the context of the present disclosure, when one layer / element is referred to as being located "on" another layer / element, the layer / element can be directly on the other layer / element, or there can be an intervening layer / element therebetween. In addition, if one layer / element is located "on" another layer / element in one orientation, the layer / element can be located "under" the other layer / element when the orientation is reversed. In the context of the present disclosure, similar or identical components can be denoted by the same or similar reference numerals.

[0025] In order to better understand the above technical solutions, the above technical solutions will be described in detail below in combination with specific embodiments. It should be understood that the embodiments of the present disclosure and the specific features in the embodiments are detailed descriptions of the technical solutions of the present disclosure, rather than limitations of the technical solutions of the present disclosure. In the case of no conflict, the technical features in the embodiments of the present disclosure and the embodiments can be combined with each other.

[0026] Figure 1 is a flowchart of a conference recording method based on a language large model provided by an embodiment of the present disclosure, as shown in Figure 1 The conference recording method based on the language large model includes the following steps: S1, obtaining original conference audio data, performing text preprocessing on the original conference audio data to obtain conference text data; The original conference audio data is an audio file in a format such as wav or mp3. After obtaining the original conference audio data, the embodiment of the present disclosure converts it into conference text data through text preprocessing.

[0027] The text preprocessing of the original conference audio data to obtain conference text data includes: performing frame cutting processing on the original conference audio data to obtain a slice sequence; clustering the slice sequence through a DBSCAN density clustering algorithm to obtain a clustered segment; extracting speaker audio representation in the clustered segment through a voiceprint recognition model, and combining a time sequence relationship and a similarity to perform identity labeling on the speaker to obtain a speaker-labeled speech paragraph set; performing speech recognition processing on each speaker-labeled speech paragraph set to generate corresponding speech text information; performing text regularization processing on the speech text information, and splicing the speech text information after the text regularization processing according to the time sequence relationship to obtain the conference text data.

[0028] Specifically, the embodiment of the present application performs frame cutting processing on the original conference audio data, for example, frame cutting processing on the original conference audio data based on a time sequence relationship, with 25 milliseconds as a frame and 10 milliseconds as a step, extracts short-time energy and a zero-crossing rate of each slice (or frame) for judging whether the current slice contains valid speech signals. The short-time energy is an index for measuring signal strength in a frame and can reflect the presence or absence of sound. Its calculation method is to average the square of the amplitudes of all sampling points in the frame. The energy of a silent frame is close to zero, while a speech frame has obvious energy fluctuations, so the energy feature can be used as the first judgment basis for speech detection. The zero-crossing rate represents the number of times the signal crosses zero in a frame, which reflects the frequency composition of the sound. The zero-crossing rate of a silent signal is usually very low or irregular, while a speech signal (especially a clear sound) exhibits a higher and stable crossing rate. By comprehensively judging the trends of energy and zero-crossing rate, the embodiment of the present application can accurately detect continuous non-silent segments, i.e., potential speaking sections. The final obtained slice sequence is, for example, {start_time: 75ms, end_time: 100ms, signal: [01001011110]}.

[0029] After obtaining the slice sequence, speech segment clustering and sentence boundary recognition need to be implemented to obtain the time stamps before and after each sentence. Then, the embodiment of the present application clusters the slice sequence by using a DBSCAN density clustering algorithm to obtain clustered segments, including: sorting the slice sequence according to a time sequence relationship, calculating the average signal amount intensity of a plurality of continuous slices in the sorted slice sequence by using a sliding window, the average signal amount intensity being the average signal amount intensity of all slices in the sliding window; confirming whether the average signal amount intensity is greater than a preset intensity threshold, if the average signal amount intensity is greater than the preset intensity threshold, taking all slices in the sliding window as a speech active segment, and if the average signal amount intensity is not greater than the preset intensity threshold, taking all slices in the sliding window as a silent segment; collecting all speech active segments in the slice sequence, and merging to obtain the clustered segments with time stamps. The time stamps of the clustered segments are the time stamps of the starting point and the ending point, and the finally obtained clustered segments are, for example, {start_time: 3.5s, end_time: 8.25s, voice_data: split_wav_data}.

[0030] After the sentence-level segment construction is completed, the embodiment of the present application further extracts the voiceprint features of each speech segment on the time axis, performs speaker recognition and clustering, the purpose is to identify the acoustic feature differences of the participants in the meeting, and attribute their speech content to a specific "speaker", so as to realize speaker separation. Specifically, the embodiment of the present application extracts the speaker audio representation in the clustering segment through the voiceprint recognition model, and combines the time sequence relationship and the similarity to label the identity of the speaker, to obtain a speech paragraph set with speaker labels, including: extracting the speaker audio representation in the clustering segment through the voiceprint recognition model, calculating the similarity of the speaker audio representation of a plurality of continuous interval segments based on the time sequence relationship using a sliding window, the plurality of interval segments are located within the sliding window range; confirming whether the similarity is greater than a preset similarity threshold, if the similarity is greater than the preset similarity threshold, defining the speaker audio representation of the plurality of continuous interval segments as the same speaker, and labeling the identity of the speaker; after the identity of all speakers is labeled, the speech paragraph set with speaker labels is obtained.

[0031] The voiceprint recognition model may be, for example, an ECAPA-TDNN model, which can extract the speaker audio representation (speaker embedding representation) of each clustering segment (split_way_data), and cluster based on similarity on the time axis, combine the context stability and continuity, label the identity of the speaker, and obtain the speech paragraph set with speaker labels, for example, {start_time: 3.5s, end_time: 20.5s, speaker: speaker1, voice_data: [split_wav_data1, split_wav_data2, split_wav_data3, …]}.

[0032] The embodiment of the application inputs each speech paragraph with a clear timestamp in the speaker-labeled speech paragraph set into an automatic speech recognition (ASR) module for speech recognition processing to generate corresponding speech text information, for example, input: {start_time: 3.5s, end_time: 20.5s, speaker: speaker1, voice_data: [split_wav_data1, split_wav_data2, split_wav_data3, …]} and output of the speech recognition module after speech recognition processing: {start_time: 3.5s, end_time: 20.5s, speaker: speaker1, text: [Well, I think we need to postpone this project, that is, the resources may not be fully prepared yet]}.

[0033] After obtaining the speech text information, in order to improve the readability of the text information, it is necessary to remove colloquial expressions, tone fillers and non-semantic language fragments in the speech text information that do not have semantic value, such as “ah”, “um”, “just” and the like; the embodiment of the application pre-constructs a high-frequency tone word dictionary based on corpus statistics and language specifications, which covers single words, phrases and multi-word combination structures, including words and vocabularies of this type such as “ah”, “um”, “just” and the like. The embodiment of the application adopts a regular rule and a sliding window mechanism to accurately match and identify the speech text information based on the high-frequency tone word dictionary, and to batch remove high-frequency tone word chains with loose structures, for example, the obtained conference text data is {start_time: 3.5s, end_time: 20.5s, speaker: speaker1, text: [Well, I think we need to postpone this project, that is, the resources may not be fully prepared yet]}.

[0034] S2, performing syntax analysis on the conference text data to identify predicates in the conference text data, calling a SRL model to perform role labeling on each predicate to obtain a basic SRL structure; The embodiment of the present application performs syntactic analysis on the conference text data through a natural language processing tool such as LTP, Stanford CoreNLP or HanLP, and extracts the subject-verb, verb-object, core verb and other dependency relationships in each sentence to identify the predicate in the conference text data; wherein the predicate verb is matched based on a high-frequency verb predicate dictionary using a regular matching method to enhance the model's ability to identify non-common verbs; the high-frequency verb predicate dictionary is obtained by combining artificial construction and corpus statistics, and the high-frequency verb predicate dictionary covers verb entries commonly used in contexts such as policy, business, administrative instructions, etc. (such as "promote", "strengthen", "implement", "complete", "establish", "require", "provide", etc.); for example, the identified predicate is predicate_verb: [delay, prepare, think].

[0035] The SRL model may be, for example, an SRL model based on BERT, ERNIE, LTP or AllenNLP, which performs role labeling on each predicate to identify the agent (ARG0), the recipient (ARG1), the time, the place, the manner and other additional information (ARGM series) of the action. The output of this stage is a preliminary semantic structure, i.e. the basic SRL structure (or event unit) of "predicate + role + other", for example, the basic SRL structure obtained on the basis of the above embodiment is: [{"predicate": "delay", "roles": {"ARG0": "we", "ARG1": "this project", "ARGM-MOD": "possible"}}, {"predicate": "prepare", "roles": {"ARG0": "resources", "ARGM-NEG": "not", "ARGM-ADV": "completely"}}, …].

[0036] S3, performing semantic analysis on the conference text data through a language large model, extracting semantic elements, performing written rewriting based on the semantic elements to obtain a written rewriting result, and performing content reconstruction on the written rewriting result based on the basic SRL structure to obtain reconstructed text data; The language large model (LLM) of the embodiment of the present application learns the statistical rules and semantic structure of language by training on a large amount of text, thereby having the ability to generate coherent and logical text. Through a preset prompt, the user can clearly convey the task target and expected output form to the language large model, and the language large model automatically generates a matching answer based on its understanding of the language context.

[0037] The embodiment of the present application performs semantic analysis on the conference text data through a language large model, extracts semantic elements, and performs written rewriting on the expression form of the conference text data, thereby improving semantic clarity and structural standardization. The semantic elements may be, for example, core topics, event stems, and pragmatic intentions. Specifically, the embodiment of the present application constructs a prompt containing task instructions, guides the language large model to identify the core topics, event stems, and pragmatic intentions in the paragraphs of the conference text data, and requires output of structured elements and written rewriting results.

[0038] The embodiment of the present application requires semantic analysis on the conference text data, extracts semantic elements, and performs written rewriting based on the semantic elements to obtain a written rewriting result. A prompt example is: “Please perform semantic analysis on the following conference speech content, extract its core topic, main event description, and speaker intention, and rewrite it into a formal and clear written sentence. The output format is: 1, core topic; 2, main event; 3, speaker intention; 4, rewritten sentence. The paragraph content is as follows: ‘That, we now have a project, there seems to be a problem with resources, um, I think the start time can be moved back? Or wait first, wait until the preparation is more sufficient before starting’ ”.

[0039] The language large model outputs: Core topic: project start plan and resource preparation; Main event: due to insufficient resources, propose to postpone the project start time; Speaker intention: tactfully express the postponement suggestion and hope to wait for more sufficient preparation conditions; Rewritten sentence: “Due to the current resources not being fully in place, it is suggested to appropriately postpone the start time of the project to ensure that the preparation work is more sufficient”.

[0040] The embodiment of the present application guides the language large model to perform content reconstruction on the written rewriting result based on the basic SRL structure through a pre-defined content reconstruction prompt, generates a natural language paragraph with clear stem structure and complete semantic expression, and ensures expression accuracy and structural stability.

[0041] For example, input the following information into the language large model for content restructuring: [{"predicate": "delay", "roles": {"ARG0": "we", "ARG1": "this project", "ARGM-MOD": "possibly"}}, {"predicate": "prepare", "roles": {"ARG0": "resources", "ARGM-NEG": "not", "ARGM-ADV": "completely"}}, …] and "Due to the current resources not being fully in place, it is suggested to appropriately delay the start time of the project to ensure more thorough preparation" and the speaker label as input information. The restructuring text data obtained is, for example: "Zhang San said that due to the current resources not being fully in place, it is suggested to appropriately delay the start time of the project to ensure more thorough preparation". By taking the basic SRL structure as the reference basis, together with the specific written rewriting result as the input to let the language large model perform content restructuring, the purpose of fusion is achieved; in the restructuring text data, not only is the accurate restoration of the sentence stem action ("delay", "prepare") completed, but also the explicit expression of the subject, the syntactic integrity, and the style transition from spoken language to written language are achieved. In addition, the speaker information is also explicitly included in the generated content, meeting the semantic consistency requirements of subsequent multi-speaker structure fusion.

[0042] S4, normalizing the restructuring text data to obtain normalized text data; The normalization processing of the restructuring text data to obtain normalized text data includes: taking the restructuring text data as the input of the SRL model, performing SRL normalization processing on the basic SRL structure through the SRL model to obtain a normalized SRL structure; and performing joint normalization processing on the normalized SRL structure and the restructuring text data through a language large model to obtain normalized text data.

[0043] Since the obtained basic SRL structure may have problems of missing roles, inconsistent naming or unclear semantic boundaries; the main reason is caused by the colloquial expression of the original text, which may omit the subject and use different pronoun forms to express the same semantic entity, therefore, the embodiment of the application needs to perform SRL normalization processing on the basic SRL structure to supplement the omitted roles, unify the role label style (such as ARGM-TMP for "time"), and clean up redundant components, and the reconstructed text data obtained through the foregoing steps is taken as the input of the SRL model to perform SRL normalization processing, further improving the accuracy and integrity of the structure, and the normalized SRL structure has more clear subject-predicate-object information; for example, the obtained normalized SRL structure is [{"predicate": "delay", "roles": {"ARG0": "Zhang San", "ARG1": "this project", "ARGM-MOD": "suggestion"}}, {"predicate": "prepare", "roles": {"ARG0": "current resources", "ARGM-NEG": "not yet", "ARGM-ADV": "in place"}}, …].

[0044] After obtaining the reconstructed text data, since the content reconstruction and the foregoing SRL normalization processing focus on the expression of units at the sentence level or the paragraph level, the pragmatic consistency, the term uniformity and the structural coherence of the whole text are not systematically fused, therefore, in order to ensure the uniformity of the language style of the whole text, the embodiment of the application can also perform unified processing and written expression generation of the paragraph text at the whole text level based on the joint input of the previous multiple reconstructed paragraphs and multiple normalized SRL structures after the SRL normalization processing, that is, the normalized text data is obtained by performing joint normalization processing on the normalized SRL structure and the reconstructed text data through a language large model, the obtained normalized text data only fuses the reconstructed text data of the natural language content, and further utilizes the normalized SRL structure (a version after the naming normalization and the term standardization) to integrate multiple event semantic units and multiple text paragraphs across paragraphs, so as to ensure the uniformity of the language style, the coherence of the logical structure and the completeness of the information expression of the whole text.

[0045] The prompt for performing joint normalization processing on the normalized SRL structure and the reconstructed text data defined in the embodiment of the application is, for example: "Please generate a complete, style-consistent and written standard conference minutes text according to the following multiple semantic structures and content paragraphs, and ensure that the subject is clear, the terms are uniform, the reference is clear and the logic is coherent".

[0046] The obtained normalized text data is, for example: "During the meeting, Zhang San proposed that due to the fact that the key resources required by the current project have not been fully prepared, there are still coordination obstacles in the execution level of the relevant team, therefore it is suggested to postpone the original start plan to ensure that the preparation work is more sufficient and the execution process is more smooth. Li Si added that although the pre-project process has been basically completed, there are still omissions in the arrangement of market research data, and the feedback from some partners has not been collected. If the subsequent process is carried out recklessly, it may affect the overall project rhythm. Therefore, he suggested that the official promotion instruction should be issued after the resources are prepared and the data are verified."

[0047] S5, obtaining a predefined structured slot template, filling the normalized text data into the structured slot template to generate a structured meeting minutes text.

[0048] The structured slot template is a set of standardized minutes slot systems constructed based on human predefinition, and the standardized minutes slot system usually includes a plurality of slots, each slot corresponding to a semantic element in the normalized text data. The embodiment of the application provides structural support for the generation of subsequent meeting minutes text by extracting semantic elements in the normalized text data and filling them into the slots.

[0049] In the embodiment of the application, obtaining a predefined structured slot template includes obtaining a predefined structured slot template, the slots in the structured slot template including at least topic, speaker, key point, action item and meeting decision; based on the SRL structure, extracting structured information corresponding to the slots in the normalized text data, and filling the structured information into the corresponding slots in the structured slot template.

[0050] In the embodiment of the application, the slots are defined in the form of key fields; for example, the topic (Topic) represents the topic name or topic number, usually corresponding to the meeting process or agenda item; for example, the speaker (Speaker) represents the speaker's identity information, which can be name, position or organization unit, etc.; for example, the key point (KeyPoint) represents the key point or opinion statement, which is usually neutral or factual statement; for example, the action item (ActionItem) represents the explicit task arrangement or to-do list, etc., with execution instruction or object; for example, the meeting decision (Decision) represents the conclusion, decision or consensus formed by the meeting, etc. In other embodiments of the application, the key fields corresponding to the slots can also be changed to other semantic elements or the number of slots can be increased or decreased according to actual application requirements.

[0051] For example, when the input normalized text data is "Regarding the problem of customer satisfaction decline in the second quarter, Wang of the marketing department suggests strengthening the customer follow-up mechanism, and submitting an improvement plan within the next month, which is responsible for Li", the structured information corresponding to the slot in the normalized text data is extracted, for example: "Topic": "second quarter customer satisfaction decline", "Speaker": "Zhang San", "KeyPoint": "Wang (marketing department) suggests strengthening the customer follow-up mechanism and submitting an improvement plan", "ActionItem": "Li is responsible for submitting an improvement plan within the next month", "Decision": "The meeting decides to adopt the plan" }.

[0052] The structured meeting minutes text generated by the embodiment of the application includes: collecting the topics filled into the slot, classifying and merging the topics according to the preset merging rule by using a language large model, confirming the paragraph texts corresponding to the substantially same topics in the normalized text data according to the classification and merging results, merging and reconstructing the paragraph texts corresponding to the substantially same topics by using a language large model, and obtaining the structured text segments corresponding to each topic; and converting the structured text segments into the structured meeting minutes text according to the preset meeting minutes template format by using a language large model.

[0053] After completing the semantic element filling of the slot, the embodiment of the application collects all the topics filled into the slot, classifies and merges the topics according to the preset merging rule (which can be artificially defined or open source) by using a language large model, confirms the paragraph texts corresponding to the substantially same topics in the normalized text data according to the classification and merging results, classifies the substantially same topics into the same label by using a language large model, and re-groups and arranges all the paragraph texts according to the labels corresponding to the normalized topics, to obtain the structured text segments corresponding to each topic, so as to ensure that the related speeches under the same topic are uniformly archived, and facilitate the generation and expression of subsequent structured minutes.

[0054] The embodiment of the application converts the structured text segments into the structured meeting minutes text according to the preset meeting minutes template format by using a language large model, so as to convert the structured meeting information into natural language expression, and organizes and outputs the content according to the preset meeting minutes template format.

[0055] For example, the prompt for converting the structured text segments into the structured meeting minutes text is "merge according to the meeting minutes template to generate a complete meeting minutes"; The input is a plurality of structured text segments, including: Structured text segment 1 { "Topic": "Customer satisfaction decline", "Speaker": "Mr. Wang (Marketing Department)", "KeyPoint": "Customer return visit process is not clear, customer feedback cannot be closed loop", "ActionItem": "Suggest redesigning the customer return visit process", "Decision": "The meeting agrees in principle, and needs to develop specific improvement plan" }, Structured text segment 2 { "Topic": "Customer satisfaction decline", "Speaker": "Mr. Liu (Customer Service Department)", "KeyPoint": "Customer service response timeliness is unstable, affecting satisfaction score", "ActionItem": "Plan to introduce automatic work order distribution mechanism", "Decision": "The meeting agrees to pilot optimization process by the customer service department" }。

[0056] The preset meeting minutes template format is, for example, [event name: xxx, specific event: xxx, problem item: xxx, improvement measures: xxx, follow-up person: xxx].

[0057] The resulting structured meeting minutes text is, for example: [Event name: Customer satisfaction decline problem analysis; Specific event: In recent meetings, the marketing department and the customer service department analyzed and fed back on the relevant situation of customer satisfaction decline; Problem item: On the one hand, the customer return visit process has design defects, and customer feedback cannot form a closed loop; on the other hand, the customer service response timeliness is unstable, leading to fluctuation in satisfaction score; Improvement measures: Mr. Wang suggests redesigning the customer return visit process to ensure that customer opinions are timely tracked and fed back; Mr. Liu proposes to introduce an automatic work order distribution mechanism to improve customer service processing efficiency; Follow-up person: The market department is responsible for drafting the customer follow-up process improvement plan, and the customer service department will start the work order distribution mechanism pilot optimization. The two schemes will be submitted to the next meeting for progress report.

[0058] Compared with the prior art, the conference recording method based on a language large model according to the embodiment of the present application extracts a basic SRL structure by performing syntactic analysis on the conference text data through an SRL model, and extracts semantic elements by performing semantic analysis on the conference text data through a language large model. The SRL model and the language large model are combined, not only to be fused at the final result layer, but also to be alternately guided in the middle link of text reconstruction, gradually enhancing each other's effect, fusing information at two levels of micro (verb event layer) and macro (paragraph abstract layer), ensuring that the information granularity is detailed and the integrity is not lost. At the same time, the SRL model provides clear semantic slots, so that the minutes content has the characteristics of traceable source, controllable content and verifiable structure, avoiding the problem that the traditional language large model minutes generation sentence is smooth but lacks factual support, making the generated result natural and having structure traceability, realizing the complementation of the functions of the SRL model and the language large model, and solving the problems of information omission, unclear expression or logical jump, context ambiguity and omission understanding, unstable output structure and insufficient generation credibility.

[0059] On the basis of the above embodiment, as an implementation of the method shown in the above Figure 1 The present application provides an embodiment of a conference recording device based on a language large model. The device embodiment corresponds to the method embodiment shown in the above Figure 1 The device can be applied to various electronic devices, as shown in the above Figure 2 The conference recording device based on a language large model includes: An audio preprocessing module 100 is configured to obtain original conference audio data, and perform text preprocessing on the original conference audio data to obtain conference text data. A structured processing module 200 is configured to perform syntactic analysis on the conference text data, identify predicates in the conference text data, call an SRL model to perform role labeling on each predicate to obtain a basic SRL structure. A text reconstruction module 300 is configured to perform semantic analysis on the conference text data through a language large model to extract semantic elements, perform written rewriting based on the semantic elements to obtain a written rewriting result, and perform content reconstruction on the written rewriting result based on the basic SRL structure to obtain reconstructed text data. A text normalization module 400 is configured to perform normalization processing on the reconstructed text data to obtain normalized text data. The conference minutes generation module 500 is configured to acquire a predefined structured slot template, fill the normalized text data into the structured slot template, and generate structured conference minutes text.

[0060] The conference recording device based on the language large model provided by the embodiment of the present application can execute the conference recording method based on the language large model provided by the above-mentioned embodiment, and has the corresponding functional steps and beneficial effects of the conference recording method based on the language large model provided by the above-mentioned embodiment. For details, please refer to the embodiment of the conference recording method based on the language large model. The embodiment of the present application will not be repeated here.

[0061] The embodiment of the present application further provides an electronic device, which can include a processor and a memory. The processor and the memory can be connected through a bus or other means. The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. chips, or combinations of the above-mentioned chips. The memory, as a kind of non-transitory computer readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules of the conference recording method based on the language large model in the embodiment of the present application. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, realizes the conference recording method based on the language large model in the above-mentioned method embodiment.

[0062] The memory can include a program storage area and a data storage area, where the program storage area can store an operating system, application programs required by at least one function, and the like, and the data storage area can store data created by the processor and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. The one or more modules are stored in the memory and, when executed by the processor, perform the conference recording method based on the language large model as in the above method embodiments. The above electronic device specific details can correspond to the relevant description and effects of the corresponding description in the above method embodiments, which will not be described here. Those skilled in the art can understand that all or part of the processes in the above embodiments can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above embodiments. The storage medium can be a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc. The storage medium can also include a combination of the above types of memories.

[0063] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order to not obscure the understanding of this description.

[0064] Similarly, it is to be understood that the above description is only illustrative of the application and certain examples thereof, and is subject to the prior art. Numerous modifications and changes can be devised by those skilled in the art without departing from the true spirit and scope of the application. It is intended that the scope of the application be defined by the Scope of the Claims and their equivalents. It is therefore intended that the application not be limited to the particular treatment described herein, but that the application will include all implementations which fall within the scope of the claims.

Claims

1. A conference recording method based on a large language model, characterized in that: The meeting recording method based on the language large model includes: Acquire original conference audio data, perform text preprocessing on the original conference audio data, and obtain conference text data; Performing syntactic analysis on the conference text data to identify predicates in the conference text data, calling an SRL model to perform role labeling on each predicate to obtain a basic SRL structure; Performing semantic analysis on the conference text data using a large language model to extract semantic elements, performing written rewriting based on the semantic elements to obtain a written rewriting result, and reconstructing the content of the written rewriting result based on the basic SRL structure to obtain reconstructed text data; Normalizing the reconstructed text data to obtain normalized text data; A predefined structured slot template is obtained, and the normalized text data is filled into the structured slot template to generate a structured meeting minutes text.

2. The method for recording a meeting based on a large language model according to claim 1, characterized in that: The performing text preprocessing on the original conference audio data to obtain conference text data includes: Performing frame slicing on the original conference audio data to obtain a slice sequence; Clustering the slice sequence using the DBSCAN density clustering algorithm to obtain clustered segments; Extracting the speaker audio representation in the clustered segments through a voiceprint recognition model, and labeling the speaker's identity based on the temporal relationship and similarity, thereby obtaining a set of speech paragraphs with speaker labels; Performing speech recognition processing on each of the speech paragraph sets to generate corresponding speech text information; The speech text information is subjected to text regularization processing, and the speech text information after text regularization processing is spliced ​​according to a time sequence relationship to obtain the conference text data.

3. The method for recording a meeting based on a large language model according to claim 2, characterized in that: The clustering of the slice sequence by the DBSCAN density clustering algorithm to obtain cluster segments includes: Sorting the slice sequence according to a time sequence relationship, and using a sliding window to calculate the average signal strength of a plurality of consecutive slices in the sorted slice sequence, wherein the average signal strength refers to the average signal strength of all slices in the sliding window; Determining whether the average signal strength is greater than a preset strength threshold, and if the average signal strength is greater than the preset strength threshold, taking all slices in the sliding window as voice activity segments; All speech activity segments in the slice sequence are collected and merged to obtain the clustered segments with timestamps.

4. The method for recording a meeting based on a large language model according to claim 2, characterized in that: The speaker audio representation in the clustered segments is extracted by the voiceprint recognition model, and the speaker identity is labeled based on the temporal relationship and similarity. The obtained speech paragraph set with the speaker label includes: Extracting the speaker audio representation in the clustered segments using a voiceprint recognition model, and calculating the similarity of the speaker audio representation of multiple consecutive intervals using a sliding window based on a temporal relationship, wherein the multiple intervals are all within the sliding window; confirming whether the similarity is greater than a preset similarity threshold; if the similarity is greater than the preset similarity threshold, defining the speaker audio representations of multiple consecutive intervals as the same speaker, and labeling the speaker; After all speakers have completed identity labeling, a set of speech paragraphs with speaker labels is obtained.

5. The method for recording a meeting based on a large language model according to claim 1, characterized in that: The step of normalizing the reconstructed text data to obtain normalized text data comprises: Using the reconstructed text data as input to the SRL model, and performing SRL normalization processing on the basic SRL structure through the SRL model to obtain a normalized SRL structure; The standardized SRL structure and the reconstructed text data are jointly normalized by using a large language model to obtain standardized text data.

6. The method for recording a meeting based on a large language model according to claim 5, characterized in that: The obtaining of a predefined structured slot template and filling the normalized text data into the structured slot template comprises: Obtaining a predefined structured slot template, wherein the slots in the structured slot template at least include topics, speakers, key points, task arrangements, and meeting resolutions; Based on the standard SRL structure, the structured information corresponding to the slots in the standard text data is extracted, and the structured information is filled into the corresponding slots in the structured slot template respectively.

7. The method for recording a meeting based on a large language model according to claim 6, characterized in that: Generating a structured meeting minutes text includes: Collect the topics filled in the slots, classify and merge the topics using the language macro model according to preset merging rules, confirm the paragraph texts corresponding to substantially the same topics in the normalized text data based on the classification and merging results, merge and reconstruct the paragraph texts corresponding to substantially the same topics using the language macro model to obtain structured text fragments corresponding to each topic; The structured text fragment is converted into the structured meeting minutes text according to a preset meeting minutes template format using a language macro model.

8. A conference recording device based on a large language model, applied to the method according to any one of claims 1 to 7, characterized in that: The conference recording device based on the language large model includes: An audio preprocessing module is used to obtain original conference audio data, perform text preprocessing on the original conference audio data, and obtain conference text data; a structured processing module, configured to perform syntactic analysis on the conference text data, identify predicates in the conference text data, call an SRL model to perform role labeling on each predicate, and obtain a basic SRL structure; A text reconstruction module is used to perform semantic analysis on the conference text data using a large language model to extract semantic elements, perform written rewriting based on the semantic elements to obtain a written rewriting result, and reconstruct the content of the written rewriting result based on the basic SRL structure to obtain reconstructed text data; A text normalization module, configured to perform normalization processing on the reconstructed text data to obtain normalized text data; The meeting minutes generation module is used to obtain a predefined structured slot template, fill the normalized text data into the structured slot template, and generate a structured meeting minutes text.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program instructions, which are loaded and executed by a processor to implement the operations performed by the method according to any one of claims 1 to 7.

10. An electronic device comprising a processor and a memory, characterized in that: The memory stores computer program instructions that can be executed by the processor, and when the processor executes the computer program instructions, the processor implements the method according to any one of claims 1 to 7.

Images