Conference recording method and device based on language large model, medium and equipment

Abstract

The application discloses a conference recording method and device based on a language large model, a medium and equipment, the method comprises the following steps: text preprocessing is performed on original conference audio data to obtain conference text data; syntax analysis is performed on the conference text data to obtain a basic SRL structure; semantic analysis is performed on the conference text data through a language large model to obtain a written rewriting result, and content reconstruction is performed on the written rewriting result based on the basic SRL structure to obtain reconstructed text data; the reconstructed text data is normalized to obtain normalized text data; and the normalized text data is filled into a structured slot template to generate a structured conference minutes text. The application combines an SRL model with a language large model, realizes the complementation of the SRL model and the language large model in function, and solves the problems of information omission, unclear expression or logical jump, context ambiguity and omission understanding, unstable output structure and insufficient credibility of generation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a conference recording method and device based on a language large model, a medium and equipment. BACKGROUND

[0002] With the popularity of remote work, online meetings and multi-team collaboration, meeting minutes play an increasingly important role in enterprise management, project advancement and scientific research discussion. High-quality meeting minutes not only record key information and distribute to-do lists, but also serve as an important basis for subsequent decision-making and responsibility tracing. However, in the past, manually written conference content has the problems of low efficiency, strong subjectivity and inconsistent content, so it needs the support of automatic generation technology. In recent years, relying on the progress of natural language processing (NLP) and speech recognition (ASR), automatic conference recording has become a research hotspot in intelligent office.

[0003] In recent years, language large models (such as GPT-3.5 / 4, Claude, Gemini) have shown strong text understanding and generation capabilities. The existing method is to directly input the complete conference text into the large model, and prompt the large model to generate meeting minutes through prompt words, which has achieved good results.

[0004] However, language large models lack deep modeling of semantic structures at the sentence level and even the chapter level, and cannot accurately capture the "action-target-participant" relationship in multi-party dialogue. The existing large model directly generates conference minutes, which has the problems of missing structural information and shallow semantic understanding.

[0005] Therefore, there is an urgent need for a new conference recording method based on a language large model. SUMMARY

[0006] In view of the above problems, the present application is proposed in order to provide a conference recording method based on a language large model, device, medium and equipment which overcome the above problems or at least partially solve the above problems.

[0007] Other characteristics and advantages of the present application will become apparent from the following detailed description, or will be learned by practice of the present application.

[0008] According to a first aspect of the embodiments of the present application, a conference recording method based on a language large model is provided, which comprises:

[0009] obtaining original conference audio data, performing text preprocessing on the original conference audio data to obtain conference text data;

[0010] performing syntax analysis on the conference text data, identifying predicates in the conference text data, calling an SRL model to perform role labeling on each predicate to obtain a basic SRL structure;

[0011] performing semantic analysis on the conference text data by a language large model, extracting semantic elements, performing written rewriting based on the semantic elements to obtain a written rewriting result, and performing content reconstruction on the written rewriting result based on the basic SRL structure to obtain reconstructed text data;

[0012] performing normalization processing on the reconstructed text data to obtain normalized text data;

[0013] obtaining a predefined structured slot template, filling the normalized text data into the structured slot template, and generating a structured conference minutes text.

[0014] In some embodiments of the present application, the text preprocessing on the original conference audio data to obtain conference text data comprises:

[0015] performing frame cutting processing on the original conference audio data to obtain a slice sequence;

[0016] performing clustering on the slice sequence by a DBSCAN density clustering algorithm to obtain clustered segments;

[0017] extracting speaker audio representations in the clustered segments by a voiceprint recognition model, and combining temporal relationships and similarities to perform identity labeling on speakers to obtain a set of speaker-labeled speech paragraphs;

[0018] performing speech recognition processing on each set of speech paragraphs to generate corresponding speech text information;

[0019] performing text normalization processing on the speech text information, and splicing the speech text information after text normalization processing according to temporal relationships to obtain the conference text data.

[0020] In some embodiments of the present application, the clustering of the slice sequence by the DBSCAN density clustering algorithm to obtain clustered segments comprises:

[0021] sorting the slice sequence according to temporal relationships, and calculating average signal intensity of a plurality of continuous slices in the sorted slice sequence by a sliding window, the average signal intensity being the average signal intensity of all slices in the sliding window;

[0022] determining whether the average signal intensity is greater than a preset intensity threshold, and if the average signal intensity is greater than the preset intensity threshold, taking all slices in the sliding window as a speech active segment;

[0023] Collecting all voice activity segments in the slice sequence, and merging to obtain the clustered segment with a timestamp.

[0024] In some embodiments of the present application, the speaker audio feature in the clustered segment is extracted by the voiceprint recognition model, and the identity of the speaker is labeled based on the timing relationship and the similarity, to obtain a set of speech passages with speaker labels, which includes:

[0025] The speaker audio feature in the clustered segment is extracted by the voiceprint recognition model, and the similarity of the speaker audio features of multiple continuous interval segments is calculated based on the timing relationship using a sliding window, and the multiple interval segments are located within the sliding window range.

[0026] If the similarity is greater than the preset similarity threshold, the speaker audio features of the multiple continuous interval segments are defined as the same speaker, and the identity of the speaker is labeled.

[0027] After all the speakers have completed identity labeling, the set of speech passages with speaker labels is obtained.

[0028] In some embodiments of the present application, the normalization processing of the reconstructed text data to obtain normalized text data includes:

[0029] The reconstructed text data is taken as the input of the SRL model, and the SRL model is used to perform SRL normalization processing on the basic SRL structure to obtain a normalized SRL structure.

[0030] The normalized SRL structure and the reconstructed text data are jointly normalized by a language large model to obtain normalized text data.

[0031] In some embodiments of the present application, the pre-defined structured slot template is obtained, and the normalized text data is filled into the structured slot template, which includes:

[0032] A pre-defined structured slot template is obtained, and the slots in the structured slot template include at least topics, speakers, key points, task arrangements, and meeting resolutions.

[0033] Based on the normalized SRL structure, the structured information corresponding to the slots in the normalized text data is extracted, and the structured information is filled into the corresponding slots in the structured slot template.

[0034] In some embodiments of the present application, the structured meeting minutes text is generated, which includes:

[0035] collect the topics filled in the slot, classify and merge the topics according to a preset merging rule by using a language large model, confirm the corresponding paragraph text of the substantially same topics in the standardized text data according to the classification and merging result, and merge and reconstruct the corresponding paragraph text of the substantially same topics by using a language large model to obtain a structured text segment corresponding to each topic;

[0036] The structured text segment is converted into the structured meeting minutes text according to a preset meeting minutes template format by using a language large model.

[0037] According to a second aspect of the embodiment of the present application, a language large model-based meeting recording device is provided, which comprises:

[0038] An audio preprocessing module is configured to acquire original meeting audio data, and perform text preprocessing on the original meeting audio data to obtain meeting text data.

[0039] A structured processing module is configured to perform syntax analysis on the meeting text data, identify predicates in the meeting text data, call a SRL model to perform role labeling on each predicate to obtain a basic SRL structure.

[0040] A text reconstruction module is configured to perform semantic analysis on the meeting text data by using a language large model, extract semantic elements, perform written rewriting based on the semantic elements to obtain a written rewriting result, and perform content reconstruction on the written rewriting result based on the basic SRL structure to obtain reconstructed text data.

[0041] A text normalization module is configured to perform normalization processing on the reconstructed text data to obtain standardized text data.

[0042] A meeting minutes generation module is configured to acquire a predefined structured slot template, fill the standardized text data into the structured slot template, and generate a structured meeting minutes text.

[0043] According to a third aspect of the embodiment of the present application, a computer readable storage medium is provided, which stores computer program instructions, the computer program instructions are loaded and executed by a processor to implement the operations performed by the method according to any one of the above.

[0044] According to a fourth aspect of the embodiment of the present application, an electronic device is provided, which comprises a processor and a memory, the memory stores computer program instructions capable of being executed by the processor, and the processor implements the instructions of the method according to any one of the above when executing the computer program instructions.

[0045] The technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages.

[0046] The conference recording method, device, medium and equipment based on a large language model provided by the embodiments of the present application extract a basic SRL structure by performing syntactic analysis on the conference text data through an SRL model, and extract semantic elements by performing semantic analysis on the conference text data through a large language model. The SRL model and the large language model are combined, the SRL model provides clear semantic slots, the minutes content has the characteristics of traceable source, controllable content and verifiable structure, the problem that the generated sentences of the traditional large language model minutes are smooth but lack factual support is avoided, the generated results are natural and have structure traceability, the complementation of the functions of the SRL model and the large language model is realized, and the problems of information omission, unclear expression or logical jump, context ambiguity and omission understanding, unstable output structure and insufficient generation credibility are solved.

[0047] The above description is only a summary of the technical solutions of the present application, in order to more clearly understand the technical means of the present application, the content of the specification can be implemented, and in order to make the above and other purposes, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described below. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0049] Figure 1 The flowchart of the conference recording method based on a large language model provided by the embodiments of the present application is shown in the figure.

[0050] Figure 2 The principle structure schematic diagram of the conference recording device based on a large language model provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0051] The exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0052] Various structural diagrams according to embodiments of the present disclosure are shown in the drawings. These diagrams are not drawn to scale in which certain details are shown exaggerated in scale for purposes of clarity and understanding, and certain other details are omitted. The shapes of various regions, layers, and the relative sizes and positional relationships among them shown in the drawings are merely exemplary, and in actuality, they can deviate due to manufacturing tolerances or technical limitations, and a person skilled in the art can additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0053] In the context of the present disclosure, when a layer / element is referred to as being located "on" another layer / element, the layer / element can be directly located on the other layer / element, or there can be an intermediate layer / element between them. In addition, if a layer / element is located "on" another layer / element in one orientation, it can be located "under" the other layer / element when the orientation is reversed. In the context of the present disclosure, similar or identical components can be denoted by the same or similar reference numerals.

[0054] In order to better understand the above technical solutions, the above technical solutions will be described in detail below in conjunction with specific embodiments. It should be understood that the embodiments of the present disclosure and the specific features in the embodiments are detailed descriptions of the technical solutions of the present disclosure, rather than limitations of the technical solutions of the present disclosure. In the case of no conflict, the technical features in the embodiments of the present disclosure and the embodiments can be combined with each other.

[0055] Figure 1 is a flow diagram of a conference recording method based on a language large model provided by an embodiment of the present disclosure, as shown in Figure 1 The conference recording method based on a language large model includes the following steps:

[0056] S1, obtaining original conference audio data, performing text preprocessing on the original conference audio data to obtain conference text data;

[0057] The original conference audio data is an audio file in a format such as wav or mp3. After obtaining the original conference audio data, the embodiment of the present disclosure converts it into conference text data through text preprocessing.

[0058] The text preprocessing on the original conference audio data to obtain conference text data includes: performing frame cutting processing on the original conference audio data to obtain a cutting sequence; performing clustering on the cutting sequence by a DBSCAN density clustering algorithm to obtain clustered segments; extracting speaker audio representation in the clustered segments by a voiceprint recognition model, and combining a time sequence relationship and a similarity to perform identity labeling on a speaker to obtain a speaker-labeled speech paragraph set; performing speech recognition processing on each speech paragraph set to generate corresponding speech text information; performing text regularization processing on the speech text information, and splicing the text-regularized speech text information according to a time sequence relationship to obtain the conference text data.

[0059] Specifically, the original conference audio data is frame cut, for example, frame cutting is performed on the original conference audio data based on a time sequence relationship, 25 milliseconds is taken as one frame, and 10 milliseconds is taken as a step, short-time energy and a zero-crossing rate of each cutting (or frame cutting) are extracted, and it is judged whether the current cutting contains valid speech signals. The short-time energy is an index for measuring signal strength in a frame, and can reflect the presence or absence of sound, and the calculation method is the average value of the square of the amplitudes of all sampling points in the frame. The energy of a silent frame is close to zero, and a speech frame has obvious energy fluctuation, so the energy feature can be used as the first judgment basis for speech detection. The zero-crossing rate represents the number of times that the signal crosses zero in a frame, which reflects the frequency composition of the sound. The zero-crossing rate of a silent signal is usually very low or irregular, while a speech signal (especially a clear tone) shows a higher and stable crossing rate. By comprehensively judging the change trend of energy and zero-crossing rate, the continuous non-silence segment, i.e., the potential speaking section, can be accurately detected. The finally obtained cutting sequence is, for example, {start_time: 75ms, end_time: 100ms, signal: [01001011110]}.

[0060] After the slice sequence is obtained, speech segment clustering and sentence boundary recognition need to be implemented, and the time stamps before and after each sentence are obtained, and then the DBSCAN density clustering algorithm is used to cluster the slice sequence to obtain a clustered segment, including: sorting the slice sequence according to the time sequence relationship, calculating the average signal intensity of a plurality of continuous slices in the sorted slice sequence by using a sliding window, the average signal intensity refers to the average signal intensity of all slices in the sliding window; determining whether the average signal intensity is greater than a preset intensity threshold, if the average signal intensity is greater than the preset intensity threshold, all slices in the sliding window are taken as a speech activity segment, if the average signal intensity is not greater than the preset intensity threshold, all slices in the sliding window are taken as a silence segment; collecting all speech activity segments in the slice sequence, and merging to obtain the clustered segment with a time stamp. The time stamp of the clustered segment is the time stamp of the starting point and the ending point, and finally the clustered segment obtained is, for example, {start_time: 3.5s, end_time: 8.25s, voice_data: split_wav_data}.

[0061] After the sentence-level segment construction is completed, the voiceprint feature of each speech segment is further extracted on the time axis, speaker recognition and clustering are performed, the purpose is to identify the acoustic feature difference of participants in the conference, and the speech content is attributed to a specific “speaker”, so that speaker separation is realized. Specifically, the voiceprint recognition model is used to extract the speaker audio representation in the clustered segment, and the identity of the speaker is labeled by combining the time sequence relationship and the similarity, to obtain a speech paragraph set with a speaker label, including: extracting the speaker audio representation in the clustered segment by using the voiceprint recognition model, calculating the similarity of the speaker audio representation of a plurality of continuous interval segments based on the time sequence relationship by using a sliding window, and the plurality of interval segments are located in the sliding window range; determining whether the similarity is greater than a preset similarity threshold, if the similarity is greater than the preset similarity threshold, the speaker audio representation of the plurality of continuous interval segments is defined as the same speaker, and the identity of the speaker is labeled; after the identity of all speakers is labeled, the speech paragraph set with a speaker label is obtained.

[0062] The voiceprint recognition model may be, for example, an ECAPA-TDNN model, and may be used to extract a speaker audio representation (speaker embedding representation) of each clustered split_way_data, cluster based on similarity on a time axis, and label the speaker based on context stability and continuity to obtain a speaker-labeled speech paragraph set, for example, {start_time: 3.5s, end_time: 20.5s, speaker: speaker1, voice_data: [split_wav_data1, split_wav_data2, split_wav_data3, …]}.

[0063] In an embodiment of the present application, each speech paragraph in the speaker-labeled speech paragraph set is input into an automatic speech recognition (ASR) module for speech recognition processing, and corresponding speech text information is generated, for example, input {start_time: 3.5s, end_time: 20.5s, speaker: speaker1, voice_data: [split_wav_data1, split_wav_data2, split_wav_data3, …]} and output {start_time: 3.5s, end_time: 20.5s, speaker: speaker1, text: [Hmm, I think we need to delay this project, that is, uh, the resources may not be fully prepared]} after speech recognition processing by the speech recognition module. The speech recognition module may be, for example, an open source model such as Whisper, FunASR, etc., or a commercially available model on the market.

[0064] After obtaining the speech text information, in order to improve the readability of the text information, it is necessary to remove the colloquial expressions, tone fillers and non-semantic language fragments in the speech text information, such as "ah", "um", "just" and the like; the embodiment of the present application constructs a high-frequency tone word dictionary based on corpus statistics and language specifications in advance, which covers single words, phrases and multi-word combination structures, including words and vocabularies of this type such as "ah", "um", "just" and the like. The embodiment of the present application adopts a regular rule and a sliding window mechanism, and performs accurate matching and identification on the speech text information based on the high-frequency tone word dictionary, and a batch of tone word chains with high frequency and loose structure are removed, for example, the obtained conference text data is {start_time: 3.5s, end_time: 20.5s, speaker: speaker1, text: [that, I think we may have to delay this project again, resources may not be fully prepared]}.

[0065] S2, performing syntax analysis on the conference text data to identify the predicate in the conference text data, calling a SRL model to perform role labeling on each predicate to obtain a basic SRL structure;

[0066] The embodiment of the present application performs syntax analysis on the conference text data by a natural language processing tool such as LTP, Stanford CoreNLP or HanLP, and extracts the subject-predicate, core verb and other dependency relationships in each sentence to identify the predicate in the conference text data; wherein, the predicate verb is matched by a regular matching method based on a high-frequency verb predicate dictionary, which is used to enhance the recognition ability of the model for non-general verbs; the high-frequency verb predicate dictionary is obtained by combining artificial construction and corpus statistics, and the high-frequency verb predicate dictionary covers commonly used verb entries in contexts such as policy, business and administrative instructions (such as "promote", "strengthen", "implement", "complete", "establish", "require", "provide" and the like); for example, the identified predicate is predicate_verb: [delay, prepare, think].

[0067] The SRL model may be, for example, an SRL model based on BERT, ERNIE, LTP or AllenNLP, which performs role labeling on each predicate and identifies the agent (ARG0), the recipient (ARG1), time, place, manner and other additional information (ARGM series) of the action. The output of this stage is a preliminary semantic structure, that is, a basic SRL structure (or event unit) of "predicate + role + other", for example, the basic SRL structure obtained on the basis of the above embodiment is: [{"predicate": "delay", "roles": {"ARG0": "we", "ARG1": "this project", "ARGM-MOD": "possible"}}, {"predicate": "prepare", "roles": {"ARG0": "resources", "ARGM-NEG": "not", "ARGM-ADV": "completely"}}, …].

[0068] S3, performing semantic analysis on the conference text data by a language large model, extracting semantic elements, performing written rewriting based on the semantic elements to obtain a written rewriting result, and performing content reconstruction on the written rewriting result based on the basic SRL structure to obtain reconstructed text data.

[0069] The language large model (LLM) in the embodiment of the application learns the statistical rules and semantic structure of language by training on massive text, thereby having the ability to generate coherent and logical text. Through a preset prompt, a user can explicitly convey the task target and expected output form to the language large model, and the language large model automatically generates a matching answer according to its understanding of the language context.

[0070] The embodiment of the application performs semantic analysis on the conference text data by a language large model, extracts semantic elements, and performs written rewriting on the expression form of the conference text data to improve semantic clarity and structural standardization. The semantic elements may be, for example, core topics, event stems and pragmatic intentions. Specifically, the embodiment of the application constructs a prompt containing task instructions to guide the language large model to identify the core topics, event stems and pragmatic intentions in the paragraphs of the conference text data and require output of structured elements and written rewriting results.

[0071] The prompt example of the written rewriting result obtained by performing semantic analysis on the conference text data, extracting semantic elements, and performing written rewriting based on the semantic elements is: "Please perform semantic analysis on the following conference speech content, extract its core topic, main event description, and speaker intention, and rewrite it into a formal and clear written sentence. The output format is: 1, core topic; 2, main event; 3, speaker intention; 4, rewritten sentence. The paragraph content is as follows: 'That, we now have a project, seems to have some problems in terms of resources, um, I think the start time can be moved back? Or wait first, wait until the preparation is more sufficient before starting' ".

[0072] The language large model outputs:

[0073] Core topic: project start plan and resource preparation situation;

[0074] Main event: due to insufficient resources, propose to postpone the project start time;

[0075] Speaker intention: tactfully express the postponement suggestion, and hope to wait for more sufficient preparation conditions;

[0076] Rewritten sentence: "Due to the current resources not being fully in place, it is suggested to appropriately postpone the start time of the project to ensure that the preparation work is more sufficient".

[0077] The embodiment of the present application guides the language large model to perform content reconstruction on the written rewriting result based on the basic SRL structure through the pre-defined content reconstruction prompt, generates a natural language paragraph with clear main structure and complete semantic expression, and ensures the expression accuracy and structural stability.

[0078] For example, input the following data to the language large model for content restructuring: [{"predicate": "delay", "roles": {"ARG0": "we", "ARG1": "this project", "ARGM-MOD": "possibly"}}, {"predicate": "prepare", "roles": {"ARG0": "resources", "ARGM-NEG": "not", "ARGM-ADV": "completely"}}, …] and "Due to the current resources not being fully in place, it is suggested to appropriately delay the start time of the project to ensure more thorough preparation" and the speaker label as input information. The restructuring text data obtained is, for example: "Zhang San said that due to the current resources not being fully in place, it is suggested to appropriately delay the start time of the project to ensure more thorough preparation". By taking the basic SRL structure as the reference basis, together with the specific written rewriting result as the input to let the language large model perform content restructuring, the purpose of fusion is achieved; in the restructuring text data, not only the accurate restoration of the sentence stem action ("delay", "prepare") is completed, but also the explicit expression of the subject, the syntactic integrity and the style transition from spoken language to written language are realized. In addition, the speaker information is also explicitly included in the generated content, meeting the semantic consistency requirements of subsequent multi-speaker structure fusion.

[0079] S4, normalizing the restructuring text data to obtain normalized text data;

[0080] The normalization processing of the restructuring text data to obtain normalized text data includes: taking the restructuring text data as the input of the SRL model, performing SRL normalization processing on the basic SRL structure through the SRL model to obtain a normalized SRL structure; and performing joint normalization processing on the normalized SRL structure and the restructuring text data through a language large model to obtain normalized text data.

[0081] Since the obtained basic SRL structure may have problems of missing roles, inconsistent naming or unclear semantic boundaries; the main reason is caused by the colloquial expression of the original text, which may omit the subject and use different pronoun forms to express the same semantic entity, therefore, the embodiment of the application needs to perform SRL normalization processing on the basic SRL structure to supplement the omitted roles, unify the role label style (such as ARGM-TMP for "time"), and clean up redundant components, and the reconstructed text data obtained through the foregoing steps is taken as the input of the SRL model to perform SRL normalization processing, further improving the accuracy and integrity of the structure, and the normalized SRL structure has more clear subject-predicate-object information; for example, the obtained normalized SRL structure is [{"predicate": "delay", "roles": {"ARG0": "Zhang San", "ARG1": "this project", "ARGM-MOD": "suggestion"}}, {"predicate": "prepare", "roles": {"ARG0": "current resources", "ARGM-NEG": "not yet", "ARGM-ADV": "in place"}}, …].

[0082] After obtaining the reconstructed text data, since the content reconstruction and the foregoing SRL normalization processing focus on the expression of units at the sentence level or the paragraph level, the pragmatic consistency, the term uniformity and the structural coherence of the whole text are not systematically fused, therefore, in order to ensure the uniformity of the language style of the whole text, the embodiment of the application can also perform unified processing and written expression generation of the paragraph text at the whole text level based on the joint input of the previous multiple reconstructed paragraphs and multiple normalized SRL structures after the SRL normalization processing, that is, the normalized text data is obtained by performing joint normalization processing on the normalized SRL structure and the reconstructed text data through a language large model, the obtained normalized text data only fuses the reconstructed text data of the natural language content, and further utilizes the normalized SRL structure (a version after the naming normalization and the term standardization) to integrate multiple event semantic units and multiple text paragraphs across paragraphs, so as to ensure the uniformity of the language style, the coherence of the logical structure and the completeness of the information expression of the whole text.

[0083] The prompt for performing joint normalization processing on the normalized SRL structure and the reconstructed text data defined in the embodiment of the application is, for example: "Please generate a complete, style-consistent and written standard conference minutes text according to the following multiple semantic structures and content paragraphs, and ensure that the subject is clear, the terms are uniform, the reference is clear and the logic is coherent".

[0084] The obtained normalized text data is, for example: "During the meeting, Zhang San proposed that due to the fact that the key resources required by the current project have not been fully prepared, there are still coordination obstacles in the execution level of the relevant team, therefore it is suggested to postpone the original start plan to ensure that the preparation work is more sufficient and the execution process is more smooth. Li Si added that although the pre-project process has been basically completed, there are still omissions in the arrangement of market research data, and the feedback from some partners has not been collected. If the subsequent process is carried out recklessly, it may affect the overall project rhythm. Therefore, he suggested that the official promotion instruction should be issued after the resources are prepared and the data are verified."

[0085] S5, obtaining a pre-defined structured slot template, filling the normalized text data into the structured slot template to generate a structured meeting minutes text.

[0086] The structured slot template is a set of standardized minutes slot systems constructed based on human pre-definition, and the standardized minutes slot system usually includes a plurality of slots, each slot corresponding to a semantic element in the normalized text data. The embodiment of the application provides structural support for the generation of subsequent meeting minutes text by extracting semantic elements in the normalized text data and filling them into the slots.

[0087] In the embodiment of the application, obtaining a pre-defined structured slot template includes obtaining a pre-defined structured slot template, the slots in the structured slot template including at least topic, speaker, key point, action item and meeting decision; based on the SRL structure, structured information corresponding to the slots in the normalized text data is extracted and filled into the corresponding slots in the structured slot template.

[0088] In the embodiment of the application, the slots are defined in the form of key fields; for example, the topic (Topic) represents the topic name or topic number, usually corresponding to the meeting process or agenda item; for example, the speaker (Speaker) represents the speaker's identity information, which can be name, position or organization unit, etc.; for example, the key point (KeyPoint) represents the key point or opinion statement, which is usually neutral or factual statement; for example, the action item (ActionItem) represents the explicit task arrangement or to-do list, etc., with execution instruction or object; for example, the meeting decision (Decision) represents the conclusion, decision or consensus formed by the meeting, etc. In other embodiments of the application, the key fields corresponding to the slots can also be changed to other semantic elements or the number of slots can be increased or decreased according to actual application requirements.

[0089] For example, when the input normalized text data is "Regarding the problem of customer satisfaction decline in the second quarter, Wang of the marketing department suggests strengthening the customer follow-up mechanism, and submitting an improvement plan within the next month, which is responsible for Li", the structured information corresponding to the slot in the normalized text data is extracted, for example:

[0090] "Topic": "second quarter customer satisfaction decline",

[0091] "Speaker": "Zhang San",

[0092] "KeyPoint": "Wang (marketing department) suggests strengthening the customer follow-up mechanism and submitting an improvement plan",

[0093] "ActionItem": "Li is responsible for submitting an improvement plan within the next month",

[0094] "Decision": "The meeting decides to adopt the plan"

[0095] }.

[0096] The structured meeting minutes text generated by the embodiment of the application includes: collecting the topics filled into the slot, classifying and merging the topics according to the preset merging rule by using a language large model, confirming the paragraph texts corresponding to the substantially identical topics in the normalized text data according to the classification and merging results, merging and reconstructing the paragraph texts corresponding to the substantially identical topics by using a language large model, and obtaining the structured text segments corresponding to each topic; and converting the structured text segments into the structured meeting minutes text according to the preset meeting minutes template format by using a language large model.

[0097] After the semantic element filling of the slot is completed, the embodiment of the application collects all the topics filled into the slot, classifies and merges the topics according to the preset merging rule (which can be artificially defined or open source) by using a language large model, confirms the paragraph texts corresponding to the substantially identical topics in the normalized text data according to the classification and merging results, classifies the substantially identical topics into the same label by using a language large model, and re-groups and arranges all the paragraph texts according to the labels corresponding to the normalized topics, to obtain the structured text segments corresponding to each topic, so as to ensure that the related speeches under the same topic are uniformly archived, and facilitate the generation and expression of subsequent structured minutes.

[0098] The embodiment of the application utilizes a language large model to convert the structured text segments into the structured meeting minutes text according to a preset meeting minutes template format, so as to convert the structured meeting information into natural language expression and organize the output content according to the preset meeting minutes template format.

[0099] For example, the prompt for converting the structured text segments into the structured meeting minutes text is "generate a complete meeting minutes according to the meeting minutes template";

[0100] The input is a plurality of structured text segments, including:

[0101] Structured text segment 1

[0102] {

[0103] "Topic": "Customer satisfaction is declining",

[0104] "Speaker": "Mr. Wang (Marketing Department)",

[0105] "KeyPoint": "The customer return visit process is not clear, and customer feedback cannot be closed loop",

[0106] "ActionItem": "Suggest redesigning the customer return visit process",

[0107] "Decision": "The meeting agrees in principle, and specific improvement plans need to be developed"

[0108] },

[0109] Structured text segment 2

[0110] {

[0111] "Topic": "Customer satisfaction is declining",

[0112] "Speaker": "Mr. Liu (Customer Service Department)",

[0113] "KeyPoint": "The customer service response time is unstable, affecting the satisfaction score",

[0114] "ActionItem": "Plan to introduce an automatic work order distribution mechanism",

[0115] "Decision": "The meeting agrees to pilot the optimization process by the customer service department"

[0116] }。

[0117] The preset meeting minutes template format is, for example, [event name: xxx, specific event: xxx, problem item: xxx, improvement measure: xxx, and follow-up person: xxx].

[0118] The obtained structured meeting minutes text is, for example,

[0119] [Event name: analysis of customer satisfaction decline problem;

[0120] Specific event: the recent meeting focused on the problem of customer satisfaction decline, and the market department and the customer service department analyzed and fed back the relevant situation;

[0121] Problem item: on the one hand, the customer return visit process has design defects, and customer feedback cannot form a closed loop; on the other hand, the customer service response timeliness is unstable, resulting in fluctuation of satisfaction score;

[0122] Improvement measure: Wang suggests redesigning the customer return visit process to ensure that customer opinions are timely tracked and fed back; Liu proposes to introduce an automatic work order distribution mechanism to improve customer service processing efficiency;

[0123] Follow-up person: the market department is responsible for drafting the customer return visit process improvement plan, and the customer service department will start the work order distribution mechanism pilot optimization. The two plans will be submitted to the next meeting for execution progress report.

[0124] Compared with the prior art, the conference recording method based on the language large model extracts the basic SRL structure by performing syntactic analysis on the conference text data through the SRL model, and extracts semantic elements by performing semantic analysis on the conference text data through the language large model. The SRL model and the language large model are combined, not only in the final result layer, but also in the middle link of text reconstruction, gradually enhancing each other's effect, fusing the information of two levels of micro (verb event layer) and macro (paragraph abstract layer), ensuring that the information granularity is detailed and the integrity is not lost. At the same time, the SRL model provides clear semantic slots, so that the minutes content has the characteristics of traceable source, controllable content and verifiable structure, avoiding the problem that the traditional language large model minutes generation sentence is smooth but lacks factual support, making the generated result natural and structured and traceable, realizing the complementation of the functions of the SRL model and the language large model, and solving the problems of information omission, unclear expression or logical jump, context ambiguity and omission understanding, unstable output structure and insufficient generation credibility.

[0125] On the basis of the above embodiment, as an implementation of the method shown in the above Figure 1 , the application provides an embodiment of a conference recording device based on a language large model. The device embodiment and Figure 1The method embodiment shown corresponds to the device, which can be specifically applied to various electronic devices, refer to Figure 2 As shown, the conference recording device based on the language large model comprises:

[0126] An audio preprocessing module 100 is configured to acquire original conference audio data, perform text preprocessing on the original conference audio data, and obtain conference text data.

[0127] A structured processing module 200 is configured to perform syntax analysis on the conference text data, identify predicates in the conference text data, call a SRL model to perform role labeling on each predicate, and obtain a basic SRL structure.

[0128] A text reconstruction module 300 is configured to perform semantic analysis on the conference text data by a language large model, extract semantic elements, perform written rewriting based on the semantic elements, obtain a written rewriting result, perform content reconstruction on the written rewriting result based on the basic SRL structure, and obtain reconstructed text data.

[0129] A text normalization module 400 is configured to perform normalization processing on the reconstructed text data to obtain normalized text data.

[0130] A conference summary generation module 500 is configured to acquire a predefined structured slot template, fill the normalized text data into the structured slot template, and generate structured conference summary text.

[0131] The conference recording device based on the language large model provided by the embodiment of the present application can execute the conference recording method based on the language large model provided by the above embodiment, and has the corresponding function steps and beneficial effects of the conference recording method based on the language large model provided by the above embodiment. For details, please refer to the embodiments of the conference recording method based on the language large model. The present application will not be described here again.

[0132] The electronic device can include a processor and a memory, where the processor and the memory can be connected through a bus or other means. The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or a combination of the above chips, or the like. The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions / modules of the conference recording method based on a language large model in the embodiments of the present application. The processor performs various functional applications and data processing of the processor by running the non-transitory software programs, instructions, and modules stored in the memory, that is, implements the conference recording method based on a language large model in the above method embodiments.

[0133] The memory can include a program storage area and a data storage area, where the program storage area can store an operating system, at least one application required by a function; the data storage area can store data created by the processor, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. The one or more modules are stored in the memory, and when executed by the processor, perform the conference recording method based on a language large model in the above method embodiments. The above electronic device specific details can correspond to the relevant description and effects of the corresponding method embodiments, which will not be described here. Those skilled in the art can understand that all or part of the processes in the above embodiments can be completed by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. The storage medium can be a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc. The storage medium can also include a combination of the above types of memory.

[0134] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order not to obscure the understanding of this description.

[0135] Similarly, it is to be understood that the embodiments of the present application can be readily combined with one another, and / or with other devices and methods, to produce further embodiments of the present application. Such embodiments of the present application, which are not explicitly described herein, are nevertheless within the scope of the present application. Additionally, it is contemplated that various embodiments of the present application include any combination of the particular features described herein. Furthermore, it is intended to include all possible combinations of claim elements. It is intended to include "consisting" as well as "consisting essentially of and "comprising" any of the features of the application. It is intended that each of the additional features disclosed in this specification and those shown in the attached claims, and any additional unclaimed features that are individual combinations of the features shown herein, can be included in the application, alone or in any combination with any of the other features shown herein or any additional unclaimed features disclosed in this specification. It is intended that the application not be limited to the embodiments disclosed in the specification, but that the application can be practiced with any device that provides the general functionality described herein. It is intended that the application not be limited to the embodiments disclosed in the specification, but that the application can be practiced with any device that provides the general functionality described herein. It is intended that the application not be limited to the embodiments disclosed in the specification, but that the application can be practiced with any device that provides the general functionality described herein. It is intended that the application not be limited to the embodiments disclosed in the specification, but that the application can be practiced with any device that provides the general functionality described herein. It is intended that the application not be limited to the embodiments disclosed in the specification, but that the application can be practiced with any device that provides the general functionality described herein.

Claims

1. A meeting recording method based on a large language model, characterized in that, The meeting recording method based on the language big model includes: Obtain the original meeting audio data, and perform text preprocessing on the original meeting audio data to obtain meeting text data; Syntactic analysis is performed on the meeting text data to identify the predicates in the meeting text data. The SRL model is then used to assign roles to each predicate to obtain the basic SRL structure. The meeting text data is semantically analyzed using a large language model to extract semantic elements. Based on these semantic elements, the text is rewritten to obtain a written rewritten result. Then, based on the basic SRL structure, the rewritten result is reconstructed to obtain reconstructed text data. The reconstructed text data is then normalized to obtain normalized text data; Obtain a predefined structured slot template, fill the standardized text data into the structured slot template, and generate structured meeting minutes text; The text preprocessing of the original conference audio data to obtain conference text data includes: The original conference audio data is sliced ​​into frames to obtain a slice sequence; The slice sequence was clustered using the DBSCAN density clustering algorithm to obtain clustered segments. The speaker's audio representation in the clustered segments is extracted by the voiceprint recognition model, and the speaker's identity is labeled by combining temporal relationship and similarity, resulting in a set of speech segments labeled with the speaker. Speech recognition processing is performed on each set of spoken segments to generate corresponding spoken text information; The spoken text information is processed by text normalization, and the normalized spoken text information is then spliced ​​together according to the chronological order to obtain the meeting text data; The process of clustering the slice sequence using the DBSCAN density clustering algorithm to obtain clustered segments includes: The slice sequence is sorted according to temporal relationship, and the average signal intensity of multiple consecutive slices in the sorted slice sequence is calculated using a sliding window. The average signal intensity refers to the average signal intensity of all slices within the sliding window. Confirm whether the average signal strength is greater than a preset strength threshold. If the average signal strength is greater than the preset strength threshold, then all slices within the sliding window are used as the speech activity segment. All speech activity segments in the slice sequence are collected and merged to obtain the clustered segments with timestamps.

2. The meeting recording method based on a large language model according to claim 1, characterized in that, The process involves extracting the speaker's audio representation from the clustered segments using a voiceprint recognition model, and then identifying the speaker by combining temporal relationships and similarity. This results in a set of speech segments tagged with the speaker's name, including: The speaker's audio representation in the clustered segments is extracted by a voiceprint recognition model. The similarity of the speaker's audio representation in multiple consecutive intervals is calculated based on the temporal relationship using a sliding window. All of the intervals are within the range of the sliding window. Confirm whether the similarity is greater than a preset similarity threshold. If the similarity is greater than the preset similarity threshold, define multiple consecutive interval segments of the speaker's audio as the same speaker and label the speaker's identity. After all speakers have completed the identification process, a set of speech segments labeled with the speaker's name is obtained.

3. The meeting recording method based on a large language model according to claim 1, characterized in that, The process of normalizing the reconstructed text data to obtain normalized text data includes: The reconstructed text data is used as input to the SRL model, and the basic SRL structure is normalized by the SRL model to obtain a normalized SRL structure. The standardized SRL structure and reconstructed text data are jointly normalized using a large language model to obtain normalized text data.

4. The meeting recording method based on a large language model according to claim 3, characterized in that, The step of obtaining a predefined structured slot template and filling the structured slot template with the normalized text data includes: Obtain a predefined structured slot template, wherein the slots in the structured slot template include at least topics, speakers, key points, task assignments, and meeting resolutions; Based on the specified SRL structure, the structured information corresponding to the slot in the normalized text data is extracted, and the structured information is filled into the corresponding slot in the structured slot template.

5. The meeting recording method based on a large language model according to claim 4, characterized in that, The generated structured meeting minutes text includes: Issues are collected and filled into the slots. The language big model is used to classify and merge the issues according to the preset merging rules. Based on the classification and merging results, the paragraph texts corresponding to the substantially identical issues in the normalized text data are identified. The paragraph texts corresponding to the substantially identical issues are merged and reconstructed through the language big model to obtain the structured text fragments corresponding to each issue. The structured text fragments are converted into structured meeting minutes text using a language big data model based on a preset meeting minutes template format.

6. A conference recording device based on a large language model, applied to the method described in any one of claims 1-5, characterized in that, The conference recording device based on the language large model includes: The audio preprocessing module is used to acquire raw meeting audio data, perform text preprocessing on the raw meeting audio data, and obtain meeting text data. The structured processing module is used to perform syntactic analysis on the meeting text data, identify the predicates in the meeting text data, call the SRL model to perform role labeling on each predicate, and obtain the basic SRL structure; The text reconstruction module is used to perform semantic analysis on the conference text data through a large language model, extract semantic elements, rewrite the text based on the semantic elements to obtain a written rewrite result, and reconstruct the content of the written rewrite result based on the basic SRL structure to obtain reconstructed text data. The text normalization module is used to normalize the reconstructed text data to obtain normalized text data; The meeting minutes generation module is used to obtain a predefined structured slot template, fill the standardized text data into the structured slot template, and generate structured meeting minutes text.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which are loaded and executed by a processor to perform the operations described in any one of claims 1-5.

8. An electronic device comprising a processor and a memory, characterized in that, The memory stores computer program instructions that can be executed by the processor, and when the processor executes the computer program instructions, it implements the instructions of the method as described in any one of claims 1-5.

Images