A multimodal meeting minutes automatic generation method and system

Abstract

The application relates to the technical field of multi-modal data processing, and discloses a multi-modal conference minutes automatic generation method and system, which comprises the following steps: fusing multi-lingual speech transcription and speaker identity alignment information, and constructing a conference speech directed heterogeneous graph; generating a task-dependent simplex based on the conference speech directed heterogeneous graph and an execution semantic anchor point; and performing context-aware minutes structured rendering and intelligent action scheme generation based on the task-dependent simplex. Through the construction of the conference speech directed heterogeneous graph and the task-dependent simplex, the application realizes the automatic generation of multi-modal conference minutes, accurately encodes the correlation between conference tasks, helps the efficient conversion of conference decision-making into actual execution, and improves the execution efficiency of team conference decision-making and the project closed-loop management capability.

Claims

1. A method for automatically generating multimodal meeting minutes, characterized in that, The method includes: S1: Integrate multilingual speech transcription and speaker identity alignment information to construct a directed heterogeneous graph of conference speeches; S2: Based on the directed heterogeneous graph of conference speeches, generate a task-dependent simplicial complex with execution semantic anchors; the step of generating the task-dependent simplicial complex with execution semantic anchors includes: Each task triple is abstracted as a 0-simplex. The structure of the 0-simplex includes a task identifier, task triple information, associated issue subgraph ID, and execution semantic anchor point. The task identifier type includes a dependent source task identifier and a dependent target task identifier. For any two 0-simplexes, determine whether there is a dependency relationship based on the dependency relationship candidate set. If there is a dependency relationship, construct a 1-simplex. The structure of the 1-simplex includes edge identifiers, task identifiers of the two associated 0-simplexes, dependency type, and dependency strength. For any three 0-simplexes, if an unskippable chain dependency is formed, a 2-simplex is constructed. The structure of the 2-simplex includes face identifiers, task identifiers of the three associated 0-simplexes, edge identifiers of the two associated 1-simplexes, and dependency chain timing information. If there is no explicit 1-simplex between the first and last two 0-simplexes in any three 0-simplexes forming a chain dependency, a derivation edge of type logical is automatically generated when constructing the 2-simplex. This edge is only used to satisfy the downward closure of the topology. Logical edges do not participate in task scheduling, critical path calculation, or risk warning; they are only used for topology consistency maintenance and visualization analysis. All generated 0-simplexes, 1-simplexes, and 2-simplexes are integrated according to topological relationships to form a task-dependent simplex complex with execution semantic anchors; S3: Based on task-dependent simple complexes, it enables context-aware structured rendering of minutes and intelligent action plan generation.

2. The method for automatically generating multimodal meeting minutes according to claim 1, characterized in that, The steps for integrating multilingual speech transcription and speaker identity alignment information to construct a directed heterogeneous graph of conference speeches include: S11: Process the audio signal of the entire meeting, combine the multilingual speech transcription model and the voiceprint recognition model to generate a candidate set of transcription segment nodes with voiceprint identification; S12: Based on the candidate set of transcribed fragment nodes with voiceprint identifiers and the pre-meeting registration list in the speaker identity alignment information, a unified identity URI is generated using multilingual named entity normalization rules; S13: Based on the candidate set of transcribed fragment nodes with voiceprint identifiers, unified identity URIs, and speaker identity alignment information, construct a node set of a directed heterogeneous graph of conference speeches; S14: Construct the edge set of the directed heterogeneous graph of conference speeches based on the node set of the conference speech directed heterogeneous graph, the multilingual translation model, and the speech temporal directed edge construction method; S15: Integrate the node set and edge set of the directed heterogeneous graph of conference speeches to generate a directed heterogeneous graph of conference speeches.

3. The method for automatically generating multimodal meeting minutes according to claim 2, characterized in that, The steps to construct the edge set of a directed heterogeneous graph of conference presentations include: S141: Construct the "Said by..." edge, where the starting point of the "Said by..." edge is the participant node and the ending point is the speech segment node, which is used to represent the relationship between the participant and the speech segment; S142: Construct a "Translate to..." edge, where the starting point of the "Translate to..." edge is a speech fragment node in a language, and the ending point is a speech fragment node in the corresponding translated text. The edge's attributes include translation confidence. S143: Construct "continuing from..." edges, where the starting point of the "continuing from..." edge is the speech segment node with the earlier timestamp, and the ending point is the speech segment node with the later timestamp. Add a time interval attribute to each edge; integrate the three types of directed edges to form the edge set of the conference speech directed heterogeneous graph.

4. The method for automatically generating multimodal meeting minutes according to claim 3, characterized in that, The steps for generating a unified identity URI using multilingual named entity normalization rules include: S121: Extract the transcribed text of each transcribed segment node from the candidate set of transcribed segment nodes with voiceprint identification, and extract the participants' names from the transcribed text using a named entity recognition model to obtain a set of names associated with voiceprints. S122: Perform rule matching on the names of attendees in the set of names associated with voiceprints, and accurately match them with the multilingual identity identifiers provided by attendees in the pre-registration list; S123: For names that fail to pass an exact match, select participants' names with similarity higher than a set threshold as candidates through semantic similarity calculation; if there are still multiple candidates or no candidates after semantic association, generate a manual verification prompt, and complete the matching after confirmation; map the participants' names in all transcribed fragments to a unique unified identity URI.

5. The method for automatically generating multimodal meeting minutes according to claim 1, characterized in that, The steps for generating a task dependency simplification complex with execution semantic anchors also include: S21: Based on the directed heterogeneous graph of conference speeches, the seed set of topic keywords and the seed set of task trigger words, topic subgraphs are formed by clustering through graph attention network, resulting in K topic subgraphs; S22: Based on K topic subgraphs, perform task triple identification and dependency sentence detection operations on the speech fragment nodes in each topic subgraph to obtain task triples and dependency relationship candidate sets.

6. The method for automatically generating multimodal meeting minutes according to claim 5, characterized in that, The steps for performing the triple identification and dependency statement detection operations include: S221: For each topic subgraph, extract the transcribed and translated text of the speech fragment node to form the speech fragment text, and input it into the fine-tuned BERT-CRF joint model to generate task triples; S222: For the speech fragment text of the output task triple within each topic subgraph, perform dependency sentence pattern detection, and use a multilingual dependency sentence pattern library combined with semantic similarity matching to generate a dependency relationship candidate set.

7. The method for automatically generating multimodal meeting minutes according to claim 5, characterized in that, The steps for task-dependent simple complex execution context-aware summary structured rendering and intelligent action plan generation include: S31: Based on the task-dependent simplex, perform simplex skeleton structure traversal and generate simplex traversal sequence; S32: Based on the simplex traversal sequence, task-dependent simplex complex, directed heterogeneous graph of conference speeches, and participant information mapping table, perform context-aware structured rendering of minutes to obtain structured minutes elements; S33: Based on structured minutes elements, execute intelligent action plan generation and build interactive structured documents; S34: Based on the structured document, participant information mapping table, and task-dependent simple complex, perform consistency checks and output the final structured document and intelligent action plan.

8. The method for automatically generating multimodal meeting minutes according to claim 7, characterized in that, The steps involved in performing context-aware, structured rendering of minutes include: S321: Process the 0-simplex in the simplex traversal sequence and render it as a to-do item card. Each to-do item card is marked with the associated 1-simplex or 2-simplex by the topological association index. S322: Process the 1-simplex in the simplex traversal sequence, convert it into an arrow connection from the preceding task to the following task, and add a dependency strength label; S323: Process the 2-simplex in the simplex traversal sequence, trigger the visualization of "critical path highlighting" and "risk transmission warning", generate a risk transmission link description and embed it next to the corresponding to-do item card.

9. A multimodal meeting minutes automatic generation system, used to implement the multimodal meeting minutes automatic generation method according to any one of claims 1-8, characterized in that, The system includes: Heterogeneous graph construction module: used to integrate multilingual speech transcription and speaker identity alignment information to construct a directed heterogeneous graph of conference speeches; Task-dependent simplicial complex generation module: used to generate task-dependent simplicial complexes with execution semantic anchors based on the directed heterogeneous graph of conference speeches; Minutes and Action Plan Generation Module: Used for task-dependent simple complexes, execution context-aware structured rendering of minutes and intelligent action plan generation.

Images