A conference minutes generation method and system fusing visual layout and semantic analysis
By integrating visual layout and semantic analysis, the system accurately segments semantic line units and associates them with context, generating structured meeting minutes and to-do tasks. This solves the problem of automatically extracting key information from meeting records in existing technologies, achieving efficient information conversion and automated collaboration.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG GONGSHANG UNIVERSITY
- Filing Date
- 2026-02-28
- Publication Date
- 2026-07-03
AI Technical Summary
Existing technologies cannot efficiently and accurately extract and structure key decision and task information from unstructured meeting minutes, leading to difficulties in accumulating meeting knowledge assets and low collaboration efficiency.
By employing a method that integrates visual layout and semantic analysis, a deep learning model is used to parse meeting images, accurately segment semantic line units, and an information extraction model is used to perform contextual association across semantic line units, generating structured meeting minutes and to-do lists.
It enables end-to-end automated conversion from unstructured meeting minutes to structured knowledge assets, improving information utilization efficiency and collaboration automation, eliminating information transmission gaps, and enhancing team collaboration efficiency.
Smart Images
Figure CN122334211A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence and information processing technology, specifically relating to a method and system for generating meeting minutes that integrates visual layout and semantic analysis. Background Technology
[0002] In modern enterprise collaboration, meetings are a core scenario for decision-making and information synchronization. Currently, the recording and organization of meeting content mainly relies on the following methods:
[0003] Manual recording: A designated person manually records summaries, resolutions, and tasks to be done during the meeting. This method relies heavily on the recorder's concentration and summarizing ability, and is prone to missing key information in long or fast-paced meetings. Furthermore, post-meeting organization is time-consuming and laborious, resulting in information loss.
[0004] Speech-to-text: This method uses automatic speech recognition technology to generate a full transcript of the meeting. While it records all speeches, the resulting text is lengthy, loosely structured, lacks intuitive distinction between speaker roles and speaking order, and fails to effectively identify and extract structured information such as resolutions, responsibilities, and time commitments scattered throughout the dialogue. Participants still need to manually read through the long text to locate key points, resulting in limited efficiency improvements.
[0005] Plain text-based analysis tools: Some tools can extract keywords or provide simple summaries of conference transcripts, but they process linear text and cannot effectively understand and utilize the inherent visual structure and contextual relationships in conference dialogues. For example, they cannot identify the alternation of different speakers, emphasis highlighting, or accurately associate "resolution-responsible person-time" with the order of speaking rounds from dialogue screenshots or layouts.
[0006] Therefore, the main drawback of existing technologies is that they cannot efficiently and accurately extract and structure key decision and task information from unstructured, visually rich meeting records (such as image screenshots), resulting in difficulties in accumulating meeting knowledge assets and low collaboration efficiency. Summary of the Invention
[0007] To address the shortcomings of existing technologies and transform unstructured meeting minutes into structured knowledge assets that can directly drive work, thereby improving information utilization efficiency and automating collaboration, this invention adopts the following technical solution:
[0008] A method for generating meeting minutes that integrates visual layout and semantic analysis includes the following steps:
[0009] Retrieve image records containing historical meeting conversations;
[0010] Image records are segmented into row-level semantic units by fusing visual layout and semantic analysis, resulting in multiple semantically independent semantic row units arranged according to the speaking sequence;
[0011] Based on the long-range context formed by the semantic line unit sequence, the information extraction model is used to understand and extract at least one piece of structured key information formed in the historical meeting dialogue;
[0012] Based on the extracted structured key information, structured meeting minutes documents and to-do lists are automatically generated.
[0013] Furthermore, the image record can be a screenshot of a conversation, a photo of a whiteboard, or a photograph of a document, etc., which contains visual information that carries multiple rounds of conversation through text layout, format, color, font, etc., and the visual information is preprocessed, including but not limited to operations such as format unification, resolution adjustment, and distortion correction.
[0014] The line-level semantic unit segmentation involves visual layout analysis of the image record to distinguish different speaker blocks and their visual features, identifying text in the image record to obtain semantic segmentation features (including punctuation marks, paragraph start marks, and transition conjunctions), and then fusing the visual features with the semantic segmentation features to perform joint semantic boundary judgment in order to accurately segment multiple semantically independent line units.
[0015] Compared to existing technologies that rely solely on OCR or speech-to-text, failing to utilize visual layout information in meeting images and resulting in semantic units often containing multiple speakers or lacking complete semantics, this invention integrates visual layout analysis and semantic boundary judgment. While recognizing text content, it accurately distinguishes visual features such as speaker blocks, indentation levels, and font formats, and combines these with semantic features like punctuation and conjunctions for joint boundary determination. The resulting "semantic line units" visually correspond to a single speaker while maintaining semantic integrity, laying a clean and structured data foundation for subsequent high-precision information extraction and fundamentally avoiding semantic ambiguity caused by mixed units.
[0016] Furthermore, the semantic line unit refers to the basic processing unit obtained by fusing visual layout and semantic analysis to segment the conference dialogue image. Each semantic line unit visually corresponds to a continuous text region in the image and semantically represents a complete speech segment (such as a sentence, a semantic group, or a dialogue round), and is arranged according to the speaking sequence. Each semantic line unit contains the segmented sub-image, the identified text content, the timing information, and the optional speaker identifier.
[0017] Furthermore, the visual layout analysis utilizes a deep learning-based visual document understanding model to analyze the image, identify different text blocks, spatial relationships between blocks, and visual features (such as font size, color, indentation, and bullet points) to correspond to the speaking areas of different speakers.
[0018] Furthermore, the joint semantic boundary judgment and segmentation, within each identified text block, integrates visual line features with semantic segmentation features obtained through optical character recognition to determine the semantic integrity of each line or segment of text, and finally outputs a sequence of semantic line units arranged in chronological order. Each semantic line unit represents a semantically complete speech fragment, including the segmented sub-image, the corresponding OCR-recognized text, and associated with the speaker's identifier (such as the avatar or name tag parsed from the layout) and temporal information.
[0019] Furthermore, the structured key information includes a group of related information consisting of resolutions, responsible parties, and deadlines; the information extraction model obtains multiple semantic line units generated by segmenting meeting dialogue images and their corresponding visual and textual features to form a training sample set; the information extraction model is fine-tuned using the training sample set so that it learns to jointly identify key entities, actions, and their cross-semantic line unit relationships from the features fused from visual and textual data; it identifies key entities (such as personnel identifiers, time expressions, and project identifiers) and action keywords (such as verbs indicating task allocation and commitment) of predefined categories in the semantic line unit text; based on the speaking sequence and semantic coherence of the semantic line units, the key entities and action keywords are associated and merged within the context of multiple associated semantic line units to form a group of related information.
[0020] Existing keyword extraction technologies can only identify isolated entities and cannot understand the logical connections scattered across multiple rounds of dialogue. This invention introduces a cross-semantic line unit context association mechanism into the information extraction model, constructing a dynamic context window using the speaking sequence and semantic coherence to intelligently associate and merge "resolution matters," "responsible parties," and "time nodes" scattered across different units. This relation extraction capability based on dialogue logic significantly improves the completeness and accuracy of information extraction compared to pure text or rule-based methods, truly achieving a leap from "recognizing text" to "understanding intent."
[0021] Furthermore, the information extraction model is built on a multimodal encoder, collecting meeting dialogue images and segmenting them into semantic line units. Each semantic line unit is labeled with key entities (personnel, time, project) containing personnel, time, and project information, action keywords, and decision-responsible person-time association groups and their relationship types across multiple semantic line units. The pre-trained information extraction model is fine-tuned using labeled data, optimized by minimizing a joint loss function. This loss function includes, but is not limited to, entity recognition loss (for accurate entity type identification), action classification loss (for accurate action intent classification), and relationship classification loss (for determining the semantic relationship between entities and actions). The multimodal fusion layer of the information extraction model employs an early fusion strategy, concatenating or cross-attention interactions between the visual feature vectors of the semantic line unit images and the corresponding text embedding vectors before inputting them to the encoder. This allows the information extraction model to learn a unified representation of visual and textual features from the bottom layer, adapting to the task of jointly understanding and extracting semantic associations from the context of meeting dialogues.
[0022] Furthermore, the information extraction model identifies key entities and action keywords of predefined categories in the text from the semantic line unit sequence, and analyzes the semantic relationship between entities and actions within a context window composed of multiple related semantic line units based on the speaking sequence of the semantic line units and the encoded semantic coherence representation.
[0023] Furthermore, the automatic generation of the structured meeting minutes document and the to-do list involves filling the associated information group according to a preset template format to generate meeting minutes chapters that include resolutions, responsible parties, and deadlines; filtering items with future deadlines from the associated information group and automatically creating to-do tasks that include task content, responsible persons, and deadlines, which can be output to a specified file or synchronized to the task management system.
[0024] The structured, related information groups output by this invention can be directly mapped to a standard meeting minutes format, and items containing future deadlines can be automatically filtered to generate to-do tasks. These tasks can be synchronized to a project management system or team calendar via API, achieving automated connection between meeting decisions and work execution, completely eliminating information transmission gaps, and improving team collaboration efficiency.
[0025] A meeting minutes generation system integrating visual layout and semantic analysis includes an image receiving module, a semantic segmentation module, an information extraction module, and a minutes generation module. It employs the aforementioned meeting minutes generation method integrating visual layout and semantic analysis to perform image recording acquisition, semantic line unit generation, structured key information extraction, and meeting minutes generation.
[0026] The advantages and beneficial effects of this invention are as follows:
[0027] This invention fundamentally improves input quality by performing deep intelligent analysis on images containing historical dialogues, using high-precision unit segmentation based on visual-semantic fusion, and organically integrating visual layout information with textual semantics. Furthermore, it overcomes the limitations of isolated keyword extraction by accurately associating key information based on long-range context, achieving end-to-end structured output and realizing a business loop from meeting to execution. Its deep analysis capabilities through modal fusion approach a level of intelligence close to human understanding, automatically extracting and associating core decisions, responsible parties, and deadlines. This transforms unstructured meeting records into structured knowledge assets that can directly drive work, significantly improving information utilization efficiency and the level of collaborative automation. Attached Figure Description
[0028] Figure 1 This is a flowchart of a method according to an embodiment of the present invention.
[0029] Figure 2 This is a schematic diagram of the row-level semantic unit segmentation process in an embodiment of the present invention.
[0030] Figure 3 This is a schematic diagram of the key information extraction model training process in an embodiment of the present invention.
[0031] Figure 4 This is a schematic diagram of the context-based key information association process in an embodiment of the present invention.
[0032] Figure 5 This is a schematic diagram illustrating the generation of structured minutes and to-do tasks in an embodiment of the present invention.
[0033] Figure 6 This is a schematic diagram of the system structure according to an embodiment of the present invention. Detailed Implementation
[0034] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.
[0035] like Figure 1 As shown, this invention proposes a method for generating meeting minutes that integrates visual layout and semantic analysis, including the following steps:
[0036] Step S1: Receive image records containing historical meeting dialogues; the image records carry visual information of multi-turn dialogues distinguished by text layout and format.
[0037] Step S1 is executed by the image receiving module. The image recordings are typically screenshots of conversations, whiteboard photos, or document scans taken during or after a meeting. These images contain multi-turn conversation content distinguished by visual information such as text layout, format, color, and font. The image receiving module is responsible for receiving and preprocessing these images, such as performing operations like format standardization, resolution adjustment, and distortion correction, to facilitate subsequent processing.
[0038] Step S2: Perform row-level semantic unit segmentation on the image record by fusing visual layout and semantic analysis to obtain multiple semantically independent semantic row units arranged according to the speaking time sequence.
[0039] Line-level semantic unit segmentation, which integrates visual layout and semantic analysis, first performs visual layout analysis to distinguish different speaker blocks and their visual features. Then, it integrates visual features with semantic segmentation features obtained through optical character recognition (OCR) of the text (such as punctuation marks, paragraph start markers, and transitional conjunctions). Finally, it performs joint semantic boundary judgment to achieve precise segmentation. A semantic line unit is the basic processing unit obtained by segmenting a conference dialogue image through the integration of visual layout and semantic analysis. Each semantic line unit visually corresponds to a continuous text region in the image and semantically represents a complete speech segment (such as a sentence, a semantic group, or a dialogue round), arranged according to the speaking sequence. Each unit contains the segmented sub-image, the identified text content, temporal information, and an optional speaker identifier.
[0040] Step S2 is executed by the semantic segmentation module, and specifically includes the following two sub-steps:
[0041] Step S2.1: Visual Layout Analysis: Utilizing deep learning-based visual document understanding models (such as the LayoutLM series, DONUT, etc.), the image is analyzed to identify different text blocks, spatial relationships between blocks, and visual features (such as font size, color, indentation, and bullet points). In conference dialogue images, this typically corresponds to accurately distinguishing the speaking areas of different speakers. The layout analysis unit performs this task.
[0042] Step S2.2: Joint Semantic Boundary Judgment and Segmentation: Within each identified text block, visual line features (such as line spacing, alignment, and indentation levels) are fused with semantic segmentation features obtained through optical character recognition (especially punctuation marks such as periods, question marks, and exclamation marks, as well as transitional or initiational conjunctions such as "but" and "next") to determine the semantic integrity of each line or paragraph. The boundary judgment unit performs this joint judgment operation, ultimately outputting a sequence of semantic line units arranged in chronological order. Each semantic line unit represents a semantically complete speech fragment, containing the segmented sub-image, the corresponding OCR-recognized text, and associated with a speaker identifier (such as an avatar or name tag parsed from the layout) and temporal information.
[0043] In embodiments of the present invention, such as Figure 2 As shown, the row-level semantic unit segmentation process includes the original meeting image, the visual layout parsing result (block division), and the segmented semantic row unit sequence, intuitively demonstrating the conversion process from image to semantic unit.
[0044] Existing technologies rely solely on OCR or speech-to-text, failing to utilize visual layout information within meeting images. This results in semantic units often containing multiple speakers or lacking complete semantic meaning. This invention integrates visual layout analysis with semantic boundary judgment. While recognizing text content, it accurately distinguishes visual features such as speaker blocks, indentation levels, and font formats, and combines these with semantic features like punctuation and conjunctions for joint boundary determination. The resulting "semantic line units" visually correspond to a single speaker while maintaining semantic integrity, laying a clean and structured data foundation for subsequent high-precision information extraction and fundamentally avoiding semantic ambiguity caused by mixed units.
[0045] Step S3: Based on the long-range context formed by the semantic line unit sequence, understand it through the information extraction model and extract at least one piece of structured key information formed in the historical meeting dialogue. The structured key information includes a group of related information consisting of resolutions, responsible parties and deadlines.
[0046] Specifically, the information extraction model acquires multiple semantic line units generated from the segmentation of meeting dialogue images and their corresponding visual and textual features to form a training sample set. The training sample set is used to fine-tune the pre-trained multimodal understanding model, enabling the multimodal understanding model to learn to jointly identify key entities, actions, and their cross-semantic line unit relationships from the features fused from visual and textual data. It identifies key entities (such as personnel identifiers, time expressions, and project identifiers) and action keywords (such as verbs indicating task allocation and commitment) of predefined categories in the semantic line unit text. Based on the speaking sequence and semantic coherence of the semantic line units, the key entities and action keywords are associated and merged within the context of multiple associated semantic line units to form associated information groups.
[0047] Step S3 is executed by the information extraction module, the core of which is a specially designed and finely tuned multimodal understanding model. Its input is the semantic row unit sequence (containing both image and text modalities) obtained in the previous step, and specifically includes the following steps:
[0048] Step S3.1: Model Building and Training; The information extraction model can be built based on a multimodal encoder with a Transformer architecture (such as VL-BERT, UNITER). The model training process is as follows:
[0049] Sample construction: Collect a large number of meeting dialogue images, cut them into semantic line units through step S2, and manually label the key entities (personnel, time, project), action keywords, and "resolution-responsible person-time" association groups and their relationship types (such as "assignment" and "commitment to") across multiple semantic line units.
[0050] Model fine-tuning: The pre-trained model is fine-tuned using the labeled data described above. The model is optimized by minimizing a joint loss function, which typically includes: entity recognition loss (for accurate entity type identification), action classification loss (for accurate action intent classification), and relation classification loss (for determining the semantic relationship between entities and actions). The model's multimodal fusion layer can employ an early fusion strategy, concatenating or cross-attention interactions between the visual feature vectors of the semantic row unit images and the corresponding text embedding vectors before input to the encoder. This allows the model to learn a unified representation of visual and textual features from the bottom layer, thereby adapting to the task of jointly understanding and extracting semantic associations from the context of a conference dialogue. Figure 3 This training process is demonstrated.
[0051] Step S3.2: The information extraction process is as follows:
[0052] Step S3.2.1: Identify Units: For the input semantic line unit sequence, the model uses its encoding ability to identify key entities of predefined categories in the text (such as “Zhang San”, “next Friday”, “project A starts”) and action keywords (such as “responsible”, “promise to complete”, “agree”).
[0053] Step S3.2.2: Association Unit: Based on the speaking sequence of semantic line units and the encoded semantic coherence representation, the model analyzes the semantic relationships between entities and actions within a context window (e.g., the first 3 and the last 2 semantic line units) composed of multiple associated semantic line units. For example, after encoding the sequence of semantic line units, the model can identify the "assignment-deadline" relationship between "starting project A" (resolution item), "Zhang San" (responsible entity), and "before next Friday" (deadline), thereby associating and merging these scattered pieces of information to form a structured associated information group: {resolution item: "starting project A", responsible entity: "Zhang San", deadline: "before next Friday"}. The model can extract one or more such associated information groups. Figure 4 This demonstrates the process of extracting and associating information based on context.
[0054] Existing keyword extraction technologies can only identify isolated entities and cannot understand the logical connections scattered across multiple rounds of dialogue. This invention introduces a cross-semantic line unit contextual association mechanism into the information extraction model, utilizing the speaking sequence and semantic coherence to construct a dynamic context window. This intelligently associates and merges "resolution matters," "responsible parties," and "time nodes" scattered across different units. This dialogue-logic-based relational extraction capability significantly improves the completeness and accuracy of information extraction compared to pure text or rule-based methods, truly achieving a leap from "recognizing text" to "understanding intent."
[0055] This invention creatively integrates visual document understanding with natural language processing, making full use of the visual structural information (such as speaker blocks, emphasis formatting, and indentation levels) missing in the plain text transcription of meeting images. This makes the parsing of unstructured meeting content closer to the way humans understand reading, and it balances parsing accuracy and robustness in complex scenarios while automating the process.
[0056] Step S4: Based on the extracted structured key information, automatically generate structured meeting minutes documents and a list of tasks to be done.
[0057] Specifically, the automatic generation of structured meeting minutes documents and to-do lists includes: filling in related information groups according to a preset template format to generate meeting minutes chapters containing resolutions, responsible parties, and deadlines; and automatically creating to-do tasks containing task content, responsible persons, and deadlines for items in related information groups that have future deadlines, which can be exported to a specified file or synchronized to the task management system.
[0058] Step S4 is executed by the minutes generation module, and specifically includes the following steps:
[0059] Step S4.1: Generate meeting minutes document: The module has a built-in or connection to a preset meeting minutes template. All extracted related information groups will be automatically filled into the "Meeting Resolutions" or "Action Plan" sections according to the template format, generating a clear, formal minutes document (such as Markdown, Word, or PDF format) that includes specific matters, responsible persons, and times.
[0060] Step S4.2: Generate and synchronize the to-do list: Iterate through all related information groups and filter out items that contain future time nodes (deadlines). For each item, automatically create a to-do task, with the task content being a resolution matter, the responsible person being identified, and a deadline being set. These tasks can be generated as independent to-do list files (such as CSV or ICS calendar files) or directly synchronized to a specified task management system (such as Jira, Trello, or Asana) or team calendar via API, enabling automatic workflow integration. Figure 5 It demonstrates the process of generating and outputting related information groups, structured meeting minutes, and to-do tasks.
[0061] The structured, related information groups output by this invention can be directly mapped to a standard meeting minutes format, and items containing future deadlines can be automatically filtered to generate to-do tasks. Synchronization with a project management system or team calendar via API achieves automated connection between meeting decisions and work execution, completely eliminating information transmission gaps and improving team collaboration efficiency.
[0062] Through the above process, this invention achieves an end-to-end automated conversion process from unstructured meeting images to structured knowledge assets that can be directly used for execution and tracing, such as... Figures 1 to 3 As shown, this process covers the entire chain from image input, intelligent parsing, information association to output, effectively solving the problems in the background technology.
[0063] like Figure 6 As shown, the present invention also provides a meeting minutes generation system that integrates visual layout and semantic analysis, including an image receiving module, a semantic segmentation module, an information extraction module, and a minutes generation module.
[0064] The image receiving module is used to receive image records containing historical meeting conversations;
[0065] A semantic segmentation module, connected to the image receiving module, is used to perform row-level semantic unit segmentation on the image record by fusing visual layout and semantic analysis, to obtain multiple semantically independent semantic row units arranged according to the speaking time sequence;
[0066] The semantic segmentation module further includes a layout parsing unit and a boundary judgment unit, which are used for visual layout parsing and joint semantic boundary judgment after feature fusion, respectively.
[0067] An information extraction module, connected to the semantic segmentation module, is used to understand and extract at least one piece of structured key information formed in the historical meeting dialogue based on the long-range context formed by the semantic line unit sequence through an information extraction model.
[0068] The information extraction module further includes an identification unit and an association unit, which are used for key entity identification and context-based association merging, respectively;
[0069] The minutes generation module, connected to the information extraction module, is used to automatically generate structured meeting minutes documents and a list of tasks to be done based on the extracted structured key information.
[0070] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for generating meeting minutes that integrates visual layout and semantic analysis, characterized in that: Retrieve image records containing historical meeting conversations; The image recording is segmented into row-level semantic units by fusing visual layout and semantic analysis, resulting in multiple semantically independent semantic row units arranged according to the speaking sequence; Based on the long-range context formed by the semantic line unit sequence, an information extraction model is used to understand and extract at least one piece of structured key information formed in the historical meeting dialogue; Based on the extracted structured key information, structured meeting minutes documents and a list of tasks to be done are automatically generated. 2.The conference minutes generation method of fusing visual layout and semantic analysis according to claim 1, characterized in that: The image recordings are visual information from multi-turn dialogues distinguished by text layout and format. The row-level semantic unit segmentation involves visual layout analysis of the image record to distinguish different speaker blocks and their visual features, identification of text in the image record to obtain semantic segmentation features, and fusion of the visual features and the semantic segmentation features to perform joint semantic boundary judgment in order to segment out multiple semantically independent semantic row units. 3.The conference minutes generation method of claim 2, wherein: The semantic line unit visually corresponds to a continuous text region in an image, semantically represents a complete speech segment, and is arranged according to the speaking sequence.
4. The method of claim 2, wherein: The visual layout analysis utilizes a deep learning-based visual document understanding model to analyze images, identify different text blocks, spatial relationships between blocks, and visual features to correspond to the speaking areas of different speakers.
5. The method of claim 2, wherein: The joint semantic boundary judgment and segmentation, within each identified text block, integrates visual line features with semantic segmentation features obtained through optical character recognition to determine the semantic integrity of each line or segment of text, and finally outputs a sequence of semantic line units arranged in chronological order. Each semantic line unit represents a semantically complete speech fragment, including the segmented sub-image, the corresponding recognized text, and associated with the speaker's identifier and temporal information.
6. The meeting minutes generation method integrating visual layout and semantic analysis according to claim 2, characterized in that: The structured key information includes a group of related information consisting of resolution items, responsible parties, and deadlines. The information extraction model obtains multiple semantic line units and their corresponding visual and textual features to form a training sample set. The training sample set is used to fine-tune the information extraction model so that it can learn to jointly identify key entities, actions, and their cross-semantic line unit relationships from the features fused from visual and textual data. The model identifies key entities and action keywords of predefined categories in the semantic line unit text. Based on the speaking sequence and semantic coherence of the semantic line units, the key entities and action keywords are associated and merged within the context of multiple associated semantic line units to form a group of related information.
7. The method of claim 6, wherein: The information extraction model is built on a multimodal encoder. It collects meeting dialogue images and segments them into semantic line units. Each semantic line unit is labeled with key entities (personnel, time, project) containing personnel, time, and project information, action keywords, and decision-responsible person-time association groups and their relationship types across multiple semantic line units. The pre-trained information extraction model is fine-tuned using labeled data and optimized by minimizing a joint loss function, which includes, but is not limited to, entity recognition loss, action classification loss, and relationship classification loss. The multimodal fusion layer of the information extraction model adopts an early fusion strategy, concatenating or cross-attention interaction between the visual feature vectors of the semantic line unit images and the corresponding text embedding vectors before inputting them into the encoder. This enables the information extraction model to learn a unified representation of visual and textual features from the bottom layer, adapting to the task of jointly understanding and extracting semantic associations from the context of meeting dialogues.
8. The method of claim 7, wherein: The information extraction model identifies key entities and action keywords of predefined categories in the text from the semantic line unit sequence. Based on the speaking sequence of the semantic line units and the encoded semantic coherence representation, it analyzes the semantic relationship between entities and actions within a context window composed of multiple related semantic line units.
9. The method of claim 6, wherein: The automatic generation of the structured meeting minutes document and the to-do list involves filling the associated information group according to a preset template format to generate meeting minutes chapters that include resolutions, responsible parties, and deadlines; and automatically creating to-do tasks that include task content, responsible persons, and deadlines by filtering items with future deadlines from the associated information group.
10. A conference minutes generation system fusing visual layout and semantic analysis, comprising an image receiving module, a semantic cutting module, an information extracting module, a minutes generation module, characterized in that: The meeting minutes generation method, which integrates visual layout and semantic analysis as described in any one of claims 1 to 9, is used to perform image recording acquisition, semantic line unit generation, structured key information extraction, and meeting minutes generation.