Multi-modal conference summary generation method and system based on large language model

Abstract

The invention discloses a multi-modal conference summary generation method and system based on a large language model, and relates to the technical field of artificial intelligence, and the method comprises the steps: receiving conference basic information uploaded by a user and a conference file containing a picture and an audio; the audio file is transferred into a structured text with a timestamp through an ASR voice transfer technology, and the picture file is recognized as an editable text through an OCR technology; conference basic information, a structured text and an editable text are integrated and converted into a structured conference summary first draft, then text cleaning and content error correction are performed on the first draft in combination with a large language model and predefined logic, participant information is extracted, and finally a customized instruction is input into the large language model through a prompt project to obtain a complete conference summary. And performing accurate filling according to a structured field of the predefined Word template to generate an editable conference summary Word document. The conference summary can be automatically output, and the writing efficiency of the conference summary is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a multi-modal conference minutes generation method and system based on a large language model. BACKGROUND

[0002] With the rapid development of information technology, the management level and operation efficiency of the manufacturing industry are continuously improving, and the communication and collaboration needs within enterprises are increasingly complex. In this context, conferences as an important means of information exchange and decision-making within enterprises, the accuracy and timeliness of their minutes have a crucial impact on the operation of enterprises. Traditional conference minutes recording methods mainly rely on manual recording or basic audio equipment, which have exposed many shortcomings in practical application, such as low recording efficiency, information omission, time-consuming and laborious post-processing, etc., which seriously restrict the improvement of enterprise management efficiency and the scientific nature of decision-making.

[0003] Firstly, manual recording as the most common method of conference minutes recording, although it can capture the core content of the meeting, it depends on the professional level and experience of the recording personnel, and is prone to incomplete recording or understanding bias. In addition, during the meeting, the frequent recording behavior of the recording personnel may distract their attention, affecting the overall effectiveness and interactivity of the meeting. Especially in the manufacturing industry involving technical discussions and complex decision-making, manual recording is difficult to fully and accurately reflect the meeting content, resulting in insufficient information transmission and decision-making basis.

[0004] Secondly, although audio recording can preserve the meeting content more comprehensively, in practical application, the transcription process is time-consuming and prone to errors. Especially in the case of multiple people speaking at the same time or the presence of background noise, the accuracy of audio transcription cannot be guaranteed, increasing the workload of post-processing and proofreading. In addition, the management and storage of audio files also pose a considerable challenge, especially in enterprise environments involving a large number of conference recordings, the retrieval and use efficiency of audio files is low, making it difficult to meet the demand for quick access and analysis of meeting content.

[0005] Furthermore, basic electronic tools such as spreadsheets and word processing software, although they improve recording efficiency to some extent, still rely on manual input and cannot effectively integrate multiple information forms (such as text, pictures, audio, etc.). These tools have great limitations in information processing and organization, especially when faced with complex meeting content and diverse information sources, they are difficult to achieve efficient and accurate conference minutes generation, affecting the comprehensiveness and readability of information.

[0006] With the rapid development of artificial intelligence technologies, especially Natural Language Processing (NLP), Automatic Speech Recognition (ASR), Optical Character Recognition (OCR), and Large-Scale Language Models (LLM), intelligent meeting minutes generation technology has gradually become a research and application hotspot. Large-Scale Language Models (LLM), with their powerful natural language understanding and generation capabilities, can perform semantic analysis and content generation on massive amounts of text data. These models perform excellently in tasks such as automatically generating meeting minutes and extracting summaries, significantly improving the accuracy and readability of the minutes. However, LLM still needs further optimization in understanding and generating specific industry terminology and technical details to ensure the professionalism and accuracy of the minutes content. Furthermore, existing multimodal information processing technologies often rely on different independent modules when integrating different types of data, lacking an effective integration mechanism, resulting in low collaborative efficiency in information processing and making it difficult to achieve comprehensive and unified minute generation.

[0007] While intelligent meeting minutes generation technology has made some progress in both theory and application, some shortcomings still exist in practical applications. First, existing systems often rely on different independent modules when processing various information formats, lacking an effective integration mechanism. This leads to low collaborative efficiency in information processing and makes it difficult to achieve comprehensive and unified minutes generation. Second, different enterprises and industries have varying meeting needs, and existing systems have limitations in adapting to different scenarios and requirements, making it difficult to provide highly customized solutions and affecting the system's universality and application effectiveness. Furthermore, although LLM performs well in natural language processing, further optimization is needed in understanding and generating specific industry terminology and technical details to ensure the professionalism and accuracy of the minutes content. Summary of the Invention

[0008] This invention addresses the needs and shortcomings of current technological development by providing a multimodal meeting minutes generation method and system based on a large language model, thereby automating meeting minutes generation, reducing manual workload, and improving the efficiency of meeting minutes writing.

[0009] Firstly, the present invention provides a method for generating multimodal meeting minutes based on a large language model, and the technical solution adopted to solve the above-mentioned technical problems is as follows:

[0010] A method for generating multimodal meeting minutes based on a large language model includes the following steps:

[0011] S1. Receive basic meeting information uploaded by users and record it in the Meeting table of the database; receive meeting files containing images and audio, store them in a specified directory on the server, and record the file metadata in the database at the same time;

[0012] S2. Using ASR speech-to-text technology, the audio file received in step S1 is transcribed into structured text with a timestamp. Using OCR technology, the image file received in step S1 is recognized as editable text.

[0013] S3. Integrate the basic meeting information uploaded by the user, the structured text and editable text output in step S2 into a draft of structured meeting minutes. Then, combine the large language model and predefined logic to perform text cleaning and content correction on the draft of structured meeting minutes, extract the information of the participants, and finally input customized instructions into the large language model through the prompting process. The large language model then outputs a complete meeting minutes that meets the format requirements.

[0014] S4. The complete meeting minutes output by the large language model are accurately filled in according to the structured fields of the predefined Word template. The text format is automatically matched during the filling process, and finally an editable meeting minutes Word document that conforms to the specifications is generated, which can be directly downloaded, printed or shared.

[0015] Optionally, step S1 specifically includes the following operations:

[0016] S1.1 Utilize the FastAPI framework to build an efficient and scalable API interface. This API interface is used to receive basic meeting information and meeting files containing images and audio uploaded by users via HTTP POST requests.

[0017] S1.2 Record the basic meeting information into the Meeting table in the database;

[0018] S1.3 For meeting files uploaded by users, a multi-factor verification mechanism is used to check the corresponding format of the file header, size and content. If the verification fails, an error message is returned and the result is recorded to ensure that the file is complete and valid.

[0019] S1.4 Generate a unique UUID for each valid meeting file, and then store it in a specified directory under the server's distributed storage architecture according to file type;

[0020] S1.5 Extract the metadata of the meeting documents and store it in the database.

[0021] Optionally, step S2 is performed to transcribe the audio file received in step S1 into structured text with a timestamp using ASR speech-to-text technology. Specific operations include:

[0022] S2.1.1 Based on the unique UUID of the audio file, the target audio file is searched and located in the preset specified directory, and the complete storage path and file name of the audio file are extracted to ensure that the subsequent transcription operation can accurately call the file;

[0023] S2.1.2 Call the ASR speech-to-text engine, load the located audio file, and convert the speech content in the audio into text through speech feature recognition and semantic parsing. Then, mark the start and end timestamps of each sentence in the text in the original audio to form a structured text with timestamps.

[0024] S2.1.3 Save the generated timestamped structured text to a preset specified file. During the saving process, perform preliminary formatting, including: unifying the text encoding format, standardizing the timestamp display format, removing redundant spaces / garbled characters from the text, and sorting the text content according to the audio time order to ensure the readability and reusability of the structured text.

[0025] Optionally, step S2 is performed to convert the image file received in step S1 into editable text using OCR technology. Specific operations include:

[0026] S2.2.1 Based on the unique UUID of the image file, the target image file is retrieved and read from the preset specified directory. Then, preprocessing operations such as rotation, noise reduction, grayscale conversion, cropping, edge detection and contrast enhancement are performed in sequence to ensure that the text in the image is clearly visible, thus ensuring the accuracy of subsequent OCR recognition.

[0027] S2.2.2 The preprocessed image is sent to the OCR service, which performs text recognition and returns the recognized editable text. The core model used by the OCR service is an architecture that combines convolutional neural networks and recurrent neural networks, which can effectively adapt to complex text layouts and various fonts.

[0028] S2.2.3 Save the editable text corresponding to a single image returned by the OCR service to the specified file directory in a preset format; then, concatenate the editable text corresponding to all images into a complete string in the order of the original images, and associate the complete string with the corresponding image UUID and store it in the database to realize the structured storage and subsequent retrieval of text data.

[0029] Optionally, step S3 is performed to clean and correct the text of the draft structured meeting minutes, specifically including:

[0030] Text cleaning of the initial draft of structured meeting minutes was performed by removing redundant text, filtering stop words, and processing special characters.

[0031] By using OCR text correction, customized text correction, name recognition, general text correction and mapping replacement, combined with character count verification, the draft of structured meeting minutes is corrected.

[0032] Alternatively, proceed to step S3 to extract attendee information, as detailed below:

[0033] The mapping relationship between attendees and departments is loaded from a predefined JSON mapping file containing the names of all attendees and their corresponding department information, thereby ensuring the accuracy and consistency of subsequent information extraction.

[0034] By combining regular expressions and keyword matching, the names of participants and their departments are first identified from the initial draft of the structured meeting minutes. Then, the identified information is matched and associated with the context to ensure that the extraction results are accurate.

[0035] The extracted attendee names and department names are compared with the loaded mapping relationship through a matching algorithm to automatically confirm the department to which each attendee belongs, thereby generating structured data to facilitate subsequent applications and displays.

[0036] Based on the structured data obtained after information extraction and mapping, it is converted into standardized natural language descriptive sentences for meeting participants, ensuring that the meeting minutes are both professional and readable.

[0037] Alternatively, step S3 can be executed, prompting the engineering team to input customized instructions into the large language model, which will then output a complete meeting minutes that meets the format requirements. The specific operation is as follows:

[0038] The structured meeting minutes draft after text cleaning and content correction, along with extracted participant information, are used as basic inputs, combined with customized instructions input into the large language model.

[0039] The above-mentioned draft structured meeting minutes text was segmented according to the rule of "no segmentation for less than 1500 characters, segmentation into 2 parts for 1500-5000 characters, and segmentation into 3 parts for more than 5000 characters". When segmenting, consecutive characters were avoided, and symbols were first found within 50 characters to the left and right of the segmentation point as the segmentation position.

[0040] Semantic segmentation is performed on each slice. The large language model is called and customized instructions are input to divide the text into paragraphs according to the text semantics of each slice, forming logically coherent segmented text.

[0041] Based on each slice, generate basic meeting information, concatenate the segmented text of each slice with the information of the participants to form a complete input, pass it into the large language model and input customized instructions, and the large language model summarizes and outputs the basic meeting information of the corresponding slice;

[0042] The next step plan is generated based on each slice. The text after each slice is segmented and the information of the participants is concatenated into a complete input, which is then fed into the large language model along with customized instructions. The large language model then summarizes and outputs the next step plan for the corresponding slice.

[0043] After processing a single slice, perform a meeting basic information merging operation to summarize the meeting basic information generated by all slices and form complete meeting basic information content covering the entire meeting process.

[0044] Execute the next step plan merging operation: First, summarize the next step plans generated by all slices, then concatenate the summary result with the segmented complete text and the participant information as the final input, pass it into the large language model and input customized instructions, and the large language model summarizes and outputs the complete next step plan that is globally unified and logically closed.

[0045] Perform data cleaning to remove any single "<" or ">" symbols from the final generated meeting minutes text, ensuring that the text format is correct and up-to-date.

[0046] Further optionally, step S4 specifically includes:

[0047] By predefining the Word template structure, using the Docx library to set uniform style parameters, and configuring placeholders, the format and positional foundation for the automated generation of meeting minutes is laid.

[0048] Based on key-value mapping, the meeting minutes content is mapped and filled into template placeholders. Natural language processing technology is used to segment paragraphs, and the format is adjusted using a Docx library or automated tools based on the Docx library to ensure that the content is accurate and the layout is easy to read.

[0049] The Docx library is used to create Word document objects, which are then filled with content and predefined styles. The documents are then saved to a specified path and can be downloaded, printed, or shared directly. Files are managed and traced in an orderly manner through directory storage and version control.

[0050] Secondly, the present invention provides a multimodal meeting minutes generation system based on a large language model, and the technical solution adopted to solve the above-mentioned technical problems is as follows:

[0051] A multimodal meeting minutes generation system based on a large language model, used to implement the multimodal meeting minutes generation method described in the first aspect, specifically includes:

[0052] The data receiving and storage module is used to receive basic meeting information uploaded by users and record it in the Meeting table of the database; it also receives meeting files containing images and audio, stores them in a specified directory on the server, and records the file metadata in the database.

[0053] The multimodal information processing module is used to transcribe received audio files into structured text with timestamps using ASR speech-to-text technology, and to recognize received image files into editable text using OCR technology.

[0054] The minutes generation module integrates the basic meeting information uploaded by the user, the structured text output by the multimodal information processing module, and the editable text into a draft of structured meeting minutes. Then, it combines the large language model and predefined logic to perform text cleaning and content correction on the draft of structured meeting minutes, and extract the information of the participants. Finally, it inputs customized instructions into the large language model through the prompting process, and the large language model outputs a complete meeting minutes that meets the format requirements.

[0055] The formatting module is used to accurately fill in the complete meeting minutes output by the large language model according to the structured fields of the predefined Word template. During the filling process, the text format is automatically matched, and finally an editable meeting minutes Word document that conforms to the specifications is generated, which can be directly downloaded, printed or shared.

[0056] The present invention provides a multimodal meeting minutes generation method and system based on a large language model, which has the following advantages compared with the prior art:

[0057] 1. This invention can automatically process the process from recording audio and images to outputting meeting minutes, greatly improving the efficiency of meeting minutes writing and significantly reducing manual workload;

[0058] 2. This invention is applicable to the manufacturing industry and can effectively solve the shortcomings of traditional meeting minutes recording methods and existing intelligent systems, meeting the needs of manufacturing enterprises for efficient and accurate meeting management; it can significantly reduce the workload of manual recording and sorting of meeting minutes, improve the efficiency and accuracy of meeting management, and provide more comprehensive and reliable basis for enterprise decision-making through intelligent data processing and analysis, with broad application prospects and significant economic benefits. Attached Figure Description

[0059] Appendix Figure 1 This is a flowchart of the method according to Embodiment 1 of the present invention;

[0060] Appendix Figure 2 This is a module connection block diagram of Embodiment 2 of the present invention. Detailed Implementation

[0061] To make the technical solution, the technical problem solved, and the technical effect of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments.

[0062] Example 1:

[0063] Combined with appendix Figure 1 This embodiment proposes a method for generating multimodal meeting minutes based on a large language model, which includes the following steps:

[0064] S1. Receive basic meeting information uploaded by users and record it in the Meeting table of the database; receive meeting files containing images and audio, store them in a specified directory on the server, and record the file metadata in the database.

[0065] This step specifically includes the following operations:

[0066] S1.1. The FastAPI framework is used to build an efficient and scalable API interface. This API interface is used to receive basic meeting information (including meeting ID, user ID, meeting title, participants, etc.) and meeting files containing images (such as PNG, JPEG) and audio (such as MP3, WAV) uploaded by users via HTTP POST requests.

[0067] S1.2 Record the basic meeting information (including meeting ID, user ID, meeting title, participants, etc.) into the Meeting table in the database;

[0068] S1.3 For meeting files uploaded by users, a multi-factor verification mechanism is used to check the corresponding format (e.g., for audio files, check whether the file format and encoding meet the requirements; for image files, verify the image resolution and format). If the verification fails, an error message is returned and the result is recorded to ensure that the file is complete and valid. Of course, the verification results of each upload are also recorded for subsequent auditing and problem investigation.

[0069] S1.4 Generate a unique UUID for each valid meeting file, and then store it in a specified directory under the server's distributed storage architecture according to the file type (such as audio and images). (For example, audio is stored in recording / raw / {UUID} / and images are stored in recording / ocr / pic / {UUID} / . This directory structure not only helps with the orderly management of files, but also facilitates subsequent file retrieval and processing.)

[0070] S1.5 Extract the metadata of the meeting files, such as file name, size, upload time, etc., and store them in the database.

[0071] S2. Using ASR (Automatic Speech Recognition) speech-to-text technology, the audio file received in step S1 is transcribed into structured text with a timestamp. Specific operations include:

[0072] S2.1.1 Based on the unique UUID of the audio file, the target audio file is searched and located in the preset specified directory, and the complete storage path and file name of the audio file are extracted to ensure that the subsequent transcription operation can accurately call the file;

[0073] S2.1.2. An ASR speech-to-text engine is pre-built using the open-source SenseVoiceSmall model. This engine supports multiple languages ​​and dialects and can efficiently and accurately recognize speech content.

[0074] The ASR speech-to-text engine is invoked, the located audio file is loaded, and the speech content in the audio is converted into text through speech feature recognition and semantic parsing. Each sentence in the text is labeled with its start and end timestamps in the original audio, forming a structured text with timestamps.

[0075] S2.1.3 Save the generated timestamped structured text to a preset specified file (such as recording / txt / {UUID} / transcribed.txt). During the saving process, perform preliminary formatting, including: unifying the text encoding format, standardizing the timestamp display format, removing redundant spaces / garbled characters from the text, and sorting the text content according to the audio time order to ensure the readability and reusability of the structured text.

[0076] This process ensures that speech content is accurately converted into text, providing foundational data for subsequent text analysis. To improve transcription accuracy, the ASR speech-to-text engine supports multiple transcriptions and result comparisons, automatically selecting the optimal result. Furthermore, the ASR engine integrates speech enhancement technology, performing noise suppression and echo cancellation on audio files to further improve transcription quality.

[0077] Simultaneously, using OCR technology, the image file received in step S1 is recognized as editable text. Specific operations include:

[0078] S2.2.1 Based on the unique UUID of the image file, the target image file is retrieved and read from the preset specified directory. Then, preprocessing operations such as rotation, noise reduction, grayscale conversion, cropping, edge detection and contrast enhancement are performed in sequence to ensure that the text in the image is clearly visible, thus ensuring the accuracy of subsequent OCR recognition.

[0079] S2.2.2 The preprocessed image is sent to the OCR service, which performs text recognition and returns the recognized editable text. The core model used by the OCR service is an architecture that combines convolutional neural networks (CNN) and recurrent neural networks (RNN), which can effectively adapt to complex text layouts and various fonts.

[0080] S2.2.3 Save the editable text corresponding to a single image returned by the OCR service to a specified file directory (e.g., recording / ocr / txt / {UUID} / {image name}.txt file) according to a preset format; then, concatenate the editable text corresponding to all images into a complete string in the order of the original images, and associate this complete string with the corresponding image UUID and store it in the database to realize the structured storage and subsequent retrieval of text data.

[0081] This operation converts text content in images into editable text, further enriching the multimodal information in meeting materials. To improve OCR accuracy, the OCR service supports multilingual recognition and character set expansion, enabling it to handle various image types. Furthermore, the OCR service integrates layout analysis functions, automatically recognizing paragraphs, headings, and lists in images to ensure structured and formatted recognition results.

[0082] S3. Integrate the basic meeting information uploaded by the user, the structured text and editable text output in step S2 into a draft of structured meeting minutes. Then, combine the Large Language Model (LLM) and predefined logic to perform text cleaning and content correction on the draft of structured meeting minutes, and extract the information of the participants. Finally, input customized instructions into the Large Language Model through the prompting process, and the Large Language Model (LLM) will output a complete meeting minutes that meets the format requirements.

[0083] This step involves text cleaning and content correction of the initial draft of the structured meeting minutes. The specific operations include:

[0084] Text cleaning of the initial draft of structured meeting minutes is performed through redundant text removal (such as irrelevant statements, repetitive content, and colloquial redundant expressions), stop word filtering (using intelligent word segmentation technology to segment the text using a predefined stop word list, removing irrelevant or frequently occurring meaningless words, and retaining key information; the stop word list covers common conjunctions, auxiliary words, modal particles, meaningless filler words, and redundant words specific to the meeting scenario, ensuring that only information valuable to the meeting minutes is retained in the text), and special character processing (such as removing meaningless symbols, standardizing format symbols, and correcting garbled characters).

[0085] Perform content error correction on the initial draft of the structured meeting minutes through OCR text error correction (correct errors in the text recognized by OCR. If the recognized text is garbled or completely illogical, return an empty string), customized text error correction (pre-train a large model according to project requirements and perform error correction on the transcribed text through the pre-trained large model), executor name recognition (correct possible misspellings of names in the text based on the names of project-related personnel given), general text error correction (use a general typo database to correct general text and ensure the accuracy and consistency of the text content), and mapping replacement (replace specific words and names according to preset mapping rules to ensure the consistency and accuracy of the text. For example, replace abnormal words with similar pronunciations with correct words, such as replacing "CC kao" with "SaySayCall" and "Haimo" with "Hairuo", etc.), and combine character count verification.

[0086] Execute step S3 to extract the information of the participants. The specific operations are as follows:

[0087] Load the mapping relationship between the participants and their departments from a predefined JSON mapping file that contains the names of all company participants and their corresponding department information, so as to ensure the accuracy and consistency of subsequent information extraction;

[0088] Adopt a combination of regular expressions and keyword matching. First, identify the names and department names of the participants from the initial draft of the structured meeting minutes, and then match and associate the identified information based on the context relationship to ensure the accuracy of the extraction results;

[0089] Compare the information of the names and department names of the participants extracted with the loaded mapping relationship through a matching algorithm, automatically confirm the department to which each participant belongs, and then generate structured data to facilitate subsequent applications and displays;

[0090] Based on the structured data obtained after information extraction and mapping, convert it into a standardized natural language description sentence of the participants in the meeting (such as "The participants include Zhang San, Li Si from the Technology Department, Wang Wu from the Marketing Department, etc."), ensuring the professionalism and readability of the meeting minutes content.

[0091] This step can ensure the accuracy and completeness of the participant information in the meeting minutes, improve the professionalism and readability of the minutes. To further improve the accuracy of information extraction, a manual review and confirmation mechanism is supported, allowing users to make manual adjustments and corrections based on the automatic extraction results.

[0092] In this step, customized instructions are input to the large language model (LLM) through prompt engineering, and the large language model (LLM) outputs a complete meeting minutes that meets the format requirements. The specific operations are as follows:

[0093] The structured meeting minutes draft after text cleaning and content correction, along with extracted participant information, are used as basic inputs, combined with customized instructions input into the Large Language Model (LLM).

[0094] The above-mentioned draft structured meeting minutes text was segmented according to the rule of "no segmentation for less than 1500 characters, segmentation into 2 parts for 1500-5000 characters, and segmentation into 3 parts for more than 5000 characters". When segmenting, consecutive characters were avoided, and symbols were first found within 50 characters to the left and right of the segmentation point as the segmentation position.

[0095] Semantic segmentation is performed on each slice. The Large Language Model (LLM) is invoked and customized instructions are input to divide the text into segments according to the text semantics of each slice, forming logically coherent segmented text.

[0096] Based on each slice, generate basic meeting information, concatenate the segmented text of each slice with the participant information to form a complete input, pass it to the Large Language Model (LLM) and input customized instructions, and the Large Language Model (LLM) summarizes and outputs the basic meeting information of the corresponding slice;

[0097] The next step plan is generated based on each slice. The text after each slice is segmented and the information of the participants is concatenated into a complete input, which is then fed into the Large Language Model (LLM) with customized instructions. The Large Language Model (LLM) summarizes and outputs the next step plan for the corresponding slice.

[0098] After processing a single slice, perform a meeting basic information merging operation to summarize the meeting basic information generated by all slices and form complete meeting basic information content covering the entire meeting process.

[0099] Execute the next step plan merging operation: First, summarize the next step plans generated by all slices, then concatenate the summary result with the segmented complete text and the participant information as the final input, pass it to the Large Language Model (LLM) and input customized instructions, and the Large Language Model (LLM) summarizes and outputs the complete next step plan that is globally unified and logically closed.

[0100] Perform data cleaning to remove any single "<" or ">" symbols from the final generated meeting minutes text, ensuring that the text format is correct and up-to-date.

[0101] S4. The complete meeting minutes output by the large language model are accurately filled in according to the structured fields of the predefined Word template. The text format is automatically matched during the filling process, and finally an editable meeting minutes Word document that conforms to the specifications is generated, which can be directly downloaded, printed or shared.

[0102] This step specifically includes the following operations:

[0103] By predefining the structure of a Word template (including the formatting of sections such as title, abstract, attendees, meeting content, and next steps), and using the Docx library to set uniform style parameters such as font, font size, and paragraph alignment, and configuring placeholders (for example, setting the title section to {title}, the abstract section to {abstract}, and the attendees section to {participants}, etc., to facilitate automated filling and replacement), the format and positional foundation for the automated generation of meeting minutes is laid.

[0104] Based on key-value mapping, the meeting minutes content is mapped and filled into template placeholders. Natural language processing technology is used to segment paragraphs, and the format is adjusted (such as setting bold headings, paragraph indentation, bulleted lists, etc.) through Docx library or Docx library-based automation tools to ensure that the content is accurate and the layout is easy to read.

[0105] Using the Docx library, Word document objects are created, and the content is filled in and predefined styles (such as uniformly setting the font, font size, color, and paragraph format) are applied. Then, the meeting minutes Word document is saved in a specified path (such as recording / docx / {UUID} / meeting minutes.docx), and can be directly downloaded, printed, or shared. At the same time, file management and traceability are achieved through directory storage and version control.

[0106] This step can be supplemented by: ① Standardizing the naming of meeting minutes Word documents based on the meeting title and date for easier file management and retrieval. For example, using a unified naming convention, such as "Meeting Minutes_Title_Date.docx", ensures the standardization and readability of filenames. ② Assigning a unique version number to each generated meeting minutes Word document to ensure document traceability and orderly version management. Specifically, recording the generation time, modifier, and modifications for each version in a database ensures manageability of document versions and the integrity of historical records. ③ Automatically sending meeting minutes Word documents to a designated email address or storing them in the enterprise document management system according to preset rules, simplifying the distribution process. ④ Ensuring document security and confidentiality through an access control mechanism, ensuring that only authorized users can access and download generated meeting minutes Word documents. For example, using a role-based access control (RBAC) policy to define access permissions for different users ensures the security and privacy protection of sensitive information.

[0107] Regarding the multimodal meeting minutes generation method in this embodiment, it should be further noted that the deployed system supports high-concurrency data upload and processing requests through the FastAPI framework and multithreading technology. Utilizing ThreadPoolExecutor for thread pool management enables efficient handling of multiple meeting minutes generation tasks, ensuring stable system operation even under high load. The multithreaded design alleviates the performance bottleneck caused by single-threaded processing. Furthermore, the system employs load balancing technology to dynamically allocate tasks based on server load, improving resource utilization and overall system response speed.

[0108] Example 2:

[0109] Combined with appendix Figure 2 This embodiment proposes a multimodal meeting minutes generation system based on a large language model, which is used to implement the multimodal meeting minutes generation method described in Embodiment 1. Specifically, it includes a data receiving and storage module, a multimodal information processing module, a minutes generation module, and a formatting module.

[0110] The data receiving and storage module is used to receive basic meeting information uploaded by users and record it in the Meeting table of the database; it also receives meeting files containing images and audio, stores them in a specified directory on the server, and records the file metadata in the database.

[0111] To achieve the above functions, the data receiving and storage module mainly performs the following operations: ① It uses the FastAPI framework to build an efficient and scalable API interface. This API interface is used to receive basic meeting information (including meeting ID, user ID, meeting title, participants, etc.) and meeting files containing images (such as PNG, JPEG) and audio (such as MP3, WAV) uploaded by users via HTTP POST requests; ② It records the basic meeting information (including meeting ID, user ID, meeting title, participants, etc.) in the Meeting table of the database; ③ For the meeting files uploaded by users, it uses a multi-factor validation mechanism of file header, size, and content to check the corresponding format (e.g., for audio files, it checks whether the file format and encoding meet the requirements; for image files, it verifies the image resolution and format). If the validation fails, it returns an error message and records the result. Ensure the files are complete and valid; of course, record the verification results of each upload for subsequent auditing and troubleshooting; ④ Generate a unique UUID for each verified meeting file, and then store them in a designated directory under the server's distributed storage architecture according to file type (such as audio and images) (e.g., audio is stored in recording / raw / {UUID} / , and images are stored in recording / ocr / pic / {UUID} / . This directory structure not only helps with the orderly management of files, but also facilitates subsequent file retrieval and processing); ⑤ Extract the meeting file's metadata, such as file name, size, and upload time, and store it in the database.

[0112] The multimodal information processing module is used to transcribe received audio files into structured text with timestamps using ASR speech-to-text technology, and to recognize received image files into editable text using OCR technology.

[0113] Specifically, the multimodal information processing module uses ASR speech-to-text technology to transcribe received audio files into structured text with timestamps. It primarily performs the following operations: ① Based on the unique UUID of the audio file, it searches for and locates the target audio file in a pre-defined directory, simultaneously extracting the complete storage path and filename of the audio file to ensure accurate file retrieval during subsequent transcription operations; ② It pre-builds an ASR speech-to-text engine using the open-source SenseVoiceSmall model. This engine supports multiple languages ​​and dialects and can efficiently and accurately recognize speech content; The module then calls the ASR speech-to-text engine, loads the located audio file, and performs speech feature recognition... The process involves semantic parsing, converting the speech content in the audio into text, and marking each sentence in the text with its start and end timestamps from the original audio, forming a time-stamped structured text; ③ The generated time-stamped structured text is saved to a preset specified file (such as recording / txt / {UUID} / transcribed.txt file). During the saving process, preliminary formatting is performed simultaneously, including: unifying the text encoding format, standardizing the timestamp display format, removing redundant spaces / garbled characters from the text, and sorting the text content according to the audio time order to ensure the readability and subsequent reusability of the structured text.

[0114] Specifically, the multimodal information processing module uses OCR technology to recognize received image files as editable text, mainly performing the following operations: ① Based on the unique UUID of the image file, it searches for and reads the target image file in a preset specified directory, and then sequentially performs preprocessing operations such as rotation, noise reduction, grayscale conversion, cropping, edge detection, and contrast enhancement to ensure that the text in the image is clearly visible, thus guaranteeing the accuracy of subsequent OCR recognition; ② The preprocessed image is sent to the OCR service, which performs text recognition and returns the recognized editable text; the core model used by the OCR service is... The architecture combining Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) can effectively adapt to complex text layouts and various fonts; ③ The editable text corresponding to a single image returned by the OCR service is saved to a specified file directory (such as recording / ocr / txt / {UUID} / {image name}.txt file) in a preset format; then, the editable text corresponding to all images is concatenated into a complete string in the order of the original images, and the complete string is associated with the corresponding image UUID and stored in the database to realize the structured storage and subsequent retrieval of text data.

[0115] The minutes generation module is used to integrate and transform the basic meeting information uploaded by the user, the structured text and editable text output by the multimodal information processing module into a preliminary draft of structured meeting minutes. Subsequently, in combination with the large language model and predefined logic, text cleaning and content correction are performed on this preliminary draft of structured meeting minutes, and the participant information is extracted. Finally, customized instructions are input into the large language model through prompt engineering, and the large language model outputs a complete meeting minutes that meets the format requirements.

[0116] Specifically, the specific operations of the minutes generation module for text cleaning and content correction of the preliminary draft of structured meeting minutes include: ① Performing text cleaning on the preliminary draft of structured meeting minutes through redundant text removal (such as meeting-irrelevant expressions, repeated content, colloquial redundant expressions), stop word filtering (through intelligent word segmentation technology, using a predefined stop word list to perform word segmentation on the text, removing irrelevant or frequently occurring meaningless words, and retaining key information; the stop word list covers common conjunctions, auxiliary words, modal particles, meaningless filler words, and redundant words unique to the meeting scenario, ensuring that only information valuable to the meeting minutes is retained in the text), and special character processing (such as removing meaningless symbols, unifying format symbols, and correcting garbled characters); ② Performing content correction on the preliminary draft of structured meeting minutes through OCR text correction (correcting the text recognized by OCR, and returning empty if the recognized text is garbled or completely illogical), customized text correction (pre-training a large model according to project requirements and correcting the transcribed text through the pre-trained large model), executor name recognition (correcting possible misspellings of names in the text based on the given names of project-related personnel), general text correction (using a general typo database to perform general text correction to ensure the accuracy and consistency of the text content), and mapping replacement (replacing specific words and names according to preset mapping rules to ensure the consistency and accuracy of the text. For example, replacing abnormally spelled words with similar pronunciations with correct words, such as replacing "CC靠" with "SaySayCall", "海默" with "海若", etc.), and combining character count verification.

[0117] Specifically, the minutes generation module extracts attendee information in the following ways: ① It loads the mapping relationship between attendees and departments from a predefined JSON mapping file containing the names and corresponding department information of all attendees in the company, ensuring the accuracy and consistency of subsequent information extraction; ② It uses a combination of regular expressions and keyword matching to first identify the names and department names of attendees from the initial draft of the structured meeting minutes, and then matches and associates the identified information with the context to ensure the accuracy of the extraction results; ③ It compares the extracted attendee names and department names with the loaded mapping relationship through a matching algorithm to automatically confirm the department to which each attendee belongs, thereby generating structured data to facilitate subsequent applications and presentations; ④ Based on the structured data obtained after information extraction and mapping, it converts it into standardized natural language attendee description sentences (e.g., "Attendees include Zhang San and Li Si from the Technology Department, Wang Wu from the Marketing Department, etc."), ensuring that the meeting minutes content is both professional and readable.

[0118] Specifically, the minutes generation module prompts the Large Language Model (LLM) to input customized instructions, which in turn output a complete meeting minutes that meets the format requirements. The specific operations are as follows: ① The initial draft of the structured meeting minutes after text cleaning and content correction, along with extracted participant information, serves as the basic input, combined with customized instructions, and is input into the Large Language Model (LLM); ② The initial draft of the structured meeting minutes is segmented according to the rule of "no segmentation for less than 1500 characters, segmentation into 2 parts for 1500-5000 characters, and segmentation into 3 parts for more than 5000 characters." During segmentation, consecutive characters are avoided, and symbols within 50 characters to the left and right of the segmentation point are prioritized as segmentation positions; ③ Semantic segmentation is performed on each segment, calling the Large Language Model (LLM) and inputting customized instructions to divide the text into paragraphs based on the semantics of each segment, forming logically coherent segmented text; ④ Basic meeting information is generated based on each segment, and the segmented text of each segment is concatenated with the participant information to form a complete input, which is then fed into the Large Language Model (LLM). ⑤ Based on each slice, generate the next step plan, concatenate the segmented text of each slice with the participant information into a complete input, input it into the large language model (LLM) with customized instructions, and the large language model (LLM) summarizes and outputs the next step plan for the corresponding slice; ⑥ After completing the processing of a single slice, perform the basic meeting information merging operation, summarize the basic meeting information generated by all slices, and form a complete basic meeting information content covering the entire meeting process; ⑦ Perform the next step plan merging operation: first summarize the next step plans generated by all slices, then concatenate the summary result with the segmented complete text and participant information into the final input, input it into the large language model (LLM) with customized instructions, and the large language model (LLM) summarizes and outputs a globally unified, logically closed complete next step plan; ⑧ Perform data cleaning operation to remove the "<" or ">" symbols that exist alone in the final generated meeting minutes text to ensure that the text format is standardized and error-free.

[0119] The formatting module is used to accurately fill in the complete meeting minutes output by the large language model according to the structured fields of the predefined Word template. During the filling process, the text format is automatically matched, and finally an editable meeting minutes Word document that conforms to the specifications is generated, which can be directly downloaded, printed or shared.

[0120] The formatting module performs the following operations to achieve the above functions: ① By predefining the Word template structure (including the layout format of the title, abstract, attendees, meeting content, next steps plan, etc.), it uses the Docx library to set uniform style parameters such as font, font size, and paragraph alignment, and configures placeholders (for example, the title is set to {title}, the abstract to {abstract}, and the attendees to {participants}, etc., to facilitate automated filling and replacement), laying the format and position foundation for the automated generation of meeting minutes; ② Based on the key-value correspondence, it maps and fills the meeting minutes content to the template placeholders, using... Natural language processing technology is used to segment paragraphs, and the format is adjusted using the Docx library or automated tools based on the Docx library (such as setting bold headings, paragraph indentation, bulleted lists, etc.) to ensure accurate content and easy-to-read layout; ③ Word document objects are created using the Docx library, and the content is filled in and predefined styles (such as uniformly setting the font, font size, color and paragraph format) are applied. Then the meeting minutes Word document is saved in the specified path (such as recording / docx / {UUID} / meeting minutes.docx), and supports direct download, printing or sharing. At the same time, the file is managed and traced in an orderly manner through directory storage and version control.

[0121] In summary, the multimodal meeting minutes generation method and system based on a large language model of the present invention can automatically process the process from images and audio recordings to output meeting minutes, improve the efficiency of meeting minutes writing and generation, solve the shortcomings of traditional meeting minutes recording methods and existing intelligent systems, and meet the needs of manufacturing enterprises for efficient and accurate meeting management.

[0122] The above specific examples illustrate the principles and implementation methods of the present invention in detail. These embodiments are merely for the purpose of helping to understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made to the present invention by those skilled in the art without departing from the principles of the present invention should fall within the patent protection scope of the present invention.

Claims

1. A method for generating a multi-modal meeting minutes based on a large language model, characterized in that, Comprise the following steps: S1, receiving user uploaded meeting basic information and record to database Meeting table; receiving meeting files containing pictures, audio, stored in the server specified directory, while recording file metadata to the database; S2, through ASR speech transcription technology, the audio file received in step S1 is transcribed into a timestamped structured text, and the picture file received in step S1 is recognized as an editable text through OCR technology; S3, the user uploaded meeting basic information, the structured text output by step S2 and the editable text are integrated and converted into a structured meeting minutes draft, then combined with a large language model and a predefined logic, the structured meeting minutes draft is executed for text cleaning and content correction, and the information of the participants is extracted, finally through the prompting project, the customized instructions are input to the large language model, and the complete meeting minutes meeting the format requirements are output by the large language model; S4, the complete meeting minutes output by the large language model are accurately filled according to the structured fields of the predefined Word template, the text format is automatically matched during the filling process, and finally the editable meeting minutes Word document meeting the specification is generated, and direct download, printing or sharing is supported.

2. The method of claim 1, wherein the method further comprises: The step S1 specifically comprises the following operations: S1.1, a FastAPI framework is used to build an efficient and scalable API interface, which is used to receive the meeting basic information and meeting files containing pictures and audio uploaded by the user through HTTP POST request; S1.2, the meeting basic information is recorded to the database Meeting table; S1.3, for the meeting files uploaded by the user, the file header, size and content are checked respectively through the multiple verification mechanism, and the corresponding format is checked, and the error prompt is returned and the result is recorded if the verification fails, so as to ensure the completeness and validity of the file; S1.4, a unique UUID is generated for each meeting file passing the verification, and then stored in the specified directory under the server distributed storage architecture according to the file type; S1.5, the metadata of the meeting file is extracted and stored in the database.

3. The method of claim 2, wherein the method further comprises: Step S2 is executed, and the audio file received in step S1 is transcribed into a timestamped structured text through ASR speech transcription technology, and the specific operation comprises: S2.1.1, based on the unique UUID of the audio file, the target audio file is retrieved and located in the preset specified directory, and the complete storage path and file name of the audio file are extracted, so that the subsequent transcription operation can accurately call the file; S2.1.2, call the ASR speech transcription engine, load the located audio file, convert the speech content in the audio into text through speech feature recognition and semantic analysis processing, and label the start time stamp and end time stamp of each sentence in the text in the original audio, forming a timestamped structured text; S2.1.3 Save the generated timestamped structured text to a preset specified file. During the saving process, perform preliminary formatting, including: unifying the text encoding format, standardizing the timestamp display format, removing redundant spaces / garbled characters from the text, and sorting the text content according to the audio time order to ensure the readability and reusability of the structured text.

4. The method of claim 2, wherein the method further comprises: Step S2 involves using OCR technology to convert the image file received in step S1 into editable text. Specific operations include: S2.2.1 Based on the unique UUID of the image file, the target image file is retrieved and read from the preset specified directory. Then, preprocessing operations such as rotation, noise reduction, grayscale conversion, cropping, edge detection and contrast enhancement are performed in sequence to ensure that the text in the image is clearly visible, thus ensuring the accuracy of subsequent OCR recognition. S2.2.2 The preprocessed image is sent to the OCR service, which performs text recognition and returns the recognized editable text. The core model used by the OCR service is an architecture that combines convolutional neural networks and recurrent neural networks, which can effectively adapt to complex text layouts and various fonts. S2.2.3 Save the editable text corresponding to a single image returned by the OCR service to the specified file directory in a preset format; then, concatenate the editable text corresponding to all images into a complete string in the order of the original images, and associate the complete string with the corresponding image UUID and store it in the database to realize the structured storage and subsequent retrieval of text data.

5. The method of claim 1, wherein the method further comprises: Step S3 involves cleaning and correcting the text of the initial draft of the structured meeting minutes, specifically including: Text cleaning of the initial draft of structured meeting minutes was performed by removing redundant text, filtering stop words, and processing special characters. By using OCR text correction, customized text correction, name recognition, general text correction and mapping replacement, combined with character count verification, the draft of structured meeting minutes is corrected.

6. The method of claim 5, wherein the method further comprises: Execute step S3 to extract attendee information. The specific steps are as follows: The mapping relationship between attendees and departments is loaded from a predefined JSON mapping file containing the names of all attendees and their corresponding department information, thereby ensuring the accuracy and consistency of subsequent information extraction. By combining regular expressions and keyword matching, the names of participants and their departments are first identified from the initial draft of the structured meeting minutes. Then, the identified information is matched and associated with the context to ensure that the extraction results are accurate. The extracted attendee names and department names are compared with the loaded mapping relationship through a matching algorithm to automatically confirm the department to which each attendee belongs, thereby generating structured data to facilitate subsequent applications and displays. Based on the structured data obtained after information extraction and mapping, it is converted into standardized natural language descriptive sentences for meeting participants, ensuring that the meeting minutes are both professional and readable.

7. The method of claim 6, wherein the method further comprises: Execute step S3, prompting the project to input customized instructions into the large language model, which then outputs a complete meeting minutes that meets the format requirements. The specific operation is as follows: The structured conference minutes draft after text cleaning and content error correction and the extracted participant information are used as the basis input, and a large language model is input combined with customized instructions; The structured conference minutes draft text is divided according to the rule of "not divided if less than 1500 characters, divided into 2 parts if 1500-5000 characters, and divided into 3 parts if more than 5000 characters", and continuous characters are avoided during division, and symbols within a range of 50 characters on the left and right of the division point are preferred as division positions; Semantic segmentation is performed on each slice, a large language model is called and customized instructions are input, so that the large language model performs paragraph division according to the text semantics of each slice to form logically coherent segmented text; Meeting basic information is generated based on each slice, the segmented text of each slice is spliced with participant information to form complete input, which is input into a large language model and customized instructions are input, and the large language model is used to summarize and output the meeting basic information of the corresponding slice; The next step plan is generated based on each slice, the segmented text of each slice is spliced with participant information to form complete input, which is input into a large language model and customized instructions are input, and the large language model is used to summarize and output the next step plan of the corresponding slice; After completing the processing of a single slice, meeting basic information merging is performed, and the meeting basic information generated by all slices is summarized to form complete meeting basic information content covering the whole process of the meeting; The next step plan merging operation is performed: the next step plans generated by all slices are summarized, and the summary result is spliced with the complete segmented text and participant information to form the final input, which is input into a large language model and customized instructions are input, and the large language model is used to summarize and output a globally unified and logically closed-loop complete next step plan; Data cleaning is performed to remove the "<" or ">" symbols existing alone in the finally generated conference minutes text, so as to ensure that the text format is correct and accurate.

8. The method of claim 7, wherein the method further comprises: The step S4 specifically includes: A predefined Word template structure is used to set uniform style parameters and configure placeholders by using a Docx library, thereby laying a format and position foundation for automatic generation of conference minutes; Based on the key-value correspondence, the content of the conference minutes is mapped and filled into the template placeholders, the paragraphs are segmented by using natural language processing technology, and the format is adjusted by using a Docx library or an automatic tool based on the Docx library, so as to ensure that the content is accurate and the layout is easy to read; A Word document object is created by using a Docx library, content filling and pre-defined style application are sequentially completed, then the document is saved according to a specified path, and direct downloading, printing or sharing are supported, and file order management and traceability are realized through directory storage and version control. 9.A multi-modal meeting minutes generation system based on a large language model, characterized by, The method is used to implement the multi-modal conference minutes generation method according to any one of claims 1-8, and specifically includes: A data receiving and storage module is used to receive conference basic information uploaded by a user and record it to a database Meeting table, receive conference files containing pictures and audio, store them in a specified directory of a server, and record file metadata to a database; A multi-modal information processing module is configured to convert received audio files into timestamped structured text through ASR speech transcription technology, and convert received image files into editable text through OCR technology; A summary generation module is configured to integrate and convert the conference basic information uploaded by the user, the structured text and the editable text output by the multi-modal information processing module into a structured conference summary draft, then perform text cleaning and content correction on the structured conference summary draft in combination with a large language model and predefined logic, extract participant information, and finally input customized instructions to the large language model through a prompting process to output a complete conference summary meeting the format requirements; A formatting module is configured to accurately fill the structured fields of a predefined Word template with the complete conference summary output by the large language model, automatically match the text format during the filling process, and finally generate a standardized editable conference summary Word document, which can be directly downloaded, printed or shared.

Images