System and method for protecting user privacy through artificial intelligence automatic conference summary
By employing a layered processing architecture that identifies and replaces privacy information locally, generates a summary in the cloud, and restores it locally, the problem of privacy information leakage in existing technologies is solved, achieving high-quality meeting minutes generation and privacy protection.
Patent Information
- Application Number
- CN202511672262.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-14
- Publication Date
- 2026-02-27
AI Technical Summary
In existing technologies, AI-based meeting minutes generation solutions pose a risk of privacy breaches, as users' personal identity information and sensitive internal corporate data may be leaked when uploaded to cloud-based models.
Adopting an architecture that separates local processing from cloud processing, the system achieves local identification and replacement of privacy information through an audio acquisition module, a local transcription and preprocessing module, a privacy cleaning module, a cloud summarization module, and a semantic restoration module. It then generates a high-quality summary in the cloud and finally restores the complete meeting minutes locally.
This approach leverages the intelligent capabilities of cloud-based large language models while preventing the leakage of personal and corporate privacy information, thus ensuring the integrity and security of meeting minutes.
Smart Images
Figure CN121580433A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of artificial intelligence and natural language processing technology, in particular to a system and method for protecting user privacy in artificial intelligence automatic meeting minutes. BACKGROUND
[0002] There are various meeting minutes generation schemes based on artificial intelligence in the prior art: Software scheme: automatically enter the meeting, record the audio, and call the cloud large language model (LLM) to generate the meeting summary.
[0003] Hardware scheme: audio data is collected by a conference recording device, and then uploaded to a cloud model to generate a minutes text.
[0004] The above schemes generally have privacy risks: The user's personal information (name, phone number, email, etc.) and enterprise internal sensitive data (project code, business secrets, financial information, etc.) are directly uploaded to the cloud model.
[0005] The cloud model may indirectly leak these private information during training or caching. SUMMARY
[0006] The present application provides a system and method for protecting user privacy in artificial intelligence automatic meeting minutes, which solves the problem of lack of privacy protection in current meeting systems.
[0007] In order to achieve the above purpose, the present application provides the following technical scheme: A system for protecting user privacy in artificial intelligence automatic meeting minutes, comprising: an audio acquisition module, a local transcription and preprocessing module; The audio acquisition module acquires audio data from a conference terminal or a system audio driver or a separate recording device and transmits the audio data to the local transcription and preprocessing module; The local transcription and preprocessing module is used to transcribe and preprocess the audio data and convert it into a meeting transcription original text; Further comprising a privacy cleaning module, a cloud summary module and a semantic restoration module; The privacy cleaning module is used to receive the meeting transcription original text from the local transcription and preprocessing module, identify and replace the text of personal information and enterprise sensitive information in the meeting transcription original text, and output the cleaned text to the cloud summary module; The cloud summary module is used to receive the cleaned text, process it, generate a summary text, and transmit it to the semantic restoration module; The semantic restoration module is used to receive the summary text and restore it to a meeting minutes with real entities before cleaning.
[0008] Preferably, further comprising a management module, the privacy cleaning module, the cloud summary module and the semantic restoration module are connected with the management module.
[0009] Preferably, further comprising a secure transmission and permission control module, the privacy cleaning module is connected with the secure transmission and permission control module.
[0010] Preferably, further comprising a user interface and result display module, the local transcription and preprocessing module, the privacy cleaning module, the cloud summary module and the semantic restoration module are connected with the user interface and result display module.
[0011] A method for protecting user privacy of an artificial intelligence automatic meeting minutes, comprising the system for protecting user privacy of an artificial intelligence automatic meeting minutes described in any of the above.
[0012] Preferably, comprising the following steps: S1: the audio acquisition module acquires conference audio data from a conference terminal; S2: the local transcription and preprocessing module transcribes and preprocesses the audio data and converts it into conference transcription original text; S3: the privacy cleaning module identifies and replaces the text of personal information and enterprise sensitive information in the conference transcription original text, and generates a privacy entity list; S4: the privacy cleaning module replaces the privacy entity with a placeholder according to a preset replacement rule, and generates and saves a mapping relationship of "original entity -> placeholder"; S5: the privacy cleaning module sends the cleaned text after replacement to the cloud summary module and requests to generate a conference summary text; S6: the semantic restoration module receives the summary text returned by the cloud summary module containing the placeholder; S7: the semantic restoration module performs placeholder backfilling on the summary text according to the mapping relationship locally, and restores it to a conference minutes containing real privacy information; S8: the semantic restoration module outputs the final conference minutes to the user or the downstream system.
[0013] Preferably, further comprising: S9: the final conference minutes can be reviewed and polished by a local LLM to ensure the correctness of the text and grammar.
[0014] Preferably, the local transcription and preprocessing module comprises a speech recognition subunit, a speaker separation subunit and a text normalization subunit.
[0015] Preferably, the privacy cleaning module comprises an entity recognition module, an entity classification module, a placeholder generation module, a replacement mapping record module and a cleaned text output module.
[0016] Preferably, the semantic restoration module comprises a mapping acquisition module, a placeholder backfilling module, a semantic consistency correction module and a secure output module.
[0017] By implementing the above technical solutions, the following technical effects are achieved: the system and method for protecting user privacy of an artificial intelligence automatic meeting minutes provided by the application have a layered privacy processing architecture of "locality-cloud-locality restoration". The architecture places voice collection, voice-to-text transcription, privacy identification and cleaning in the local area, places high-quality summarization and intelligent assistant functions in the cloud, and places semantic backfilling of privacy information back to the local area for completion, thereby achieving the intelligent capabilities of a cloud large language model and avoiding leakage of personal and enterprise privacy, and realizing the best solution for an artificial intelligence automatic meeting minutes. BRIEF DESCRIPTION OF DRAWINGS
[0018] Figure 1 FIG. 1 is a structural schematic diagram of a system for protecting user privacy of an artificial intelligence automatic meeting minutes provided by the application; Figure 2 FIG. 2 is a structural schematic diagram of a method for protecting user privacy of an artificial intelligence automatic meeting minutes provided by the application. DETAILED DESCRIPTION
[0019] In order to better understand the technical solutions of the application, the embodiments provided by the application are described in detail below with reference to the accompanying drawings. Embodiment One
[0020] As shown in FIG. 1, it can be understood that the system for protecting user privacy of an artificial intelligence automatic meeting minutes provided by the embodiment comprises: an audio collection module, a local transcription and preprocessing module, a privacy cleaning module, a cloud summarization module and a semantic restoration module. Figure 1 The audio collection module obtains audio data from a conference terminal or a system audio driver or an independent recording device and transmits the audio data to the local transcription and preprocessing module. The local transcription and preprocessing module is configured to transcribe and preprocess the audio data and convert the audio data into a meeting transcription original text. The privacy cleaning module is configured to receive the meeting transcription original text from the local transcription and preprocessing module, identify and replace personal information and enterprise sensitive information in the meeting transcription original text, and output the cleaned text to the cloud summarization module. The cloud summarization module is configured to receive the cleaned text from the privacy cleaning module, perform summarization on the cleaned text, and output a meeting summary text to the semantic restoration module. The semantic restoration module is configured to receive the meeting summary text from the cloud summarization module, perform semantic backfilling on the meeting summary text, and output a meeting summary text with semantic backfilling to the secure output module. The cloud-based summarization module is used to receive the cleaned text, process it, generate a summary text, and transmit it to the semantic restoration module. The semantic restoration module is used to receive the summary text and restore it to the meeting minutes with the original entities before cleaning.
[0021] In this embodiment, the audio acquisition module is used to acquire conference audio data from conference terminals (PCs, mobile phones, dedicated conference recording hardware, smart docks, etc.) in real time or near real time. Its output is connected to the input of the local transcription and preprocessing module, which can use a streaming audio interface to transmit the audio data to the local transcription and preprocessing module. The local transcription and preprocessing module converts the audio into the original conference transcription text. Local audio can be used... ASR Model (Automatic Speech Recognition) (such as locally deployed) Whisper (an open-source ASR algorithm) (Small-scale Chinese speech recognition model or enterprise self-trained model).
[0022] The unique feature is that the local transcription and preprocessing module outputs the original meeting transcript to the local privacy cleansing module. Before the data leaves the local device, the privacy cleansing module identifies and replaces text containing personal and sensitive corporate information, ensuring that the cleaned text uploaded to the cloud-based summarization module is "generalized text." During this cleansing process, conservative strategies can be implemented when identification fails, such as adding placeholders to suspicious words; setting a whitelist (e.g., publicly listed companies, publicly listed organizations) to exclude certain words from replacement; and using semantic block replacement instead of word-by-word replacement to avoid cutting off the context. The privacy cleansing module in this embodiment employs a local small-scale LLM (Locally Streamed Multiprocessing). (Large language model) If the Qwen3 4B (4 billion parameters) model is used, various privacy cleaning modules can be used in other embodiments, one of which is a local small-scale LLM model. During the cleaning process, the privacy cleaning module generates a cleaning dictionary for subsequent anti-cleaning (backfilling) operations.
[0023] After receiving the cleaned text, the cloud-based summarization module utilizes a large number of parameters. Cloud LLM (large language model) Figure 2 This system generates high-quality meeting summaries, action items, decision points, and structured meeting minutes, or executes intelligent agent operations related to the meeting, generating summary text, action item lists, and slot information that needs to be filled. In this embodiment, the cloud model can treat placeholders as original entities, thus maintaining syntactic integrity; and the cloud can output structured results that retain placeholders, facilitating local filling.
[0024] Next, the semantic restoration module restores the summary text returned by the cloud, which still contains placeholders, to the meeting minutes with real entities before cleaning. In this embodiment, the backfill operation must be bound to the session ID, timestamp or file name to prevent incorrect backfill to the original text of other conference transcripts. If part of the placeholder is deleted or merged when generated in the cloud, there needs to be a fault tolerance strategy locally (for example, keep the corresponding part in the original transcription text). Through the above modules, the layered privacy processing architecture of "local → cloud → local restoration" is realized. This architecture places voice collection, speech-to-text transcription, privacy identification and cleaning locally, high-quality summary and intelligent assistant functions in the cloud, and semantic backfill of privacy information back to the local, which not only achieves the intelligent capabilities of the cloud large language model, but also avoids the leakage of personal and enterprise privacy, and realizes the best solution for automatic meeting minutes of artificial intelligence.
[0025] Further, a management module is also included, and the privacy cleaning module, the cloud summary module and the semantic restoration module are connected with the management module. The management module is used to manage all "original text → placeholder" mappings, speaker information, conference ID, timestamp, segmentation information generated in a meeting. It is stored locally in priority, and can be encrypted and backed up to the enterprise private cloud if necessary; clear text is strictly prohibited from being sent out. When the privacy cleaning module cleanses the data, it writes the mapping relationship of "original entity → placeholder" into the mapping table, which is saved by the management module and stored locally or in a locally controlled secure area, and is not uploaded to the cloud.
[0026] On the basis of the above embodiment, a secure transmission and permission control module is also included, and the privacy cleaning module is connected with the secure transmission and permission control module. The secure transmission and permission control module provides encryption, authentication and audit functions when transmitting data between the local and the cloud. This module only sends out "cleaned text", and the original text and the mapping table are not sent out, thereby avoiding privacy leakage from the architecture.
[0027] Further, a user interface and result display module is also included, and the local transcription and preprocessing module, the privacy cleaning module, the cloud summary module and the semantic restoration module are connected with the user interface and result display module. The user interface and result display module is used to show the final meeting minutes, action items, timelines, participants, task assignments and other information to the user; it can show the states such as "local processing / cloud processing / backfilling completed" to let the user know which stages are in the local and which stages are in the cloud. The semantic restoration module outputs the restored final meeting minutes to the user interface and result display module for display. Embodiment Two
[0028] As NER (Named Entity Recognition), small LLMAs shown, the embodiment provides a method for protecting user privacy by artificial intelligence automatic meeting minutes, which comprises the system for protecting user privacy by artificial intelligence automatic meeting minutes described above.
[0029] comprising the following steps: S1: the audio acquisition module acquires conference audio data from the conference terminal; S2: the local transcription and preprocessing module transcribes and preprocesses the audio data and converts it into conference transcription original text; S3: the privacy cleaning module identifies and replaces the text of personal information and enterprise sensitive information in the conference transcription original text to generate a privacy entity list; S4: the privacy cleaning module replaces the privacy entity with a placeholder according to a preset replacement rule, and generates and saves a mapping relationship of "original entity placeholder"; S5: the privacy cleaning module sends the cleaned text after replacement to the cloud summary module to request to generate a conference summary text; S6: the semantic restoration module receives the summary text returned by the cloud summary module containing the placeholder; S7: the semantic restoration module fills the placeholder according to the mapping relationship to restore the conference minutes containing the real privacy information; S8: the semantic restoration module outputs the final conference minutes to the user or the downstream system.
[0030] In the above steps, the privacy cleaning module can be based on rules (regular matching of mobile phone numbers and emails), based on models
NLP (Natural Language Processing) )
[0031] In addition, when the local transcription and preprocessing module preprocesses, the local model size used can be a small model with 1B-7B parameter level running on PC, Jetson Orin, and Mac to realize text-level LLM NLP tasks after transcription.
[0032] The cleaned text can be pushed to the cloud in segments of 5-15 seconds to realize quasi-real-time conference minutes.
[0033] The placeholder naming rule involved in the privacy cleaning module when implementing literal replacement can adopt the "type + serial number" manner, such as PERSON_1, ORG_1, PROJECT_1; or adopt the naming with context, such as SPEAKER_ALICE→SPEAKER_1.
[0034] On the basis of the above-mentioned embodiments, the local transcription and preprocessing module comprises a speech recognition subunit, a speaker separation subunit and a text regularization subunit. The speech recognition subunit is used for converting audio into timestamped text; the speaker separation subunit is used for distinguishing different speakers to provide semantic roles for subsequent privacy replacement; and the text regularization subunit is used for denoising, punctuation restoration and paragraph segmentation.
[0035] And further comprising: S9: The final meeting minutes can be polished by the local LLM to ensure the correctness of the text and grammar. The above-mentioned step 9 is a supplementary step of the method and is an optional item.
[0036] Further, the privacy cleaning module comprises an entity recognition module, an entity classification module, a placeholder generation module, a replacement mapping record module and a cleaned text output module.
[0037] The entity recognition module uses a local small-scale (NER) or local named entity recognition model to identify names, departments, company names, project codes, addresses, emails, phone numbers, client names and order numbers, and can identify "entities unique to this meeting" in combination with the context of the meeting history, such as "X project" first appearing in this meeting.
[0038] The entity classification module classifies the identified entities according to preset privacy levels, for example: first-level privacy (real name, mobile phone number), second-level privacy (enterprise name, project code), third-level privacy (meeting location), and different levels adopt different replacement strategies.
[0039] The placeholder generation module uniformly replaces the identified entities with placeholders that do not contain privacy, such as "Speaker_1", "Company_A", "Project_X" and "Client_Y". The placeholder needs to have reversibility when generated, that is, the same entity appearing multiple times in the same conversation is replaced by the same placeholder.
[0040] The replacement mapping record module writes the mapping relationship of "original entity → placeholder" into a mapping table, which is saved by the management module, and the storage location is local or a local controlled security area, and is not uploaded to the cloud.
[0041] The cleaned text output module outputs the "cleaned text" after replacement as input for subsequent cloud processing.
[0042] Further, the semantic restoration module includes a mapping acquisition module, a placeholder backfilling module, a semantic consistency correction module, and a safe output module. The mapping acquisition module reads the mapping table corresponding to the current meeting from the management module. The placeholder backfilling module replaces the placeholders in the cloud summary text with the original entities one by one according to the mapping table, supporting one-to-many or many-to-one backfilling scenarios, such as restoration when multiple speakers are combined. The semantic consistency correction module is used when the cloud makes changes to the entities to improve readability (such as replacing "Company_A" with "the company"), and the local can use a small model or rules to further polish the text to ensure that the sentence is natural after backfilling. The safe output module outputs the final meeting minutes recovered to the user interface and result display module, or writes them into the local database.
[0043] Based on the system and method involved in the above two embodiments, if the enterprise does not allow access to the external network, the "cloud summary module" can be deployed in the enterprise private large model service, and the overall process remains unchanged.
[0044] Privacy cleaning not only replaces entities, but also performs "semantic summary cleaning", i.e., replacing "Shanghai XX Semiconductor Co., Ltd." with "the supplier". This way can reduce the number of placeholders.
[0045] If it involves a multi-round meeting scenario, the management module can be reused across multiple meetings, ensuring that the same customer is restored to the same real name in multiple meetings.
[0046] Based on the present application, a hardware all-in-one machine solution can be implemented, which solidifies the audio acquisition module, the local transcription and preprocessing module, the privacy cleaning module, the semantic restoration module, and the management module in the meeting terminal or AI meeting base, and only sends the data of the privacy cleaning module outputting the cleaned text to the cloud summary module. This can reduce the complexity of enterprise deployment.
[0047] If only the local solution is considered, for highly classified scenarios, the cloud model can not be called, but a local large model can be called to generate a concise summary. In this case, the cleaning and backfilling of the present application still hold, but the cloud stage is replaced by a local high-performance node.
[0048] The above has described in detail the system and method for protecting user privacy in artificial intelligence automatic meeting minutes provided by the embodiments of the present application. For those skilled in the art, according to the idea of the embodiments of the present application, the specific implementation and application range can be changed. In view of the above, the content of the specification should not be understood as a limitation of the present application.
Claims
1. An artificial intelligence automatic meeting minutes system that protects user privacy, comprising: An audio acquisition module, a local transcription and preprocessing module; The audio acquisition module obtains audio data from a conference terminal or a system audio driver or an independent recording device and transmits the audio data to the local transcription and preprocessing module; The local transcription and preprocessing module is configured to transcribe and preprocess the audio data and convert the audio data into conference transcription original text; In particular, the system further comprises a privacy cleaning module, a cloud summary module, and a semantic restoration module; The privacy cleaning module is configured to receive the conference transcription original text from the local transcription and preprocessing module, identify and replace personal information and enterprise sensitive information in the conference transcription original text, and output the cleaned text to the cloud summary module; The cloud summary module is configured to receive the cleaned text, process the cleaned text, generate a summary text, and transmit the summary text to the semantic restoration module; The semantic restoration module is configured to receive the summary text and restore the summary text to a conference minutes with real entities before cleaning. 2.The system for protecting user privacy by automatically taking meeting minutes using artificial intelligence according to claim 1, wherein, The system further comprises a management module, and the privacy cleaning module, the cloud summary module, and the semantic restoration module are connected to the management module. 3.The system of claim 1, wherein, The system further comprises a secure transmission and permission control module, and the privacy cleaning module is connected to the secure transmission and permission control module. 4.The system of claim 1, wherein, The system further comprises a user interface and result display module, and the local transcription and preprocessing module, the privacy cleaning module, the cloud summary module, and the semantic restoration module are connected to the user interface and result display module.
5. An artificial intelligence automatic meeting minutes method for protecting user privacy, characterized in that, The system comprises the artificial intelligence automatic conference minutes protection user privacy system of any one of claims 1-4.
6. The method for protecting user privacy by artificial intelligence automatic meeting minutes according to claim 5, characterized in that, The system comprises the following steps: S1: The audio acquisition module obtains conference audio data from a conference terminal or an operating system audio driver or an independent recording device; S2: The local transcription and preprocessing module transcribes and preprocesses the audio data and converts the audio data into conference transcription original text; S3: The privacy cleaning module identifies and replaces personal information and enterprise sensitive information in the conference transcription original text and generates a privacy entity list; S4: The privacy cleaning module replaces the privacy entities with placeholders according to a preset replacement rule, generates and saves a mapping relationship of "original entity→placeholder", and sends the cleaned text after replacement to the cloud summary module to request generation of a conference summary text; S6: The semantic restoration module receives the summary text returned by the cloud summary module and containing the placeholders; S7: The semantic restoration module performs placeholder backfilling on the summary text according to the mapping relationship to restore the conference minutes containing real privacy information; S8: The semantic restoration module outputs the final conference minutes to a user or a downstream system. Further comprising:
7. The method for protecting user privacy of an artificial intelligence automatic meeting minutes according to claim 5, characterized in that, S9: The final conference minutes can be reviewed by a local LLM to ensure the correctness of the text and grammar. The local transcription and preprocessing module comprises a speech recognition subunit, a speaker separation subunit, and a text normalization subunit. 8.The method for protecting user privacy by artificial intelligence automatic meeting minutes according to claim 5, wherein, The privacy cleaning module comprises an entity identification module, an entity classification module, a placeholder generation module, a replacement mapping record module, and a cleaned text output module. 9.The method for protecting user privacy by artificial intelligence automatic meeting minutes according to claim 5, wherein, 10. The method of claim 5, wherein the artificial intelligence automatic meeting minutes protect the user privacy. The semantic restoration module comprises a mapping acquisition module, a placeholder backfilling module, a semantic consistency correction module and a safe output module. The semantic restoration module comprises a mapping acquisition module, a placeholder backfilling module, a semantic consistency correction module and a safe output module.