End-to-end conference intelligent processing system and method

Abstract

The present application relates to the technical field of end-to-end conference intelligent processing, and particularly refers to an end-to-end conference intelligent processing system and method, which acquires streaming PCM audio through a USB dual-microphone array; transmits the streaming to a cloud ASR in segments through a WebSocket long connection for real-time transcription; realizes speaker separation and real-time identification by combining dual-microphone phase difference sound source positioning and voiceprint characteristics; sends the transcription text with speaker identification to an LLM to generate a structured analysis result containing conference highlights, to-do items and mind maps through a prompt word engineering; and outputs in a multi-modal form, with the mind map nodes and audio time stamps being bidirectionally bound to realize linkage interaction. The present application solves the problems of existing conference recording schemes, such as scattered equipment, poor data flow, high transcription delay of ASR batch processing, lack of intelligent structured analysis, insufficient multi-language support and difficulty in realizing accurate speaker separation in a multi-person scene.

Description

Technical Field

[0001] This invention relates to the field of end-to-end intelligent conferencing technology, specifically to end-to-end intelligent conferencing systems and methods. Background Technology

[0002] The existing meeting recording solution has the following technical problems:

[0003] (1) Device fragmentation problem: It requires multiple links such as independent recording equipment, transcription software, and note-taking tools, resulting in a fragmented user experience and poor data flow between devices.

[0004] (2) High transcription delay: Traditional ASR uses batch processing, which cannot achieve real-time transcription. Users need to wait for the entire recording to be completed before they can see the transcription result.

[0005] (3) Lack of intelligent analysis: The existing solution can only output text and cannot automatically extract structured information such as meeting points, to-do items, and mind maps. Users need to organize them manually.

[0006] (4) Weak multilingual support: Most solutions only support a single language or a few languages, which cannot meet the needs of multinational conferences.

[0007] (5) Difficulty in speaker separation: In multi-person meeting scenarios, it is impossible to effectively distinguish different speakers, resulting in chaotic transcription results and difficulty in tracing the speakers.

[0008] The present invention aims to solve the above-mentioned technical problems and provide an end-to-end integrated solution from recording acquisition to intelligent analysis output. Summary of the Invention

[0009] The technical problems to be solved by this invention are: the existing meeting recording solutions suffer from scattered equipment and poor data flow; high latency in ASR batch processing and transcription; lack of intelligent structured analysis; insufficient multi-language support; and difficulty in achieving accurate speaker separation in multi-person scenarios.

[0010] To address the above problems, the technical solution adopted in this invention is an end-to-end intelligent conferencing processing system and method, comprising the following steps:

[0011] Audio data is acquired via a USB dual-microphone array, which adopts a linear layout and outputs streaming PCM audio data.

[0012] The audio data is segmented according to a preset time length and sent to the cloud ASR service for real-time transcription via a WebSocket long connection to obtain real-time transcribed text;

[0013] The sound source azimuth angle is obtained by using the phase difference of a dual-microphone array to locate the sound source, and the speaker is separated in real-time transcribed text by combining the speaker feature extraction algorithm. Different speaker identifiers are marked in real time, and the speaker switching detection delay is less than 500ms.

[0014] The transcribed text carrying speaker identifiers is sent to the LLM service, which uses prompt word engineering to guide the LLM to generate structured meeting analysis results that include meeting highlights, key decisions, to-do items, and mind map structures.

[0015] The results of structured meeting analysis are output in a multimodal format, including transcribed text, meeting minutes, and mind maps. Each node in the mind map is mapped to its corresponding recording timestamp, enabling interactive communication between the recording and the mind map.

[0016] Furthermore, the steps of streaming audio data in segments according to a preset time length specifically include: using a 200ms sliding window mechanism to segment the audio data, using a 50% overlap window between adjacent segments, triggering an ASR transcription request for each segment, and communicating bidirectionally with the cloud ASR service through a WebSocket long connection to achieve streaming processing of recording and transcription simultaneously, with an overall transcription delay of less than 2 seconds.

[0017] Furthermore, the steps of sound source localization based on the phase difference of the dual-microphone array and speaker separation combined with the voiceprint feature extraction algorithm specifically include:

[0018] The cross-correlation function of the dual-microphone audio signals is calculated based on the Time Difference of Arrival (TDOA) algorithm. The time delay difference is obtained and the azimuth angle of the sound source is estimated. The sound source localization results of the left, center and right azimuth regions are output.

[0019] Extract i-vector or x-vector voiceprint features from audio signals, cluster speakers using cosine similarity, and generate voiceprint templates for different speakers;

[0020] By combining changes in the azimuth angle of the sound source and short-term energy mutations, speaker switching events are detected, and the transcribed text is bound to the corresponding speaker identifier in real time.

[0021] Furthermore, the steps for guiding LLM to generate structured meeting analysis results through cue word engineering specifically include: constructing a cue word template containing rules for extracting meeting key points, rules for identifying decisions, rules for extracting to-do items, and rules for generating mind maps hierarchically; inputting transcribed text carrying speaker identifiers and timestamps into LLM; and outputting structured data in JSON format. The JSON format data includes a list of meeting key points, key decisions and their decision-makers, to-do items and their responsible persons and deadlines, as well as a hierarchical mind map structure.

[0022] Furthermore, the specific steps to achieve interactive integration between audio recording and mind mapping include: binding each node of the mind map with the corresponding recording timestamp range of the meeting content; responding to the user's click operation on a mind map node, jumping to the starting timestamp corresponding to that node and starting playback, while simultaneously highlighting the mind map node corresponding to the current playback position, thus realizing audio recording, text transcription, and mind mapping integration. Figure 3 The timestamps of the users are synchronized.

[0023] Furthermore, the cloud-based ASR service adapts to multiple vendors through a unified ASRProvider abstract interface. The ASRProvider interface defines unified methods for initialization, audio segmentation and transmission, transcription result callback, and hot word library loading. It supports dynamic switching between iFlytek, Alibaba, and Baidu ASR services in the Chinese region and Google and Amazon ASR services in the global region, achieving real-time transcription support for six mainstream languages.

[0024] Furthermore, the USB dual microphone array includes two MEMS microphones spaced 40mm apart. It works with an integrated audio DSP processing chip to achieve hardware-level noise reduction and echo cancellation. The sampling rate is 48kHz / 16bit, the signal-to-noise ratio is greater than 65dB, the effective pickup distance is 0.5 to 3 meters, and it communicates with the host via the USBAudioClass protocol.

[0025] Furthermore, LLM supports two processing modes for generating structured meeting analysis results: real-time mode and post-processing mode. In real-time mode, preliminary meeting minutes and to-do items are generated while the meeting is being recorded. In post-processing mode, complete meeting analysis results and mind maps are generated in batches after the meeting recording is completed.

[0026] Furthermore, it also includes steps to synchronize meeting minutes and structured analysis results to a mobile app via Bluetooth Low Energy (BLE), as well as export functionality supporting multiple formats such as TXT, PDF, Word, and XMind.

[0027] The present invention also provides a system for implementing the above-described intelligent processing method for end-to-end conferencing, comprising:

[0028] The audio recording module is used to acquire audio data via a USB dual-microphone array. The dual-microphone array adopts a linear layout and outputs streaming PCM audio data.

[0029] The streaming ASR transcription module is used to segment audio data according to a preset time length and send it to the cloud ASR service for real-time transcription via a WebSocket long connection to obtain real-time transcribed text.

[0030] The speaker separation module is used to locate the sound source based on the phase difference of the dual-microphone array and obtain the azimuth angle of the sound source. It also combines the speaker feature extraction algorithm to separate the speakers in the real-time transcribed text, and marks different speakers in real time. The speaker switching detection delay is less than 500ms.

[0031] The LLM Intelligent Analysis Module is used to send transcribed text carrying speaker identifiers to the LLM service, and guides the LLM to generate structured meeting analysis results containing meeting highlights, key decisions, to-do items, and mind map structures through prompt word engineering.

[0032] The multimodal output module is used to output the results of structured meeting analysis in a multimodal format, including transcribed text, meeting minutes, and mind maps. Each node of the mind map establishes a bidirectional mapping relationship with the corresponding recording timestamp, enabling interactive linkage between the recording and the mind map.

[0033] The advantages of this invention compared to the prior art are:

[0034] 1. End-to-end integration for enhanced user experience: This invention deeply integrates USB dual-microphone hardware acquisition, streaming ASR transcription, speaker separation, LLM intelligent analysis, and multimodal output, enabling fully automated processing from recording to transcription to analysis with a single click. This solves the problems of existing solutions where devices are scattered and data transfer requires manual intervention, and avoids operational fragmentation caused by switching between multiple stages.

[0035] 2. Low-latency real-time transcription improves meeting efficiency: It adopts a 200ms sliding window combined with a 50% overlap streaming segmentation mechanism, and communicates bidirectionally with the cloud ASR service through a WebSocket long connection to achieve simultaneous recording and transcription. The overall transcription latency is less than 2 seconds, which solves the high latency problem in the traditional batch processing mode that requires waiting for the complete recording to be completed before viewing the transcription results.

[0036] 3. LLM-driven structured intelligent analysis reduces manual processing costs: LLM is guided by prompt word engineering to automatically extract meeting key points, critical decisions, and to-do items and generate hierarchical mind maps, outputting structured data in JSON format. This solves the problem that existing solutions can only output plain text transcripts and require manual processing of meeting minutes, greatly improving the efficiency of meeting information processing.

[0037] 4. Dual-microphone array + voiceprint feature fusion speaker separation improves the accuracy of multi-person conferences: Combining TDOA sound source localization with i-vector / x-vector voiceprint feature clustering, supplemented by azimuth angle change and energy mutation detection speaker switching, the switching detection latency is less than 500ms, and the speaker separation accuracy can reach more than 90%, which solves the problems of messy transcription results and inability to trace speakers in multi-person conference scenarios.

[0038] 5. Multi-vendor and multi-language adaptation, covering cross-border meeting scenarios: Dynamic switching of ASR services from multiple vendors is achieved through a unified ASRProvider abstract interface, supporting services from iFlytek, Alibaba, and Baidu in Chinese regions, as well as Google and Amazon in global regions. It covers real-time transcription of 6 mainstream languages, solving the problem of weak multi-language support and inability to meet the needs of cross-border meetings in existing solutions.

[0039] 6. Audio Recording-Mind Map Interactive Integration for Enhanced Content Browsing Experience: Mind map nodes are bidirectionally linked to audio recording timestamps. Clicking a mind map node jumps to the corresponding recording location and highlights it synchronously, enabling audio recording, text transcription, and mind mapping to work together seamlessly. Figure 3 The linkage of timestamps between meeting minutes and content browsing solves the problem of disconnect between meeting minutes and content browsing, making it easier for users to quickly locate and review key meeting content.

[0040] 7. Hardware-level audio processing to ensure acquisition quality: The dual MEMS microphone array, combined with an integrated audio DSP processing chip, achieves hardware-level noise reduction and echo cancellation, with a signal-to-noise ratio greater than 65dB and an effective pickup distance of 0.5 to 3 meters. This solves the problem of online conferencing software relying on the device's built-in microphone and poor transcription quality in noisy environments. Attached Figure Description

[0041] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0042] Figure 1 This is a diagram showing the overall system architecture of the end-to-end intelligent conferencing system and method of the present invention.

[0043] Figure 2 This is a schematic diagram illustrating the speaker separation principle of the dual-microphone array in the end-to-end intelligent conferencing system and method of the present invention.

[0044] Figure 3 This is a timing diagram of streaming ASR processing in the end-to-end intelligent conferencing system and method of the present invention.

[0045] Figure 4 This is a schematic diagram of the APP interface in the end-to-end intelligent conferencing system and method of the present invention.

[0046] Figure 5 This is a data flow diagram in the end-to-end intelligent conferencing system and method of the present invention.

[0047] Figure 6 This is a flowchart of the LLM processing in the end-to-end intelligent conferencing system and method of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] The end-to-end intelligent conference processing system and method of the present invention will be described below with reference to the accompanying drawings.

[0050] Combined with appendix Figure 1-6 This invention discloses an end-to-end intelligent conferencing processing system and method, including a recording acquisition module. The recording acquisition module employs a USB dual-microphone array structure with a linear layout, outputting streaming PCM audio data. The dual-microphone array includes multiple MEMS microphones, working in conjunction with an integrated audio DSP processing chip to achieve hardware-level noise reduction and echo cancellation. It communicates with the host via the USBAudioClass protocol, enabling communication between the audio device and the host without additional drivers. The recording acquisition module supports high-fidelity recording, uses streaming data output, and its effective pickup distance covers typical conferencing scenarios, meeting the audio pickup needs of multi-person conferences.

[0051] The streaming ASR transcription module communicates with the cloud-based ASR service via a WebSocket persistent connection, supporting real-time streaming transcription of multiple languages. It employs a sliding window mechanism to segment audio data, with overlapping windows between adjacent segments, triggering a transcription request for each segment. The module features a multi-vendor adaptation layer, defining a unified ASRProvider interface to support SDK integration from different vendors and allowing the loading of user-defined terminology hotword libraries to improve recognition accuracy. This multi-vendor adaptation layer supports different ASR services in Chinese-speaking and global regions, covering multiple mainstream languages ​​to meet the multilingual transcription needs of cross-border conferences.

[0052] Speaker separation module. This module locates the sound source based on the phase and energy differences of the dual-microphone array and uses a voiceprint feature extraction algorithm to cluster speakers. The module identifies different speakers in real time, and speaker switching detection is based on energy abrupt changes and azimuth angle variations. The module extracts i-vector or x-vector voiceprint features and uses cosine similarity for speaker clustering. Sound source localization calculates the cross-correlation function of the dual-microphone audio signals using a time-of-arrival algorithm, obtains the time delay difference, estimates the sound source azimuth angle, and outputs sound source localization results for multiple azimuth regions, assisting the speaker separation module in improving the differentiation accuracy in multi-person conference scenarios.

[0053] The LLM Intelligent Analysis Module uses prompt word engineering to guide LLM in generating structured information such as meeting minutes, to-do lists, and mind maps. It supports both real-time and post-processing generation modes. The module constructs prompt word templates containing rules for extracting meeting key points, identifying decisions, extracting to-do lists, and generating hierarchical mind maps, outputting structured data in JSON format. This structured data includes a list of meeting key points, key decisions and their decision-makers, to-do lists and their responsible persons and deadlines, as well as a hierarchical mind map structure, facilitating subsequent multimodal output and data processing.

[0054] Multimodal output module. The multimodal output module supports exporting in multiple formats, including plain text transcripts, formatted meeting minutes, mind maps, and structured data files. It features synchronized browsing functionality, synchronizing recording, text transcription, and mind map timestamps. Clicking a mind map node jumps to the corresponding recording position. Each node in the mind map is mapped to a corresponding recording timestamp. Responding to user actions on a mind map node, the audio playback position jumps to the corresponding timestamp, while simultaneously highlighting the mind map node at the current playback position, thus enabling simultaneous recording, text transcription, and mind map... Figure 3 The timestamps of the users are synchronized and linked for browsing.

[0055] System Overall Architecture. This system also includes a storage layer and a synchronization layer. The storage layer uses local database storage, with optional cloud storage as a supplement, to meet the data storage needs of different users. The synchronization layer synchronizes meeting minutes to the mobile app via BLE Bluetooth Low Energy, achieving cross-device data synchronization. The system data flow is as follows: audio is captured by the dual-microphone array at the hardware layer, processed by the processing chip, and then transmitted to the desktop app via USB Audio. The desktop app interacts with the cloud-based ASR service and LLM service to complete end-to-end processing from audio acquisition to intelligent analysis and output.

[0056] The working principle of this application is as follows: The end-to-end intelligent conferencing processing system and method of this invention works by the following: After the user starts the system, the USB dual-microphone array begins to collect audio data from the meeting. After hardware DSP noise reduction and echo cancellation processing, the audio data is transmitted to the desktop application in streaming PCM format via the USB interface. The desktop application segments the audio data according to a sliding window mechanism and sends it to the cloud ASR service for real-time transcription via a WebSocket long connection to obtain real-time transcribed text. At the same time, the speaker separation module calculates the phase difference and energy difference based on the dual-channel audio signals collected by the dual-microphone array, uses the TDOA algorithm to locate the sound source and obtain the azimuth angle of the sound source, and combines the i-vector or x-vector voiceprint feature extraction algorithm to perform speaker clustering on the audio signal, marking different speaker identifiers in real time. When a sudden change in energy and azimuth angle is detected, the speaker is determined to switch and the corresponding identifier is updated. The transcribed text, carrying speaker identifiers and timestamps, is sent to the LLM service. The LLM intelligent analysis module guides the large language model to perform structured analysis of the meeting content using preset prompt templates. It automatically extracts key points, crucial decisions, and to-do items, generating a hierarchical mind map structure and outputting structured data in JSON format. The multimodal output module presents the transcribed text, meeting minutes, and mind maps to the user in multiple formats. Each node in the mind map establishes a bidirectional mapping with its corresponding recording timestamp. When a user clicks a node in the mind map, the system jumps the audio playback position to the corresponding timestamp and highlights the currently playing node, realizing the integration of recording, transcription, and mind mapping. Figure 3 The system supports both real-time and post-processing modes. In real-time mode, preliminary analysis results are generated while recording, while in post-processing mode, complete meeting analysis content is generated in batches after the meeting recording is completed. The system also synchronizes meeting minutes to mobile devices via Bluetooth Low Energy (BLE) and supports exporting files in various formats, making it convenient for users to view and use meeting records in different scenarios.

[0057] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art, inspired by this description, design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. An intelligent processing method for end-to-end conferencing, characterized in that: Includes the following steps: Audio data is acquired via a USB dual-microphone array, which adopts a linear layout and outputs streaming PCM audio data. The audio data is segmented according to a preset time length and sent to the cloud ASR service for real-time transcription via a WebSocket long connection to obtain real-time transcribed text; The sound source azimuth angle is obtained by locating the sound source based on the phase difference of the dual microphone array, and the speaker is separated by combining the voiceprint feature extraction algorithm. Different speaker identifiers are marked in real time, and the speaker switching detection delay is less than 500ms. The transcribed text carrying speaker identifiers is sent to the LLM service, which uses prompt word engineering to guide the LLM to generate structured meeting analysis results that include meeting highlights, key decisions, to-do items, and mind map structures. The structured meeting analysis results are output in a multimodal format, including transcribed text, meeting minutes, and mind maps. Each node of the mind map is mapped bidirectionally to its corresponding recording timestamp, enabling interactive linkage between the recording and the mind map.

2. The end-to-end intelligent conferencing processing method according to claim 1, characterized in that: The specific steps of streaming audio data into segments according to a preset time length include: using a 200ms sliding window mechanism to segment the audio data, using a 50% overlap window between adjacent segments, triggering an ASR transcription request for each segment, and communicating bidirectionally with the cloud ASR service through a WebSocket long connection to achieve streaming processing of recording and transcription simultaneously, with an overall transcription delay of less than 2 seconds.

3. The end-to-end intelligent conferencing processing method according to claim 1, characterized in that: The steps of sound source localization based on the phase difference of a dual-microphone array and speaker separation combined with a voiceprint feature extraction algorithm specifically include: The cross-correlation function of the dual-microphone audio signals is calculated based on the Time Difference of Arrival (TDOA) algorithm. The time delay difference is obtained and the azimuth angle of the sound source is estimated. The sound source localization results of the left, center and right azimuth regions are output. Extract i-vector or x-vector voiceprint features from audio signals, cluster speakers using cosine similarity, and generate voiceprint templates for different speakers; By combining the changes in the azimuth angle of the sound source and short-term energy mutations, speaker switching events are detected, and the transcribed text is bound to the corresponding speaker identifier in real time.

4. The end-to-end intelligent conferencing processing method according to claim 1, characterized in that: The steps for generating structured meeting analysis results using prompt word engineering to guide LLM include: constructing a prompt word template containing rules for extracting meeting key points, rules for identifying decisions, rules for extracting to-do items, and rules for generating hierarchical mind maps; inputting transcribed text carrying speaker identifiers and timestamps into LLM; and outputting structured data in JSON format. The JSON format data includes a list of meeting key points, key decisions and their decision-makers, to-do items and their responsible persons and deadlines, and a hierarchical mind map structure.

5. The end-to-end intelligent conferencing processing method according to claim 1, characterized in that: The steps to achieve the linkage and interaction between recording and mind map specifically include: binding the recording timestamp range of the corresponding meeting content to each node of the mind map; responding to the user's click operation on the mind map node, jumping to the starting timestamp corresponding to the node and starting playback, while highlighting the mind map node corresponding to the current playback position, so as to realize the synchronous linkage of timestamps of recording, transcribed text and mind map.

6. The end-to-end intelligent conferencing processing method according to claim 1, characterized in that: The cloud-based ASR service is adapted to multiple vendors through a unified ASRProvider abstract interface. The ASRProvider interface defines unified methods for initialization, audio segmentation and transmission, transcription result callback and hot word library loading. It supports dynamic switching between iFlytek, Alibaba, and Baidu ASR services in Chinese regions and Google and Amazon ASR services in global regions, and achieves real-time transcription support for 6 mainstream languages.

7. The end-to-end intelligent conferencing processing method according to claim 1, characterized in that: The USB dual microphone array includes two MEMS microphones spaced 40mm apart. It works with an integrated audio DSP processing chip to achieve hardware-level noise reduction and echo cancellation. The sampling rate is 48kHz / 16bit, the signal-to-noise ratio is greater than 65dB, the effective pickup distance is 0.5 to 3 meters, and it communicates with the host via the USBAudioClass protocol.

8. The end-to-end intelligent conferencing processing method according to claim 1, characterized in that: The LLM generates structured meeting analysis results and supports two processing modes: real-time mode and post-processing mode. In real-time mode, preliminary meeting minutes and to-do items are generated while the meeting is being recorded. In post-processing mode, complete meeting analysis results and mind maps are generated in batches after the meeting is recorded.

9. The end-to-end intelligent conferencing processing method according to claim 1, characterized in that: It also includes steps to synchronize meeting minutes and structured analysis results to a mobile app via Bluetooth Low Energy (BLE), and supports exporting in multiple formats such as TXT, PDF, Word, and XMind.

10. The system upon which the end-to-end intelligent conferencing processing method according to claims 1-9 is based, characterized in that, include: The audio recording module is used to acquire audio data via a USB dual-microphone array, wherein the dual-microphone array adopts a linear layout and outputs streaming PCM audio data. The streaming ASR transcription module is used to segment the audio data according to a preset time length and send it to the cloud ASR service for real-time transcription via a WebSocket long connection to obtain real-time transcribed text. The speaker separation module is used to locate the sound source based on the phase difference of the dual microphone array to obtain the azimuth angle of the sound source, and to perform speaker separation on the real-time transcribed text in combination with the voiceprint feature extraction algorithm, and to mark different speaker identifiers in real time, with a speaker switching detection delay of less than 500ms. The LLM Intelligent Analysis Module is used to send transcribed text carrying speaker identifiers to the LLM service, and guides the LLM to generate structured meeting analysis results containing meeting highlights, key decisions, to-do items, and mind map structures through prompt word engineering. The multimodal output module is used to output the structured meeting analysis results in a multimodal form, including transcribed text, meeting minutes, and mind maps. Each node of the mind map establishes a bidirectional mapping relationship with the corresponding recording timestamp, enabling interactive linkage between the recording and the mind map.

Images