Multilingual conference recording and interpretation method and system based on two-way audio comparison

Abstract

The application discloses a kind of multilingual conference recording and translation method and system based on two-way audio comparison, it is related to the cross field of audio communication and artificial intelligence speech processing.The method comprises: each participant logs in server by intelligent terminal;When speaking, the voice is collected by intelligent terminal and embedded identity ID to generate the first audio stream upload, and the mixed original audio without identity mark is collected by conference terminal as the second audio stream upload;After receiving two-way audio, the server carries out time axis synchronization alignment, and according to the identity ID in the first audio stream, the voice content in the second audio stream is attributed to the corresponding speaker;The voice attributed to each identity is independently transcribed and translated, and the translated text is synthesized into speech and played on the conference terminal;The structured conference record containing the identity of the speaker, the original text of the speech and the translated content is generated and pushed to each intelligent terminal.The application realizes the combination of high-fidelity audio and accurate identity attribution, effectively solves the problem of speaker identity tracing in the scene of multi-person speaking and remote conference.

Description

Technical Field

[0001] This invention relates to the interdisciplinary fields of audio communication, artificial intelligence speech processing, multilingual machine translation, and intelligent conference systems, specifically to a method and system for multilingual conference recording and interpretation based on dual-channel audio comparison.

[0002] This invention is particularly suitable for scenarios such as cross-language remote meetings, international business negotiations, and multilingual academic seminars, which require simultaneous accurate identification of speakers, real-time multilingual transcription and translation, and automatic generation of structured meeting minutes. Background Technology

[0003] With the acceleration of globalization and the widespread adoption of remote work, multi-person meetings spanning multiple languages ​​and regions have become commonplace in business activities, academic exchanges, and government communications. Meeting participants often come from diverse language backgrounds, placing extremely high demands on real-time translation, accurate recording, and subsequent organization of meeting content. While existing technologies have made some progress in speech recognition, machine translation, and meeting recording, the following significant shortcomings remain: (i) The accuracy of speaker identification is low, and it is particularly difficult to deal with scenarios where multiple people speak at the same time. Most existing meeting recording systems rely on voiceprint recognition technology to distinguish different speakers. However, voiceprint recognition technology faces many challenges in practical applications: First, voiceprint recognition requires pre-establishing a voiceprint registration model for each participant, which cannot be effective immediately in ad-hoc meetings or scenarios with guest participation; second, when there are many participants or the speech content is short, the accuracy of voiceprint recognition drops significantly; more challenging is that when two or more people speak simultaneously, the voiceprint features in the mixed audio overlap and interfere with each other, making it almost impossible for existing voiceprint recognition algorithms to effectively distinguish the speech boundaries of each speaker. This leads to a serious mismatch between the speech content and the speaker's identity in the transcribed text, severely affecting the usability of the meeting records.

[0004] (ii) There are inherent defects in the voice capture and identity tagging of remote speakers in remote conferencing scenarios. In remote video conferencing scenarios, the audio of remote participants is mixed by the server and then distributed to the local conferencing terminal in single or multiple-channel mixed format. The local terminal only receives the mixed audio stream and cannot directly distinguish the identities of different remote speakers from this audio stream. Although some conferencing software provides speaker identity metadata information, this information is usually only used for local screen display and is not persistently recorded with the audio stream bound to a timeline, nor is it used for subsequent attribution labeling in speech-to-text transcription. This results in the local recording system only recording "someone spoke at the remote end" in cross-regional multi-person remote conferences, but failing to accurately label "which specific participant at the remote end was speaking," thus breaking the complete chain of speaker attribution in the conference.

[0005] (iii) The processes of speech transcription, translation, and meeting record generation are fragmented and lack an integrated collaborative mechanism. In existing technical solutions, voice acquisition, identity recognition, speech transcription, machine translation, and meeting minutes generation are typically performed by independent modules or systems, lacking effective data transfer and collaborative mechanisms between the various stages. For example, a transcription system only converts audio into text, without considering the speaker's affiliation; a translation system only performs language conversion on plain text, failing to adapt terminology preferences based on speaker identity information; and meeting minutes generation systems often require manual processing of transcription and translation results, which is inefficient and prone to errors. This fragmentation not only reduces overall processing efficiency but also makes end-to-end traceability from raw audio to structured meeting minutes difficult.

[0006] (iv) There is a structural contradiction between the quality of the raw audio collected by the conference terminal and the requirements for identity verification. High-quality speech-to-text transcription requires high-fidelity original audio signals. However, adding identification information to the audio often causes significant damage to audio quality, reducing transcription accuracy. On the other hand, if identification information is not added to the original audio to maintain sound quality, accurate speaker identification cannot be achieved in multi-person conference scenarios. Current technology has failed to effectively resolve this structural contradiction—either sacrifice sound quality for identification, or sacrifice identification for sound quality; both are mutually exclusive.

[0007] (v) Lack of real-time "audio + readable" bimodal interpretation capabilities in multilingual conference scenarios Existing conference translation systems either only display translated text or only provide translated audio playback, making it difficult to simultaneously meet the auditory needs of simultaneous interpretation in a conference room and the visual needs of individual participants to access translated text. Especially in multi-person, multilingual conference scenarios, participants from different language backgrounds have different needs for translated content, and existing systems cannot flexibly adapt to these needs.

[0008] To address the shortcomings of the existing technologies, this invention provides a method and system for multilingual conference recording and interpretation based on dual-channel audio comparison. Through an innovative dual-channel audio acquisition and comparison architecture, it fundamentally solves a series of technical problems, such as the inability to simultaneously achieve identity identification and audio quality, the inability to attribute multiple speakers speaking at the same time, and the inability to uniformly trace remote and local speakers. Summary of the Invention

[0009] The present invention aims to provide a method and system for multilingual conference recording and interpretation based on dual-channel audio comparison, so as to overcome the above-mentioned shortcomings of the prior art.

[0010] Specifically, the main objectives of this invention include: First, it provides an audio acquisition and identity tagging scheme that can accurately bind the voice content of each participant to their identity without compromising the original audio quality. It resolves the structural contradiction between identity tagging and audio quality protection through a dual-channel audio comparison architecture. Second, it provides a sound source separation and identity attribution mechanism that can handle scenarios where multiple people speak at the same time. By comparing multiple identity candidate sets with acoustic feature anchor points, it accurately splits the overlapping mix and completely assigns it to each speaker, ensuring that there is no time slice interweaving or semantic confusion during overlapping speaking periods. Third, a solution is provided that can uniformly handle the identity identifiers of local speakers and remote participants, so that no matter where the speaker is located, their speech can be accurately attributed to the corresponding identity identifier, forming a complete cross-regional meeting speech identity tracing chain. Fourth, it provides an end-to-end integrated system that integrates voice acquisition, identity association, sound source separation, speech transcription, multilingual translation, speech synthesis and broadcasting, and structured meeting minutes generation, realizing fully automated processing from audio signal input to structured meeting document output; Fifth, a conference interpretation solution with dual-modal output capabilities of real-time simultaneous interpretation broadcasting and individual translation text viewing is provided. This solution can meet the needs of the entire conference room to listen to the translated audio, as well as the needs of individuals to view the original text and the translated text, adapting to different usage habits in multilingual conferences.

[0011] The present invention achieves the above-mentioned objectives through the following technical solutions: 1. System Overall Architecture The multilingual conference recording and interpretation system based on dual-channel audio comparison described in this invention consists of a remote server, one or more conference terminals deployed at the conference venue, and a smart terminal carried by each participant. The remote server is the core of the system's data processing; the conference terminals are responsible for acquiring the original audio from the conference venue and playing the translated audio; and the smart terminals are responsible for acquiring individual voice recordings with identification tags and transcribing and displaying the translated text. All three components communicate and collaborate via the internet or a local area network.

[0012] 2. Core Inventive Concept The core concept of this invention lies in constructing two parallel audio acquisition and transmission channels. The first channel consists of smart terminals worn or held by each participant, actively embedding the speaker's unique identification (ID) information into the acquired audio. The second channel consists of conference terminals deployed in the venue, acquiring the mixed raw audio of all participants in the venue, pursuing the highest fidelity without adding any identification markers. After receiving the two audio streams, the remote server uses the identification-included audio from the first channel as the "index anchor" and the high-fidelity raw audio from the second channel as the "content carrier." Through timeline alignment and feature matching, it accurately assigns each segment of speech content in the high-fidelity raw audio to the corresponding speaker's identity, thus simultaneously achieving the two originally conflicting technical goals of "high-fidelity sound quality" and "accurate identity attribution." Based on this, the server further utilizes a built-in speech model to independently perform speech recognition transcription and multilingual translation on the speech content assigned to each identity, generating structured meeting minutes. The translated text is then synthesized into speech and broadcast through the conference terminals, achieving simultaneous interpretation.

[0013] 3. Specific technical solutions for implementation This invention also provides a multilingual conference recording and interpretation system based on dual-channel audio comparison, comprising: Remote server; The conference terminal, deployed at the conference venue, includes at least a first voice acquisition module, a first voice playback module, and a first communication module. The first voice acquisition module is used to acquire a second audio signal containing the voice content of the participants in the conference venue. The second audio signal does not carry the identity information of the participants. The first communication module is used to send the second audio signal to a remote server in the form of a second audio data stream, and to receive second language voice sent by the remote server and play it through the first voice playback module. Multiple smart terminals are provided, with one smart terminal corresponding to each participant. Each smart terminal includes at least a second voice acquisition module, a second communication module, and an identity association encoding module. The second voice acquisition module is used to acquire a first audio signal of the corresponding participant's voice content. The identity association encoding module is used to associate and encode the first audio signal with the participant's unique identity identifier (ID) to generate a first audio data stream carrying the identity identifier information. The second communication module is used to send the first audio data stream to a remote server through a first transmission channel and to receive and display the transcribed or translated text sent by the remote server. Remote servers include: The communication interface module is used to receive a second audio data stream from the conference terminal and a first audio data stream from each smart terminal. The timeline synchronization module is used to synchronize and align the first audio data stream and the second audio data stream. The identity comparison and attribution module is used to determine the speaker's identity corresponding to the voice content in the corresponding time interval of the second audio data stream based on the identity identification information carried in the first audio data stream, and to assign the voice content in the second audio data stream to the corresponding identity identification according to the determined speaker's identity. The speech recognition and transcription module is used to perform speech recognition and transcription on the speech content belonging to each identity identifier, and generate the first language transcribed text string corresponding to each speaker. The speech translation and synthesis module is used to translate the first language transcribed text string into the target second language, generate the translated text string, and synthesize the translated text string into second language speech, which is then sent to the first speech playback module of the conference terminal for playback through the communication interface module; The meeting minutes generation module is used to organize the first language transcribed text strings and translated text strings corresponding to each speaker at least according to the speaking timeline, generate structured meeting minutes containing speaker identity information, original speech content and corresponding translation content, and push the structured meeting minutes to various smart terminals for display through the communication interface module.

[0014] The system architecture described above features clearly defined functional modules that communicate via standardized data interfaces, resulting in excellent modularity and scalability. Each module can be independently upgraded or replaced without affecting the overall stability of the architecture. Furthermore, the system claims and method claims form a dual-track protection framework of "method + system," providing a comprehensive legal deterrent against infringers in patent infringement determinations.

[0015] The present invention also provides a method for multilingual conference recording and interpretation based on dual-channel audio comparison. The method includes the following steps: Step 1: Each participant logs into a remote server through a smart terminal bound to their unique identity (ID) to complete the registration. The remote server establishes an independent storage space corresponding to the unique identity (ID) for the participant. Step 2: When a participant speaks in the meeting room, their smart terminal collects a first audio signal containing the participant's voice content. The first audio signal is then associated and encoded with the participant's unique identifier (ID) to generate a first audio data stream carrying the identifier information. This first audio data stream is then sent to the remote server through a first transmission channel. Simultaneously, a conference terminal deployed at the meeting site collects a second audio signal containing the participant's voice content. This second audio signal does not carry the participant's identifier information. The conference terminal then sends this second audio signal as a second audio data stream to the remote server through a second transmission channel. Step 3: After receiving the first audio data stream and the second audio data stream, the remote server synchronizes and aligns the two on the time axis. Based on the identity information carried in the first audio data stream, it determines the speaker's identity corresponding to the voice content in the corresponding time interval of the second audio data stream, and assigns the voice content in the second audio data stream to the independent storage space under the corresponding identity according to the determined speaker's identity. Step 4: The remote server performs speech recognition and transcription on the speech content belonging to each identity identifier, generates a transcribed text string in the first language corresponding to each speaker, and translates the transcribed text string in the first language into the target second language to generate a corresponding translated text string; the remote server synthesizes the translated text string into the second language speech and sends the second language speech to the conference terminal for playback, realizing simultaneous interpretation; Step 5: The remote server organizes the text strings transcribed in the first language and the translated text strings corresponding to each speaker according to the speaking timeline, generates a structured meeting record containing speaker identity information, original speech content and corresponding translation content, and pushes the structured meeting record to the smart terminals of each participant for display.

[0016] In summary, compared with the prior art, the present invention has the following beneficial effects: First, through a dual-channel audio comparison architecture, it achieves for the first time both high-fidelity original audio and accurate identity attribution, completely resolving the structural contradiction between identity marking and audio quality protection in existing technologies. The audio from the smart terminal provides the identity index anchor point, while the audio from the conference terminal provides the high-fidelity content carrier. The two are merged into one after time axis alignment and feature matching, so that the audio used for speech recognition and transcription is a pure high-fidelity signal, and each audio segment is accurately bound to the speaker's identity.

[0017] Second, by transforming the traditional "blind source separation" into "constrained sound source separation based on known anchor points", the problem of identity attribution in scenarios where multiple people speak at the same time is effectively solved. Since the identities of candidates participating in overlapping speeches have been predetermined by watermark decoding, the separation algorithm does not need to search and match among all participants. It only needs to distinguish between a small number of known candidates, resulting in low computational complexity, high separation accuracy, and the separated audio segments are included in each identity cache in complete form, ensuring the continuity of each speaker's voice before transcription.

[0018] Third, through a unified dual-channel audio comparison framework, unified processing of local speakers and remote participants under the same identity attribution system is achieved. Regardless of whether the speaker is in the local venue or in a remote location, as long as their smart terminal can establish a data connection with the server and upload audio with an ID watermark through the first transmission channel, the server can accurately attribute their speech to the corresponding identity identifier, filling the technical gap in the remote meeting scenario where the identity of remote speakers cannot be accurately traced.

[0019] Fourth, by using the processing method of "first splicing the complete audio stream and then transcribing it as a whole", the transcribed texts of each speaker are ensured to be coherent and consistent in terms of grammatical structure and semantic logic, avoiding word breaks and semantic jumps caused by time slices; at the same time, the semantic logic rearrangement mechanism for overlapping speeches makes the meeting minutes more user-friendly in terms of presentation, taking into account both the accuracy of machine processing and the naturalness of human reading.

[0020] Fifth, the dual-modal output mechanism of synthesized speech from translated text for on-site broadcasting and intelligent terminal display of translated text simultaneously satisfies the group auditory needs of simultaneous interpretation and the individual visual needs of personal viewing. On-site broadcasting allows all attendees to hear the translated content simultaneously, while personal viewing allows each participant to read the translated text carefully as needed. The two output methods run in parallel and complement each other. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the overall architecture of the multilingual conference recording and interpretation system based on dual-channel audio comparison according to the present invention; Figure 2 This is another overall architecture diagram of the system of the present invention; Figure 3 This is a flowchart of the multilingual conference recording and interpretation method based on dual-channel audio comparison of the present invention; Figure 4 This is one specific implementation of the system of the present invention; Figure 5 This is a second specific implementation of the system of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0024] The system architecture of the present invention will be described in a unified manner first, and then the specific implementation methods under two typical application scenarios will be described in detail with reference to Embodiment 1 and Embodiment 2 respectively.

[0025] like Figure 1 and Figure 2 As shown, the multilingual conference recording and interpretation system based on dual-channel audio comparison described in this invention mainly includes the following components: Remote server (100) The remote server 100 is a cluster of computing devices deployed in the cloud or enterprise data center, possessing high-performance computing capabilities, large-capacity storage capabilities, and high-speed network communication capabilities. The remote server 100 has a built-in large-scale speech model, which includes at least an automatic speech recognition submodule 110, a machine translation submodule 120, and a text summarization submodule 130, used to perform speech-to-text transcription, cross-language text translation, and intelligent summarization and generation of meeting minutes, respectively. The remote server 100 also includes a timeline synchronization module 140, an identity comparison and attribution module 150, a speech translation and synthesis module 160, and a meeting record generation module 170, all specifically designed according to this invention.

[0026] Conference terminal (200) The conference terminal 200 is deployed at the conference venue and can be a dedicated conference room audio-visual integrated device or a personal computer equipped with an external microphone array and speakers. The conference terminal 200 includes at least a first voice acquisition module (210), a first voice playback module (220), and a first communication module (230). The first voice acquisition module 210 is preferably a microphone array, capable of picking up the mixed voices of all participants in the conference room, with its pickup range covering the entire conference room. The acquired second audio signal does not contain any digital watermarks, electronic signatures, or identifiers and is pure pulse code modulation format audio. The first voice playback module 220 includes a speaker or amplifier system for playing translated voice data sent from a remote server. The first communication module 230 supports wired Ethernet or high-speed WiFi communication for sending the second audio data stream X2 to the remote server and receiving second language voice data X3 sent from the remote server 100.

[0027] Smart terminals (300) Each participant corresponds to a smart terminal 300. The smart terminal 300 can be a portable device brought by the participant, such as a smartphone, tablet, smartwatch, or dedicated conference microphone accessory. Each smart terminal 300 includes at least a second voice acquisition module (310), a second communication module (320), and an identity association encoding module (330). The second voice acquisition module 310 is a built-in or external microphone used to acquire the first audio signal X of the corresponding participant's voice content. The identity association encoding module 330 is a software algorithm unit running on the smart terminal processor, used to associate the first audio signal X with the participant's unique identity identifier (ID) for encoding processing—embedding a digital watermark code containing the unique identity identifier (ID) in the low-frequency band of the audio signal using spread spectrum modulation. The second communication module supports wireless communication protocols such as WiFi and 4G / 5G, used to send the first audio data stream X1 to a remote server through a first transmission channel, and receive the transcribed or translated text sent by the remote server for display on the screen of the smart terminal 300.

[0028] The data interaction relationships between the above components are as follows: The smart terminal (300) sends the first audio data stream X1 containing the ID watermark to the remote server (100) through the first transmission channel; the conference terminal (200) sends the second audio data stream X2 without the identity marker to the remote server (100) through the second transmission channel; after comparing the two audio streams, the remote server (100) pushes the translated text to each smart terminal (300) for display, and simultaneously sends the translated voice to the conference terminal (200) for playback. The two transmission channels are independent of each other, with the second transmission channel enjoying a higher network transmission priority to ensure low-latency and high-reliability transmission of the original audio in the conference room.

[0029] exist Figure 1During the meeting, participant A1 collects a first voice data stream X1 loaded with an identity ID through a smart terminal 300, while the conference terminal 200 located in the venue collects a second voice data stream X2 without an identity ID; both the first voice data stream X1 and the second voice data stream X2 are in the first language, for example, both are Chinese, and the information carried by the voice content is basically the same. After aligning the first voice data stream X1 and the second voice data stream X2, the remote server 100 extracts a unique identifier ID. Then, it saves the second voice data stream X2 to the storage space or storage cache corresponding to the unique identifier ID. Next, the remote server 100 uses its internal speech recognition submodule 110 to transcribe the second voice data stream X2 into a text string in the first language (Chinese). Then, the machine translation submodule 120 transcribes the first language text string into a text string in the second language (English). The speech translation and synthesis module 160 converts the second language text string into second language voice data X3 and sends it to the conference terminal 200. The first language audio playback module of the conference terminal 200 plays the second language voice data X3, allowing it to be heard by participant A2 in the conference room. Simultaneously, the remote server 100 also uses the conference record generation module 170 to perform logical analysis on the untranslated first language text string or the translated second language text string, generating conference records, which are then sent to the smart terminals 300 of participants A1 and A2 for display.

[0030] In this Figure 1 During the meeting, attendees A1 and A2 did not speak at the same time; the conference terminal 200 only captured the voice of attendee A1.

[0031] Figure 2 This refers to the situation where attendee A1 and attendee A2 speak at the same time.

[0032] II. Explanation of the Working Principle of Core Modules (a) Identity Association Coding Module 330 like Figure 1 and Figure 2 After acquiring the user's original voice signal (first audio signal X), the identity association encoding module 330 of the smart terminal 300 performs the following processing: First, the unique identifier (ID) entered by the user during login is converted into a binary sequence. Then, at a preset time frame length (e.g., 200ms per frame), each frame of audio signal is converted to the frequency domain using a discrete cosine transform. Within a preset low-frequency band in the frequency domain (e.g., 20Hz to 200Hz, a band where human hearing is not very sensitive), the aforementioned binary sequence is modulated with a spreading code and then superimposed with an intensity lower than that of the main audio signal. After embedding, the signal is restored to the time domain audio signal using an inverse discrete cosine transform.

[0033] The selection of this low-frequency band has the following technical significance: 20Hz to 200Hz is below the frequency range where human hearing is most sensitive (usually 500Hz to 2kHz). After embedding the watermark code in this band, the change in audio quality is almost imperceptible to the human ear, ensuring that the naturalness of the sound heard by attendees through the smart terminal's speaker remains unaffected. Furthermore, the unique identifier (ID) will not be arbitrarily discarded during subsequent audio processing. Simultaneously, this frequency band is higher than the dominant frequency of normal ambient noise (usually below 20Hz), ensuring stable transmission and reception of the watermark code in the meeting environment. The use of spread spectrum modulation technology gives the watermark code a certain degree of anti-interference capability and robustness—even if the audio signal has undergone compression encoding or slight filtering, the server can still successfully demodulate the watermark information with a signal-to-noise ratio of no less than a preset threshold (15dB), ensuring the reliability of the entire system in complex network and acoustic environments.

[0034] (II) Time axis synchronization module 140 Because the physical paths, network protocol stacks, and processing delays of the first transmission channel (smart terminal → server) and the second transmission channel (conference terminal → server) are different, there will inevitably be a time difference (usually 50ms to 300ms) when the two audio signals (first audio data stream X1 and second audio data stream X2) arrive at the server. The function of the timeline synchronization module 140 is to eliminate this time difference and at the same time compare and identify the unique identifier ID.

[0035] The specific implementation is as follows: The first audio data stream X1 and the second audio data stream X2 are processed in frames (frame length 20ms), and their respective short-time energy envelope sequences are extracted. Then, a cross-correlation algorithm is used to calculate the cross-correlation function between the two energy envelope sequences. The offset corresponding to the cross-correlation peak is taken as the estimated transmission delay difference between the two audio streams. After obtaining the delay difference, this value is used as a compensation amount to adjust the delay or advance of the second audio data stream X2, so that the two data streams achieve millisecond-level matching and alignment on the time axis. The alignment accuracy is preferably controlled within ±20ms.

[0036] The necessity of this synchronization step lies in the fact that without precise timeline alignment, subsequent identity comparison and attribution would lack a spatiotemporal basis—the identity ID deciphered from the watermark and the corresponding time segment of the voice content in the second audio data stream X2 would be out of sync, resulting in incorrect attribution. Therefore, the timeline synchronization module 140 is a prerequisite for the correct operation of the entire dual-channel comparison scheme.

[0037] (III) Identity Verification and Attribution Module 150 The identity comparison and attribution module 150 is one of the core modules of this invention. It is responsible for comparing and fusing the high-fidelity original audio (second audio signal X2) collected by the conference terminal 200 with the ID watermarked audio (first audio signal X) collected by the smart terminal, so as to realize the binding of "content" and "identity".

[0038] This module processes data window by window, using a preset time window (preferably 200ms, with a sliding step of 50ms). For each window, the following operations are performed: Step 1 (Identity Decoding): Extract the watermark code from the current window of the first audio data stream X1, and demodulate and decode the embedded identity ID. If demodulation and decoding are successful and the confidence level is higher than the preset threshold (70%), the decoded ID is used as a candidate identity for the active speaker in the current window; if the confidence level is lower than the threshold, the window is marked as "unknown identity" and not processed for the time being, pending information from subsequent windows for correction.

[0039] Step 2 (Sound Source Detection): Perform short-time energy envelope analysis on the signal corresponding to the time window in the second audio data stream X2 to detect whether there are sound sources with energy levels higher than a preset energy threshold within the window, and the number of sound sources present. If the detection result shows only one active sound source, the window is marked as a "single-party exclusive window"; if the detection result shows two or more active sound sources, it is marked as an "overlapping speaking window".

[0040] Step 3 (Attribution): For a "single-party exclusive window", all the speech content of that window in the second audio data stream X2 is directly and completely assigned to the independent storage space C1 corresponding to the candidate identity ID. For an "overlapping speech window", all the candidate identity IDs solved in the current window are combined into an identity set, and then transferred to the sound source separation processing submodule to perform mixing and decomposition.

[0041] This frame-by-frame processing method, using windows as units, ensures the system's real-time performance—Server 100 can continuously track the voice boundaries of each speaker with millisecond-level precision, achieving true real-time identity attribution. Simultaneously, by distinguishing between unilateral exclusive speaking and overlapping speaking states and employing different strategies for each, the system completes attribution with extremely low computational overhead during unilateral speaking, while initiating more complex processing during overlapping speaking, achieving a reasonable balance between computational load and processing quality.

[0042] (iv) Sound source separation and processing submodule 151 When the current window is determined to be an overlapping speaking period, the sound source separation processing submodule 151 is activated. The core technical principle of this submodule 151 is to transform the traditional blind source separation problem into a "constrained separation problem with known quantity and known anchor points", thereby significantly reducing computational complexity and improving separation accuracy.

[0043] The specific processing flow is as follows: First, acoustic feature anchors are extracted from the accumulated speech data of each candidate identity in historical non-overlapping time periods. Then, using these feature anchors as initialization guide vectors, a constrained independent component analysis (ICA) algorithm is executed on the mixed data within the overlapping window. This algorithm, based on classic ICA, adds the constraint that "the separation result should be similar to the features of the corresponding anchor points," effectively avoiding the inconsistency in separation results caused by random initialization in traditional ICA.

[0044] The ingenuity of this scheme lies in the fact that, since the identities of the candidates participating in overlapping speeches are pre-determined by the ID watermark decoding results in the first audio stream, the separation algorithm does not need to blindly search for matches among all participants; it only needs to distinguish between the few known candidates. This reduces the computational complexity from exponential to linear, while the separation accuracy is significantly improved due to the reliable acoustic reference anchors—achieving both without sacrificing one for the other.

[0045] After separation, the mixed signal is split into several clean audio segments, each belonging to a different candidate identity ID. Each audio segment is appended to the corresponding identity's audio buffer queue or independent storage space in a complete form, without time-slice interleaving. "Complete and without time-slice interleaving" means that audio segments belonging to the same speaker are stored continuously in the buffer, rather than being interleaved with data from different speakers in millisecond-level time slices. This feature ensures that the speech content of the same speaker is continuous and uninterrupted during subsequent speech recognition and transcription, thus avoiding grammatical inconsistencies and semantic confusion caused by time-slice breaks in the transcribed text.

[0046] (v) Speech recognition submodule 110 and machine translation submodule 120 When a sufficiently long continuous voice data X2 accumulates in the audio buffer queue or independent storage space C1 corresponding to each identity (preferably with a cumulative duration ≥ 2 seconds), the voice recognition submodule 110 is triggered. This voice recognition submodule 110 processes the audio buffer (first audio data stream X1 and second audio data stream X2) for each identity independently: Step 1 (Audio Concatenation): For voice content belonging to the same identity ID, since the content is already stored contiguously in the audio cache, this step only requires concatenating the beginning and end of each voice segment into a complete audio stream in chronological order. If there are minor time gaps (<50ms) in the cache due to network transmission jitter, silence padding is applied to maintain the continuity of the audio stream.

[0047] The second step (parallel transcription): The concatenated complete audio stream is sent to the automatic speech recognition submodule 110 of the large-scale speech model for decoding. This submodule 110 is based on an end-to-end Transformer architecture and has been trained on a large-scale multilingual corpus, enabling it to support speech recognition in mainstream languages ​​such as Chinese, English, Japanese, and French. For audio streams with multiple identifiers, the transcription module supports parallel and independent processing without interference, making full use of the server's multi-core computing resources.

[0048] Step 3 (Sentence-by-Sentence Translation): The transcribed first-language text string is fed into the translation submodule 120 of the same large speech model and translated into the target second-language text string. During the translation process, the topic vectors of at least three preceding sentences are retained through a context caching mechanism to improve the consistency and accuracy of terminology translation.

[0049] Step 4 (Speech Synthesis and Broadcast): The translated second language text string is fed into the speech translation and synthesis module 160 to be synthesized into second language speech. After compression and encoding, the synthesized second language speech is sent to the first speech playback module 220 of the conference terminal 200 for broadcast in the conference room, realizing the simultaneous interpretation function. At the same time, the original translated second language text string is pushed to the smart terminals 300 of each participant through a separate channel for personal viewing.

[0050] This approach of "first assembling the complete audio stream, then transcribing it as a whole" has the technical advantage of allowing speech recognition models to utilize longer contextual information when decoding continuous speech. When the model obtains contextual cues from the semantic and acoustic information of the previous sentence, the decoding accuracy of the subsequent sentence significantly improves. If each time segment is transcribed separately in isolation and then concatenated, each segment loses its contextual "correspondence," resulting in a significant decrease in accuracy and fluency. Therefore, this processing step plays a crucial role in ensuring the quality of the final transcribed text.

[0051] (vi) Meeting minutes generation module 170 and semantic reordering submodule 171 The meeting minutes generation module 170 is responsible for generating structured meeting minutes by organizing the transcribed text (first language text string) and translated text (second language text string) of each speaker in a certain logical manner.

[0052] The baseline organization method is to sort by the original physical timeline of the speeches, using the speaker's identity as the first-level index, and generating a four-tuple record of "speaker-time-original text-translation". However, in scenarios where multiple people speak simultaneously, directly arranging them linearly along the timeline may lead to a poor reading experience—for example, when A and B speak at the same time, if the presentation is strictly interwoven along the timeline, the reader will repeatedly jump between sentences A and B, making it difficult to understand coherently.

[0053] To this end, the meeting record generation module 170 incorporates a semantic reordering submodule 171. This semantic reordering submodule 171 is triggered when the overlap rate of the transcribed text corresponding to two or more identity IDs on the physical timeline reaches a preset threshold (≥70%). Step 1 (Pragmatic Function Classification): Invoke a lightweight semantic classifier, which is based on the BERT distillation model architecture and can classify the input single-sentence text into pragmatic function labels such as "social opening", "substantive resolution", "supplementary explanation" or "closing acknowledgment".

[0054] The second step (determining the logical order): Based on the classification results, determine whether there is a natural logical sequence between the overlapping sentence pairs. For example, "social opening phrases" (such as "Good afternoon, everyone") logically must precede "substantive decision phrases" (such as "Today's agenda is as follows"), and there is a sequential relationship between the two.

[0055] Step 3 (Rearrangement of Presentation Order): If a logical sequential relationship is determined and the classification confidence level is higher than a preset threshold (≥65%), the presentation order of overlapping sentence pairs in the final meeting minutes is rearranged—sentences of "social opening" or "closing acknowledgments" are forced to the front, and sentences of "substantive resolutions" are moved to the back. Logical connectors (such as "subsequently," "immediately following," "at the same time," etc.) are automatically inserted between adjacent sentences based on context. After the rearrangement is completed, the meeting minutes metadata is marked "This section has been semantically and logically rearranged," and the original physical timestamp information corresponding to each sentence is retained for users to trace and replay the original audio.

[0056] Step 4 (low confidence fallback): If the confidence level is lower than the threshold, or the classifier cannot determine the logical relationship, then abandon the order reordering and instead display all overlapping sentences in the meeting minutes in a parallel left and right column format, and automatically generate a merged summary above the column.

[0057] This semantic-logical rearrangement mechanism is a necessary supplement to sound source separation at the physical acoustic level. Physical acoustic separation addresses the question of "whether the mixed audio can be broken down and each part transcribed into a coherent sentence," while logical rearrangement addresses the question of "in what order should the separated audio be presented to a human for optimal reading comprehension." The combination of the two ensures both the scientific rigor of machine processing (acoustic separation) and the natural feel of human reading (logical rearrangement). The fallback solution for parallel display in low-confidence scenarios guarantees that the system has a reasonable output strategy under all circumstances, preventing output chaos due to model misjudgments.

[0058] After the structured meeting minutes are generated, the remote server 100 further calls the text summarization submodule 172 of the speech big data model to perform semantic understanding and summarization on the complete dialogue record, automatically extracting the core conclusions of the meeting, the to-do list, and the main points of each speaker, and generating a meeting minutes summary. The summary generation adopts a hard-constrained prompt word template, forcing the model to summarize only based on the original text content, prohibiting the addition of new information not included in the original text, so as to prevent AI fabrication.

[0059] Ultimately, the complete meeting minutes (including structured dialogue records, AI summaries, links to the original audio, and encrypted hash checksums) were pushed to the smart devices of all participants and simultaneously encrypted and archived on a cloud server. The addition of the encrypted hash checksums creates an immutable link between the archived file and the original recording—any tampering with the record will result in a hash mismatch, thus being detected. This meets the stringent evidentiary requirements for compliant archiving of meeting recordings.

[0060] like Figure 3 The present invention also discloses a method for multilingual conference recording and interpretation based on dual-channel audio comparison, the method comprising the following: Step 1: Registration and Storage Space Setup Each participant registers by logging into the remote server using a smart terminal linked to their unique identifier (ID). The remote server then creates an independent storage space corresponding to that unique identifier (ID) for each participant. This unique identifier (ID) can be the participant's employee ID, mobile phone number, email address, or other information that can uniquely identify a natural person. The independent storage space includes at least an audio cache queue for caching audio data and a text group pool for storing transcribed text.

[0061] By establishing independent storage space for each participant on the server side, the audio and text data of each speaker can be stored and accessed independently during subsequent processing. This avoids crosstalk and confusion between different speakers' data and allows the speech processing tasks of each speaker to be scheduled in parallel, thereby significantly improving the server's processing efficiency.

[0062] Step 2: Dual-channel audio acquisition and uploading When a participant speaks at the meeting, their smart terminal collects a first audio signal containing the participant's voice content. The first audio signal is then associated with and encoded with the participant's unique identifier (ID) to generate a first audio data stream carrying the identifier information. This first audio data stream is then sent to a remote server through a first transmission channel. Simultaneously, a conference terminal deployed at the meeting site collects a second audio signal containing the participant's voice content. This second audio signal does not carry the participant's identifier information. The conference terminal then sends this second audio signal as a second audio data stream to the remote server through a second transmission channel.

[0063] It's important to note that while both audio acquisition channels capture the same acoustic event (i.e., the same speech by the same participant) within the same time window, they differ in the acquisition devices, transmission paths, and the additional information they carry. The smart terminal captures sound close to the speaker's mouth, resulting in a high acoustic signal-to-noise ratio. With embedded watermarks, this sound can be used for reliable identity verification. The conference terminal, on the other hand, captures sound attenuated by air propagation. Due to the fixed microphone array position and typical distance from the speaker, the signal-to-noise ratio is low, but its advantage lies in its pure sound quality, free from any digital labeling degradation. The two channels complement each other: the smart terminal's audio's "identity labeling function" compensates for the conference terminal's audio's "unclear speaker" deficiency, while the conference terminal's audio's "high-fidelity characteristics" compensate for the smart terminal's audio's "minor sound quality loss due to watermark embedding."

[0064] In this step, preferably, the intelligent terminal associates and encodes the first audio signal with a unique identifier (ID) by embedding a digital watermark code containing the unique identifier (ID) within a preset frequency band of the first audio signal. The reason for choosing to embed within a preset frequency band rather than the entire frequency band is to minimize the impact of the watermark on speech intelligibility and naturalness while ensuring reliable transmission of the watermark information. More preferably, the digital watermark code is embedded in the low-frequency band of the first audio signal using spread spectrum modulation. The embedding strength is controlled by ensuring that the watermark demodulation signal-to-noise ratio is not lower than a preset threshold, and the audio signal after watermark embedding has no significant difference in subjective auditory quality from the original audio signal. Spread spectrum modulation watermarking technology is mature and reliable, possessing inherent anti-interference capabilities. The watermark signal can still be reliably detected after undergoing various adverse factors such as speech compression coding, network packet loss, and environmental noise. The choice of low frequency band (preferably 20Hz to 200Hz) has two rational reasons: this frequency band is below the most sensitive area of ​​human hearing, so the quality loss after watermark embedding is difficult for the human ear to detect; at the same time, this frequency band is above the main frequency band of most environmental noise, so the watermark signal is not easily drowned out by the environmental noise of the venue, ensuring the reliability of demodulation on the server side.

[0065] Preferably, the second audio signal acquired by the conference terminal is raw pulse code modulation (PCM) audio without any digital watermarks, electronic signatures, or identifiers, with a sampling rate of at least 16 kHz and a quantization bit depth of at least 16 bits. The requirement that the second audio signal be free of any identifiers ensures the highest possible audio fidelity, providing optimal audio input for subsequent speech recognition and transcription. PCM is used because it is the most basic audio encoding format, introducing no lossy compression distortion and preserving the acoustic details of the original speech to the greatest extent possible.

[0066] Step 3: Dual-channel comparison and identity attribution After receiving the first audio data stream and the second audio data stream, the remote server synchronizes and aligns the two on the timeline. Based on the identity information carried in the first audio data stream, it determines the speaker's identity corresponding to the voice content in the corresponding time interval of the second audio data stream, and assigns the voice content in the second audio data stream to the independent storage space under the corresponding identity according to the determined speaker's identity.

[0067] The necessity of timeline synchronization lies in the fact that the first audio data stream is transmitted to the server via the smart terminal's wireless network (WiFi / 4G / 5G), while the second audio data stream is transmitted to the server via the conferencing terminal's wired network or high-priority WiFi. The physical links, protocol stacks, and transmission delays of these two paths differ, typically exhibiting a time difference of 50ms to 300ms. If the two signals are not timeline-aligned and directly compared, an error will occur, such as "matching A's speech watermark at 10:00:05 with B's audio signal at 10:00:05.200." Therefore, timeline synchronization is the prerequisite and foundation for the correct operation of the entire dual-channel comparison scheme.

[0068] Preferably, the specific method for time axis synchronization alignment is as follows: calculate the transmission delay difference between the two audio data streams, and use this transmission delay difference as a compensation amount to match and align the two data streams on the time axis. In specific implementation, a cross-correlation algorithm can be used to calculate the peak offset between the energy envelope sequences of the two signals, and this offset can be used as an estimate of the delay difference. Then, time axis compression or delay compensation is performed on the signal with the larger delay. The cross-correlation algorithm has high computational efficiency, is simple to implement, and can complete alignment with millisecond-level accuracy, fully meeting the processing delay requirements of real-time call scenarios.

[0069] The attribution determination process needs to be accurate down to each time segment. Preferably, the specific method is as follows: using a preset time window as the unit, the identity information carried in each time window of the first audio data stream is extracted to obtain the candidate identities of active speakers within the current time window; simultaneously, a short-time energy envelope detection method is used to detect the number of active sound sources within the corresponding time window in the second audio data stream; if there is only one active sound source above a preset energy threshold within the current time window, the second audio data stream within that time window is directly assigned to a candidate identity; if there are two or more active sound sources above the preset energy threshold within the current time window, it is determined to be an overlapping speaking period, and the process proceeds to separate attribution processing based on a multi-identity candidate set. This method uses the time window as the processing unit, which can continuously track the speaker's voice boundary with millisecond-level accuracy, achieving true real-time identity attribution, rather than just offline processing. By distinguishing between unilateral exclusive speaking and overlapping speaking states and adopting different strategies for each, the system can complete the attribution with extremely low computational overhead during unilateral speaking, and initiate a more complex processing flow during overlapping speaking, achieving a reasonable balance between computational load and processing quality.

[0070] When an overlapping speaking period is identified, more preferably, the remote server extracts multiple candidate identities from the first audio data stream uploaded from each smart terminal within the current time window to determine the set of all identity identifiers participating in the overlapping speaking. Using the acoustic features accumulated by each candidate identity during the non-overlapping period as reference anchors, anchor-guided source separation processing is performed on the mixed data of the corresponding overlapping period in the second audio data stream. This splits the mixed signal into several audio segments belonging to each candidate identity, and each audio segment is completely included in the corresponding identity's speech content cache. The core advantage of this scheme is that it transforms the traditional "blind source separation" problem into a "constrained separation problem based on known anchors." Since the identities of the candidates participating in the overlapping speaking have been predetermined by the watermark decoding results, the separation algorithm does not need to blindly search across all participants; it only needs to distinguish between a small number of known candidates. The computational complexity is reduced from exponential to linear, while the separation accuracy is significantly improved due to the reliable acoustic reference. The separated pure audio segments are appended to each identity cache in a "complete inclusion" rather than "time-slice interleaving inclusion" manner, which ensures that the voices of the same speaker are stored continuously in the cache, directly ensuring the grammatical coherence and semantic consistency of the subsequent speech recognition and transcription results.

[0071] Step 4: Transcription, Translation, and Voice Broadcast The remote server performs speech recognition and transcription on the speech content belonging to each identity, generating a first-language transcribed text string for each speaker, and then translates the first-language transcribed text string into the target second language, generating a corresponding translated text string. The remote server then synthesizes the translated text string into second-language speech and sends the second-language speech to the conference terminal for playback, thus achieving simultaneous interpretation.

[0072] Each speaker's speech is transcribed and translated independently, ensuring that the transcribed text is linked to their individual identity. This avoids the inaccurate approach of transing the entire mixed audio and then guessing who spoke which sentence. For speech belonging to the same identity, it is preferable to first concatenate the audio into a continuous, complete audio stream in chronological order before transing, and then decode and transcribe this complete audio stream as a whole, rather than transing each time segment separately and then concatenating the text. The reason for "concatenating the audio first, then transing" is that speech recognition models can utilize longer contextual information when decoding continuous speech—when the model obtains contextual clues from the semantic and acoustic information of the previous sentence, the decoding accuracy of the next sentence will significantly improve. If each time segment is transcribed in isolation, the transcription of each segment loses its contextual "correspondence," and its accuracy and fluency will significantly decrease.

[0073] By synthesizing the translated text into speech and playing it on the conference terminal, simultaneous interpretation functionality was achieved, enabling all participants in the venue (including those without smart devices) to hear the translated audio content in real time. Simultaneously, the translated text itself could also be pushed to various smart devices for individual viewing via another channel, forming a dual-modal output of "audio + readable," thus meeting the usage needs of translated content in different scenarios.

[0074] Step 5: Generate and push structured meeting minutes The remote server organizes the first-language transcribed text strings and translated text strings corresponding to each speaker at least according to the speaking timeline, generates a structured meeting record containing speaker identity information, original speech content and corresponding translation content, and pushes the structured meeting record to the smart terminals of each participant for display.

[0075] When generating structured meeting minutes, if two or more speakers overlap in their statements, simply arranging them linearly along the physical timeline can lead to repeated skipping by the reader, resulting in a poor user experience and potential misunderstandings. Preferably, when the remote server detects that the overlap rate of the first-language transcribed text strings corresponding to two or more identity identifiers on the physical timeline reaches a preset threshold, the remote server calls a semantic classifier to perform pragmatic function classification on the overlapping transcribed texts and determines whether a logical sequential relationship exists based on the classification results. If such a relationship exists, the display order of the overlapping transcribed texts is logically rearranged when generating the structured meeting minutes, and the original physical timestamp information corresponding to each transcribed text is retained in the rearranged minutes. If no such relationship exists or the confidence level is below a preset threshold, the transcribed texts of each speaker are presented simultaneously at the same timeline position in a parallel display manner.

[0076] This semantic-logical reordering mechanism is a necessary supplement to the aforementioned audio-level sound source separation. Physical acoustic separation solves the problem of "whether the mixed audio can be broken down and transcribed into coherent sentences," while logical reordering addresses the question of "in what order, after being broken down, is the most fluent for human readers." The combination of the two ensures both the scientific rigor of machine processing (acoustic separation) and the natural feel of human reading (logical reordering). The significance of retaining the original physical timestamp information after reordering lies in the fact that although the display order is adjusted, the original timeline anchor point of each statement is still preserved, giving the meeting minutes both "logical readability" and "original traceability"—which is particularly crucial for serious applications such as legal compliance and auditing.

[0077] III. Example 1: Multi-person meeting scenarios within a single conference terminal like Figure 4 This embodiment describes an application scenario of the present invention in a single physical conference room. For example... Figure 4 As shown, there are three participants, A, B, and C, in the same conference room, where A is the conference host and B and C are attendees. A conference terminal (200) is deployed in the conference room. The microphone array of the first voice acquisition module 210 of the conference terminal is mounted in the center of the conference table, and the speakers (220) are installed on the walls or ceiling of the conference room. Each participant carries their own logged-in smart terminal (300A, 300B, 300C) – in this embodiment, the smart terminal is their personal smartphone, which has the dedicated APP of this invention installed and has been logged in through their enterprise account.

[0078] (I) Meeting Start-up and Equipment Preparation Phase Before the meeting begins, participants A, B, and C each open a dedicated app on their smartphones and log in by entering their unique identification ID (company ID or mobile phone number). Upon receiving the registration requests from the three IDs, the remote server (100) establishes independent storage spaces for A, B, and C, including an audio cache queue and a transcribed text group pool. The server then returns the current meeting's Session_ID to the three smart terminals and the meeting terminal, confirming that the system is ready.

[0079] (II) Voice Acquisition and Dual-Channel Upload Stage Assuming the meeting is in progress, participant A begins by saying: "Today's meeting topic is the new product launch plan." During this process, participant A's smartphone (300A) picks up A's voice through its built-in microphone (310A). The picked-up analog voice signal is converted from analog to digital and then processed by the identity association encoding module: with a time frame period of 200ms, A's unique identity identifier (ID) is embedded in the low-frequency band of each audio frame using spread spectrum modulation, generating a first audio data stream containing an ID watermark. This data stream is uploaded to a remote server (100) in real time via the smartphone's WiFi module.

[0080] Almost simultaneously, A's voice wave travels through the air for approximately 0.5 ms before reaching the microphone array (210) of the conference terminal. The microphone array picks up the voice, and after echo cancellation and noise reduction processing, directly generates a second audio signal in pulse code modulation format without any identity markers. This signal is then uploaded in real time to a remote server (100) via the wired Ethernet interface of the conference terminal.

[0081] Here we can clearly see that the two channels are collecting the same sound event at the same time, but due to the different distances of the collecting devices, the different transmission paths, and the different information attached, they form a natural complementary relationship—the smart terminal is close to the sound source and has a high signal-to-noise ratio, making it suitable for carrying watermark information; the conference terminal is farther from the sound source but has pure sound quality, making it suitable as the source of content for transcription.

[0082] (III) Dual-path comparison and identity attribution stage The remote server (100) continuously receives the first audio data stream from three smart terminals and the second audio data stream from the conference terminal. The timeline synchronization module processes continuously in 200ms time windows: it calculates the energy envelope for the first and second data streams respectively, and calculates the time delay difference through a cross-correlation algorithm.

[0083] In this embodiment, due to the difference in physical paths between the smart terminal uploading via WiFi and the conference terminal uploading via wired network, there is typically a latency difference of approximately 80ms to 150ms. The server uses this latency difference as compensation to align the second data stream with the first data stream after delaying it accordingly.

[0084] After alignment, the identity comparison and attribution module processes the current 200ms window: It demodulates the watermark encoding from this window in the first data stream, successfully decrypting the ID "EMP_A001" with a confidence level of 92%. Simultaneously, it performs short-time energy envelope detection on the audio in the corresponding window of the second data stream, detecting one active sound source (A's speech) with an energy level above a preset threshold; no other sound sources are detected. Since there is only one active sound source in the current window and the decrypted candidate identity is A, the attribution and allocation unit directly and completely adds the entire high-fidelity original audio segment within this window to the audio buffer queue, which is marked as Audio_Buffer_A.

[0085] (iv) Stage of handling overlapping statements Suppose the meeting enters the open discussion phase, and A and B begin to speak at the same time: A said, "I think it should be released in the third quarter." B said almost simultaneously, "I suggest postponing it to the fourth quarter." The two audio streams are superimposed in the air and simultaneously picked up by the microphone array of the conference terminal, forming a mixed signal. At the same time, A and B's smartphones independently capture their own audio, embed their respective ID watermarks, and upload them to the server.

[0086] When the server identity comparison and attribution module processes the 200ms window, it simultaneously extracts IDs "EMP_A001" (90% confidence) and "EMP_B002" (87% confidence) from the first data stream, determining the candidate identity set for the current window to be {A, B}. The sound source detection unit performs energy envelope analysis on the second data stream (mixed signal) of the window, detecting two peaks higher than the preset energy threshold, determining them as overlapping speaking periods, and triggering the sound source separation processing submodule.

[0087] The source separation processing submodule first extracts historical speech segments of A in non-overlapping time periods from Audio_Buffer_A, and extracts their MFCC feature centroids as anchor points A; similarly, it extracts anchor points B from Audio_Buffer_B. Then, using anchor points A and B as initial guide vectors, it performs constrained independent component analysis on the mixed signal of the 200ms window. After 15 iterations, the mixed signal is split into two signals—one dominated by the acoustic features of A, and the other dominated by the acoustic features of B.

[0088] After splitting, the clean audio segment belonging to A is completely appended to Audio_Buffer_A, and the clean audio segment belonging to B is completely appended to Audio_Buffer_B. Both audio segments are stored in a continuous and complete form, without millisecond-level time-slice interleaving. This ensures that during subsequent transcription, the speech content of A and B are continuous and complete in the buffer, and there will be no situation where "A's sentence is interrupted by B's time slice".

[0089] (V) Transcription, Translation and Structure Generation Stage Once a sufficiently long continuous audio stream has accumulated in Audio_Buffer_A, the speech recognition and transcription module is triggered. This module reads the complete continuous audio stream from Audio_Buffer_A and feeds it into the ASR submodule of the large speech model for decoding and transcription, outputting the Chinese text "I think it should be released in the third quarter," which is then translated into English by the translation submodule.

[0090] At the same time, the audio of the corresponding overlapping period of Audio_Buffer_B was also independently transcribed and translated, outputting the Chinese text "I suggest postponing it to the fourth quarter" and the English translation "I suggest postponing it to the fourth quarter".

[0091] Because the overlap between statements A and B on the physical timeline reaches 100%, the semantic reordering submodule is triggered. A lightweight semantic classifier performs pragmatic function determination on the two texts—A's "I think..." is classified as "opinion expression," and B's "I suggest..." is also classified as "opinion expression," even though there is no obvious logical order between them. The semantic classifier outputs a confidence score of 55%, which is lower than the preset threshold of 65%. Therefore, the system abandons the sequential reordering and instead displays the text in parallel—in the final generated structured meeting minutes, the statements of A and B are displayed simultaneously below this point in time in a side-by-side bubble format.

[0092] (vi) Meeting minutes generation and distribution stage After the meeting concludes, the meeting recording module on the remote server organizes the accumulated structured dialogue records into a complete meeting transcript along a timeline. Subsequently, the meeting summary generation module calls the text summarization submodule of the speech model to extract the core conclusions, to-do items, and key viewpoints of each party from the complete transcript. The complete meeting minutes file is then pushed to the smartphone apps of participants A, B, and C, and simultaneously backed up encrypted to the server cloud.

[0093] IV. Example 2: Cross-regional remote conferencing scenarios between two conferencing terminals This embodiment describes an application scenario of the present invention for cross-regional remote conferencing between conferencing terminals in two different physical locations. For example... Figure 5 As shown, the near-end venue is located in Beijing, with participants A (local host) and B (local attendees), and is equipped with near-end conferencing terminals. The far-end venue is located in Shanghai, with participants C (remote speaker) and D (remote attendees), and is equipped with remote conferencing terminals. Both locations establish connections with the remote server via the Internet. A, B, C, and D each carry a logged-in smart terminal.

[0094] (I) Explanation of Specific Issues in the Scenario This embodiment differs from Embodiment 1 in two key ways: Difference 1: The voices of participants C and D in the remote meeting room (Shanghai) are first captured by the microphone array of the remote conference terminal, mixed by the server, and then sent to the near-end conference terminal. The audio played by the near-end conference terminal is a mixed signal of C and D—the near-end system cannot directly distinguish from this mix which part is spoken by C and which part is spoken by D.

[0095] Difference 2: Although C and D were located in the Shanghai venue, their respective smart terminals were also located within the Shanghai venue. The collected audio with ID watermarks was directly uploaded to the server through the first transmission channel. This means that although C and D were not physically present in Beijing, the server was still able to receive the personal audio streams with identity tags from Shanghai, thus providing an "index anchor" for identifying the remote speakers.

[0096] This feature demonstrates the core advantage of this invention: identity attribution does not depend on the speaker's physical location, but rather on an independent data channel between the smart terminal and the server. No matter where the speaker is in the world, as long as their smart terminal can connect to the internet and upload audio containing an ID watermark, the server can accurately assign their speech to the corresponding identity identifier.

[0097] (ii) Dual-channel acquisition and uploading during remote speaker presentations Suppose that participant C in the remote meeting room begins to speak: "Based on the production capacity of the Shanghai factory, we can complete 100,000 orders in Q3." C's voice is captured by two devices simultaneously: First path: C's smartphone captures C's voice through its microphone, embeds C's ID watermark in the low-frequency band, and then uploads it directly to the remote server via WiFi; Second path: The microphone array of the remote conferencing terminal captures C's voice, and after noise reduction processing, uploads it to the hosting server in high-fidelity original audio format via a dedicated network.

[0098] (III) Server-side dual-path comparison and identity attribution After receiving the first audio data stream from smartphone C and the second audio data stream from the remote conferencing terminal, the remote server performs a dual-channel comparison operation: the time axis synchronization module calculates the time delay difference between the two data streams using a cross-correlation algorithm and completes the alignment; the identity comparison and attribution module decodes the watermark encoding from the first data stream in 200ms windows to obtain the ID "EMP_C003", and at the same time performs energy detection on the corresponding window of the second data stream, and completely assigns the high-fidelity original audio to Audio_Buffer_C.

[0099] (iv) Retention of identity information during mixing, distribution, and near-end playback The remote server uses the completed audio of C for two purposes: first, to send the speech in Audio_Buffer_C to the speech recognition, transcription and translation module for processing; second, to mix the speech of C with the speech of D and other remote participants, encode it and send it to the local conference terminal for playback.

[0100] The key point is that although the audio sent to the near-end conference terminal is in mixed form, the server has already classified the content of that time period as C through dual-channel comparison. Therefore, in the final generated conference record, no matter how the sound of C reaches the near-end speaker through the transmission link, its transcribed text is correctly labeled as "Speaker: C (remote Shanghai)" - rather than simply labeled as "remote speaker".

[0101] (v) Overlapping processing of simultaneous proximal and distal speech Suppose that A in Beijing and C in Shanghai speak simultaneously. The server simultaneously receives audio with ID watermarks from A and C's smartphones, as well as mixed audio signals from near-end and far-end conference terminals. The identity comparison and attribution module processes independently within their respective time windows—attributing the audio from the near-end in Beijing to A, and the audio from the far-end in Shanghai to C. When the audio from both overlaps on the timeline, the server performs independent transcription and translation for A and C respectively. The semantic reordering submodule performs pragmatic function classification on the two overlapping sentences and presents them in a parallel manner.

[0102] (vi) Translated voice broadcast by remote loudspeaker The simultaneous interpretation function of this invention is particularly important in remote conferencing scenarios. Assuming participant A in a Beijing meeting does not understand Chinese, while participant C speaks Chinese, the server transcribes and translates C's Chinese speech into English. The translated text is then output in two formats: the English translation is pushed to A's smart terminal screen for display (readable), and simultaneously, the English translation is synthesized into English speech and sent via a remote server to the speaker of the nearby meeting terminal for playback (audible). This dual-modal output allows simultaneous interpretation to serve both group scenarios requiring dictation and individual research scenarios requiring reading translation, without conflicting requirements.

[0103] (vii) Generation of complete meeting minutes and cross-regional archiving After the meeting, the remote server aggregates the transcription and translation results of all speakers, generating a complete structured meeting transcript covering all speakers at both the local and remote ends. Each speaker's location is indicated in the transcript, and the timeline uses the server's UTC time to ensure global consistency across geographical locations. The complete meeting minutes are pushed to the smart terminals of all participants in both locations and encrypted and archived in the cloud.

[0104] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0105] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0106] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for multilingual conference recording and interpretation based on dual-channel audio comparison, characterized in that, The method includes the following steps: Step 1: Each participant registers by logging into the remote server through a smart terminal bound to their unique identifier (ID). The remote server then creates an independent storage space for each participant corresponding to their unique identifier (ID). Step 2: When a participant speaks in the meeting room, their smart terminal collects a first audio signal containing the participant's voice content. The first audio signal is then associated and encoded with the participant's unique identifier (ID) to generate a first audio data stream carrying the identifier information. This first audio data stream is then sent to the remote server through a first transmission channel. Simultaneously, a conference terminal deployed at the meeting site collects a second audio signal containing the participant's voice content. This second audio signal does not carry the participant's identifier information. The conference terminal then sends this second audio signal as a second audio data stream to the remote server through a second transmission channel. Step 3: After receiving the first audio data stream and the second audio data stream, the remote server synchronizes and aligns the two on the time axis. Based on the identity information carried in the first audio data stream, it determines the speaker's identity corresponding to the voice content in the corresponding time interval of the second audio data stream, and assigns the voice content in the second audio data stream to the independent storage space under the corresponding identity according to the determined speaker's identity. Step 4: The remote server performs speech recognition and transcription on the speech content belonging to each identity identifier, generates a transcribed text string in the first language corresponding to each speaker, and translates the transcribed text string in the first language into the target second language to generate a corresponding translated text string; the remote server synthesizes the translated text string into the second language speech and sends the second language speech to the conference terminal for playback, realizing simultaneous interpretation; Step 5: The remote server organizes the text strings transcribed in the first language and the translated text strings corresponding to each speaker according to the speaking timeline, generates a structured meeting record containing speaker identity information, original speech content and corresponding translation content, and pushes the structured meeting record to the smart terminals of each participant for display.

2. The method according to claim 1, characterized in that, In step 2, the specific method by which the smart terminal associates the first audio signal with the unique identifier (ID) of the participant is as follows: embedding a digital watermark code containing the unique identifier (ID) within a preset frequency band of the first audio signal.

3. The method according to claim 2, characterized in that, The digital watermark encoding is embedded in the low-frequency band of the first audio signal using spread spectrum modulation. The embedding strength is controlled by ensuring that the watermark demodulation signal-to-noise ratio is not lower than a preset threshold. Furthermore, the audio signal after watermark embedding has no significant difference in subjective auditory quality from the original audio signal.

4. The method according to claim 1, characterized in that, In step 2, the second audio signal collected by the conference terminal is the original pulse code modulation format audio that does not contain any digital watermark, electronic signature or identity identifier.

5. The method according to claim 1, characterized in that, In step 3, the specific method by which the remote server performs time axis synchronization and alignment of the first audio data stream and the second audio data stream is as follows: calculate the transmission delay difference between the two audio data streams, and use the transmission delay difference as a compensation amount to match and align the two data streams on the time axis.

6. The method according to claim 1, characterized in that, In step 3, the remote server determines the speaker's identity corresponding to the voice content in the second audio data stream based on the identity information carried in the first audio data stream as follows: Using a preset time window as a unit, the identity information carried in each time window of the first audio data stream is extracted to obtain the candidate identities of active speakers within the current time window; simultaneously, a short-time energy envelope detection method is used to detect the number of active sound sources within the corresponding time window in the second audio data stream; if there is only one active sound source above a preset energy threshold within the current time window, the second audio data stream within that time window is directly assigned to the candidate identity; if there are two or more active sound sources above a preset energy threshold within the current time window, it is determined to be an overlapping speaking period, and the process is transferred to the separation and assignment processing based on a multi-identity candidate set.

7. The method according to claim 6, characterized in that, The separation and attribution processing based on the multi-identity candidate set is specifically as follows: The remote server extracts multiple candidate identities from the first audio data stream uploaded from each smart terminal within the current time window, determines the set of all identity identifiers participating in overlapping speech; using the acoustic features accumulated by each candidate identity in the non-overlapping time period as reference anchor points, it performs anchor-guided sound source separation processing on the mixing data of the corresponding overlapping time period in the second audio data stream, splits the mixing signal into several audio segments belonging to each candidate identity, and completely assigns each audio segment to the independent storage space of the corresponding identity.

8. The method according to claim 1, characterized in that, In step 4, before the remote server performs speech recognition and transcription on the speech content belonging to each identity and stored in their respective independent storage spaces, for the speech content belonging to the same identity, it first splices them into a continuous and complete audio stream in chronological order, and then performs overall decoding and transcription on the continuous and complete audio stream to ensure that the first language transcribed text string of the same speaker remains coherent in terms of grammatical structure and semantic logic.

9. The method according to claim 1, characterized in that, In step 5, when the remote server detects that the overlap rate of the first language transcribed text strings corresponding to two or more identity identifiers on the physical timeline reaches a preset threshold, the remote server calls a semantic classifier to perform pragmatic function classification on the overlapping transcribed texts and determines whether there is a logical sequential relationship based on the classification results. If there is, the display order of the overlapping transcribed texts is logically rearranged when generating the structured meeting record, and the original physical timestamp information corresponding to each transcribed text is retained in the rearranged record. If there is no relationship or the confidence level is lower than a preset threshold, the transcribed texts of each speaker are presented simultaneously at the same timeline position in a parallel display manner.

10. A multilingual conference recording and interpretation system based on dual-channel audio comparison, characterized in that, include: Remote server; A conference terminal, deployed at the conference site, includes at least a first voice acquisition module, a first voice playback module, and a first communication module; The first voice acquisition module is used to acquire a second audio signal containing the voice content of the participants in the meeting room. The second audio signal does not carry the identity information of the participants. The first communication module is used to send the second audio signal to the remote server in the form of a second audio data stream, and to receive the second language voice sent by the remote server and play it through the first voice playback module. Multiple smart terminals, with each participant corresponding to one smart terminal, each smart terminal including at least a second voice acquisition module, a second communication module, and an identity association encoding module; the second voice acquisition module is used to acquire a first audio signal corresponding to the participant's voice content; the identity association encoding module is used to associate and encode the first audio signal with the participant's unique identity identifier (ID) to generate a first audio data stream carrying identity identifier information; The second communication module is used to send the first audio data stream to the remote server through the first transmission channel, and to receive and display the transcribed or translated text sent by the remote server. The remote server includes: A communication interface module is used to receive a second audio data stream from the conference terminal and a first audio data stream from each of the smart terminals; A timeline synchronization module is used to synchronize and align the first audio data stream and the second audio data stream on the timeline. The identity comparison and attribution module is used to determine the speaker's identity corresponding to the voice content in the corresponding time interval of the second audio data stream based on the identity identification information carried in the first audio data stream, and to assign the voice content in the second audio data stream to the corresponding identity identification according to the determined speaker's identity. The speech recognition and transcription module is used to perform speech recognition and transcription on the speech content belonging to each identity identifier, and generate the first language transcribed text string corresponding to each speaker. The speech translation and synthesis module is used to translate the first language transcribed text string into the target second language, generate a translated text string, and synthesize the translated text string into second language speech, which is then sent to the first speech playback module of the conference terminal for playback through the communication interface module; The meeting record generation module is used to organize the first language transcribed text strings and the translated text strings corresponding to each speaker at least according to the speaking timeline, generate a structured meeting record containing speaker identity information, original speech content and corresponding translation content, and push the structured meeting record to each smart terminal for display through the communication interface module.

Images