Real-time bidirectional simultaneous interpretation methods, devices, and computer equipment based on multi-source audio and multi-model collaboration

Abstract

This invention discloses a real-time bidirectional simultaneous interpretation method, apparatus, and computer equipment based on multi-source audio and multi-model collaboration, relating to the field of simultaneous interpretation technology. It fundamentally solves the problems of "infinite bubble growth" and "back-off jitter" in streaming ASR by using a sentence assembler with three-state atomic transfers of "to be processed - sealed - finalized," combined with multiple sealed predicates, the longest common prefix stabilization algorithm, and an overlap stripping algorithm. Furthermore, it ensures that subtitles always close properly through an asynchronous finalized "timeout degradation" path. A single decision function simultaneously calculates four fields: source language, target language, speaker number, and display panel affiliation, eliminating the problem of inconsistency between subtitle display position and the target language. In dual-source mode, physical channels are used as anchor points to prevent panel jitter in bilingual scenarios.

Description

Technical Field

[0001] This invention relates to the field of simultaneous interpretation technology, specifically a real-time bidirectional simultaneous interpretation method, apparatus, and computer equipment based on multi-source audio and multi-model collaboration. Background Technology

[0002] In the context of economic globalization, the demand for real-time cross-language translation is becoming increasingly frequent. Currently, Automatic Speech Recognition (ASR), Neural Machine Translation (NMT), and Text-to-Speech (TTS) technologies are widely used in cloud services and desktop clients. However, integrating these three technologies into desktop software for multilingual, two-way simultaneous interpretation scenarios presents several technical challenges:

[0003] First, traditional streaming ASR directly appends each arriving partial text to the caption bubble. Because Whisper-type models repeatedly send partial text multiple times within the same silent segment, and exhibit a rollback phenomenon of "previous guesses being overturned the next," the captions seen by the end user often exhibit the following problems: ① continuously increasing length without closing; ② already displayed words being replaced; ③ "false periods" appearing at the end of sentences, causing premature punctuation that is then canceled. Existing solutions often employ a delay strategy of "displaying only at the end of the complete segment" to avoid rollback, but this contradicts the low-latency requirements of simultaneous interpretation. When ASR forcibly truncates at the maximum speech duration, the model tends to "phantom" a period at the truncation point; the next segment then re-transcribes several words from the truncated point, causing caption overlap.

[0004] Second, in two-way simultaneous interpretation scenarios, each ASR output requires the computer device to make at least the following judgments: whether the actual source language is the user-configured source language or the target language (bilingual users often switch languages); whether translation needs to be triggered (when "source == target", should it still be polished or directly passed through); which panel (left / right) in the user interface should it be displayed on; and which target language should be used for translation. Existing solutions often distribute these judgments in multiple places: the acoustic language is returned by the ASR adapter, the text language is estimated by the front-end calling the browser interface, and the panel affiliation is determined by the routing layer based on the physical channel. This results in the same sentence being judged as different languages ​​in different components, leading to problems such as subtitles being displayed on the wrong side or being incorrectly "out-of-scope" dropped.

[0005] Third, when desktop applications call large local models, they often load them in the same process as the main process, which presents the following problems: When multiple AI model frameworks (PyTorch, TensorFlow, ONNX, etc.) coexist in the same Python process, global states such as CUDA context, memory allocator, and log configuration are prone to conflict, leading to memory leaks and initialization failures; when users switch models at runtime, if the old adapter is still referenced by an unfinished session, a use-after-free error will occur; multiple concurrent "overload" requests may cause the state machine to become disordered, resulting in orphan processes or memory leaks; when a model crashes during loading, the main process cannot receive an accurate error reason, making diagnosis difficult.

[0006] Fourth, existing desktop AI software installation methods often package all dependencies into the installation package at once, resulting in installation packages that are often several gigabytes in size. Furthermore, users without GPUs are forced to download local model components even if they only want to use the cloud mode. While some solutions support on-demand installation, they lack an automatic alignment mechanism for scenarios where the versions of components already installed by the user after upgrading the software are inconsistent with the new version's declaration, requiring users to manually reinstall or maintain dependencies.

[0007] Fifth, users want to send the translation results back to the conferencing software so that the other party can hear them. The common practice is for users to manually bridge the connection using a third-party audio routing tool, which is cumbersome. If the automatic reading is played directly through the local speaker, it will be picked up again by the microphone on the same machine and enter the ASR pipeline, creating an echo and causing a dead loop of "machine-to-machine dialogue". The audio output device ID in the browser / Electron may change every time the device is re-enumerated, and simple "binding by ID" is prone to failure.

[0008] Sixth, users want to "translate only the audio from a specific conferencing software" rather than mixing the entire system, but the traditional WASAPI loopback requires a hardware-level "stereo mixing" virtual device and cannot filter by process; in addition, if users modify engine bindings, TTS timbre, prompts, and other configurations during long meetings, the traditional approach is to directly terminate the current WebSocket connection, causing the meeting to be interrupted.

[0009] Chinese invention patent CN119150891B discloses a bidirectional real-time translation method based on terminal information interaction. This method combines simultaneous interpretation services with third-party applications by creating a virtual microphone and configuring its audio link with the currently active application. The virtual microphone in this solution is implemented using Android platform AudioRecord and AudioTrack with a buffer, and the scenario type is divided into "unidirectional link / bidirectional link." Its improvement lies in the simple process splitting of "creating an independent first process to run the simultaneous interpretation service and an independent second process to run the text translation." However, this solution does not address methods for stabilizing streaming subtitles, single-decision-point routing that integrates acoustics and text language, subprocess isolation and hot reloading of multi-role AI inference engines, installation management with "on-demand installation + upgrade idempotent self-alignment," or echo suppression during automatic reading.

[0010] Chinese invention patent CN113505612B discloses a method for real-time translation of multi-person dialogue speech. Based on a multi-channel filter bank speech separation model, it separates a single mixed speech stream into N sub-audio segments, then performs ASR (Automatic Speech Recognition) + translation + TTS (Text-to-Speech) for each segment. This solution focuses on "separating multiple voices from a single mixed stream," which differs from the bidirectional simultaneous interpretation scenario addressed by this invention, which features "natural separation of multi-source audio (microphone, system playback, single-process audio)." Furthermore, it does not address the core issues this invention aims to solve, such as streaming subtitle stabilization, engine hot reloading, and on-demand installation.

[0011] Chinese invention patent application CN116312538A discloses an AI translation system for online business negotiation, which is a system-level solution consisting of sequentially connected modules such as speech sampling, speech recognition, text translation, video correction, and audio / video streaming. This solution is a module-level sequential solution and does not involve any streaming subtitle assembly mechanism, engine runtime management, or panel routing issues for bidirectional simultaneous interpretation.

[0012] Chinese invention patent application CN108769891A discloses an audio transmission method and a mobile translation device. This method utilizes the two audio channels (left / right) of the mobile translation device to output translation results in two languages ​​respectively, enabling both parties to hear the desired language. This solution focuses on the correspondence between hardware dual channels and the translated data, which is completely different from the desktop software-level virtual microphone routing involved in this invention.

[0013] Chinese invention patent application CN120032658A discloses a multi-channel speech conversion and synchronous transmission system for simultaneous interpretation. This system combines noise suppression using spectral subtraction and convolutional neural networks, blind source separation using independent component analysis, TDOA delay estimation, and a Seq2Seq end-to-end translation model. This approach focuses on optimizing the underlying signal processing algorithms, which differs from the engineering architecture innovations of this invention (sentence three-state lifecycle, engine subprocess isolation and hot reloading, on-demand pipeline installation, etc.) but are not mutually exclusive.

[0014] In summary, no existing technology can simultaneously solve the following problems within the same system: (a) the "infinite bubble growth" and "back-off jitter" issues in streaming subtitles; (b) the "language determination and panel ownership consistency" issues in two-way simultaneous interpretation; (c) the process isolation, hot reload, and session protection issues of multi-role AI inference engines on the desktop; (d) the "minimum installation + on-demand expansion + upgrade self-alignment" issues that are user-friendly for both GPU-enabled and GPU-less users; and (e) the "virtual microphone routing + ID self-healing + echo suppression" combination issues when automatically reading to third-party conferencing software. Summary of the Invention

[0015] To address the aforementioned technical problems, this invention proposes a real-time bidirectional simultaneous interpretation method, apparatus, and computer device based on multi-source audio and multi-model collaboration. The aim is to solve the technical problems in existing technologies that cannot simultaneously meet multiple requirements within the same desktop real-time translation software, such as low latency, stable and jitter-free subtitles, multi-language robustness, hot-switching of runtime models, user-friendly distribution for users without GPUs, automatic alignment during installation and upgrades, one-click re-import of automatic readings to third-party conferencing software, and accurate capture of specified application audio based on the operating system process audio loopback API.

[0016] This invention protects a real-time bidirectional simultaneous interpretation method based on multi-source audio and multi-model collaboration. This method, applied in a computer device, includes the following steps:

[0017] ◆The computer device captures audio frames from an audio source selected by the user and encodes them in standard Pulse Code Modulation (PCM) format; the audio source includes at least one of the following: the microphone device of the computer device, the system playback loopback device of the computer device, and a specified process audio capture channel based on the operating system process audio loopback application programming interface.

[0018] In other words, the software is not limited to recording from a microphone; it supports at least three sources:

[0019] ◇Microphone equipment: Captures the user's voice.

[0020] ◇System playback loopback device: Captures all sounds played throughout the entire operating system (such as the other party's voice in conferencing software, and background noise in video).

[0021] ◇Specified process audio capture channel based on the operating system process audio loopback API: Allows users to precisely select to capture only the sound played by a specific application (such as Zoom or Teams), thereby filtering out irrelevant noise such as system notifications and music players.

[0022] The software continuously and in real time reads small blocks of raw audio data, or "audio frames," from the selected "audio source" through the operating system's audio API (such as Windows' WASAPI or CoreAudio).

[0023] ◆The computer device calls the Automatic Speech Recognition (ASR) adapter to generate text from the audio frames, and calls a single decision function for each frame of generated text. The single decision function integrates the acoustic language recognition result and the text language recognition result, and outputs an immutable decision object containing four fields: source language, target language, speaker number, and display panel affiliation. The acoustic language recognition result is usually provided by the ASR model when recognizing speech, judging "what language this sound sounds like", but there may be accents and noise interference. The text language recognition result performs language detection on the plain text generated by ASR, judging "what language this text looks like", as a supplement or correction to the acoustic result.

[0024] The computer device (main process) sends the normalized PCM audio frames captured in the previous step to the Automatic Speech Recognition (ASR) adapter. This adapter (potentially calling a local Whisper model or a cloud API) outputs text in real time. Because it's streaming, the output isn't a single, complete sentence, but rather continuous, segmented blocks of text. For each frame of text produced, whether it's "partial" or "final," this function is triggered immediately as soon as ASR outputs, ensuring real-time decision-making.

[0025] Single decision means that there is only one entry function in the system responsible for making all routing decisions. This fundamentally eliminates the problems of data inconsistency, panel confusion, and language incompatibility caused by multiple decentralized decisions in acoustic models, language models, and front-end logic in traditional solutions. "Immorality" means that once an object is created at the beginning of a sentence, its content cannot be changed throughout the entire lifecycle of the sentence. This ensures that the identity of the sentence (who spoke it, where it came from, where it goes, and where it is displayed) is absolutely consistent from the first word to the last word, completely solving the persistent problem of panel and translation direction jitter.

[0026] ◆The computer device maintains a sentence assembler in memory for each audio source and assigns a globally monotonically increasing sentence identifier to each sentence; the sentence assembler atomically transfers each sentence between the three states of "pending processing - sealed - finalized" based on the decision object and the multiple sealed predicate.

[0027] The system maintains an independent assembly "pipeline" for each audio source, such as microphones and system playback. This isolates data from different speakers and avoids crosstalk. The sentence identifier is a unique, ordered "ID number" for each sentence. Its globally monotonically increasing characteristic is crucial; it's not only used for sorting but also a key anchor point to prevent asynchronous processing errors. Subsequent "preview translations" use it for verification, preventing the translation results of the previous sentence from being applied to the new sentence.

[0028] The "three-state atomic transition from 'pending processing' to 'sealed' to 'final state'" is the core state machine model of this invention, defining the lifetime of each sentence in memory:

[0029] ◇Pending Processing: The sentence is being transcribed in real time by ASR, and the text content is still being dynamically appended and corrected. In this state, the assembler uses the "longest common prefix algorithm" to absorb backtracking jitter.

[0030] ◇Sealed: When a sentence meets the "sealed" condition, the text is frozen and will not be modified. Sealing is triggered by multiple sealing predicates, including at least five parallel conditions such as detecting a period, silent timeout, and reaching the duration limit, ensuring that the sentence is promptly "sealed" under any circumstances to prevent unlimited growth. After sealing, its subtitles and preview translation will be immediately displayed to the user with low latency.

[0031] ◇Final state: The sentence has undergone final, high-quality processing (such as LLM correction and translation) and has formed a definite, archiveable version.

[0032] "Atomic transfer" means that state switching is an indivisible, thread-safe operation that ensures that a sentence is in only one definite state at any given time under any concurrent conditions, thus avoiding state chaos.

[0033] ◆After each sentence enters the "sealed" state, the computer device asynchronously and independently calls the large language model twice. The first call outputs the corrected original text in the same language with ASR error correction prompts, and the second call outputs the final target language translation with translation prompts. Each call has its own independent timeout and degradation path. The two calls are executed serially, and after both return, an atomic final state update fills the front-end subtitle with the corrected original text in the same language and the final target language translation.

[0034] After the sentence is encapsulated, the system does not wait for the slow processing result of the LLM, but immediately returns to the foreground for display (ensuring low latency). At the same time, a high-quality translation / correction task is asynchronously initiated in the background. This translation / correction task includes two independent calls to the large language model.

[0035] The first call is initiated to the large language model adapter with an ASR error correction prompt. The input is the archived original text and the context of the most recent terminalized original texts in the same language. The system prompt is limited to "only output the corrected original text in the same language, without any translation". The first call has a first timeout threshold. If the timeout occurs, the uncorrected archived original text will be used.

[0036] The second call is initiated to the large language model adapter with translation prompts. The second call is initiated serially after the first call returns. The input is the corrected original text of the first call. If the first call times out or fails, the input is the archived original text, and several recently finalized source-translation pairs are appended as paragraph context. The second call has a second timeout threshold. If the timeout occurs, the preview translation generated in the pending state stage is used.

[0037] The first call and the second call share the same sentence identifier, and each is assigned an independent call category identifier that falls into the structured log. If the sentence is backtracked and merged while either of the two calls is still in progress, and the decision is that it should be merged with a newly archived sentence, the computer device cancels the asynchronous tasks corresponding to the two calls and releases the corresponding resources.

[0038] After both calls return, the computer device performs an atomic sentence state field update, writes the corrected original text field and the final translated text field together, and promotes the state from sealed atomic to final state.

[0039] The two independent call mechanism brings multi-dimensional and systematic benefits:

[0040] ◇A qualitative improvement in user experience: Users can see high-quality, corrected original text earlier and faster, which is especially important when proper nouns, numbers, and personal names are misrecognized by ASR; users can confirm that what they say has been correctly recognized without waiting for the entire translation process to end, and receive immediate and certain feedback.

[0041] ◇Enhanced system availability and robustness: Failure of one call will not affect another. For example, if the translation service (Last Call) is temporarily unavailable or the network times out, the system can still use the successful original text correction result (First Call) and the previously generated preview translation as a fallback, ensuring that the sentence is "complete". The availability of the entire system changes from "single point of success" to "two points of success", significantly improving reliability.

[0042] ◇Controllable overall perceived latency: After the sentence is sealed, the preview of the original text and the preview of the translation are displayed in real time, thus ensuring low latency perception; the two finalization calls in the background are executed serially, and their total time does not affect the preview that the user has already seen. Only after completion are the original text and the final translation with higher quality replaced at once. Therefore, the finalization time is a quality improvement that the user does not perceive in the background.

[0043] ◇ Improved system observability and maintainability: Each call has an independent identifier and log, which enables operations and maintenance personnel to accurately track and locate whether the performance bottleneck or error occurred in the "correction" or "translation" stage, facilitating problem diagnosis and system optimization.

[0044] ◇Smarter Resource Utilization and Response: Supports canceling two tasks separately. In the "backtracking merge" scenario, if the translation of the previous sentence (Last Call) is slow, but the original text correction (First Call) has returned, the system can retrieve the translation task that was consumed, avoiding unnecessary calculations and responding more nimbly.

[0045] Preferably, the computer device further provides an on-demand knowledge base retrieval method, including:

[0046] The computer device calls the query keyword extraction prompt words and initiates a lightweight call to the large language model adapter to extract the search keywords, proper nouns, and technical terms from the most recent context of the current session into a query string;

[0047] The computer device sends the query string into the embedding adapter to obtain the query vector, and calculates the similarity of all question and answer item vectors in the currently activated knowledge base using cosine similarity. The top-K items with similarity higher than a preset threshold are taken as the recall results.

[0048] The computer device returns the recall results to the front end in the form of a result set, which is used by the user for comparison and reference in translation proofreading or conference Q&A scenarios, without automatically inserting the translation word by word;

[0049] The knowledge base retrieval is initiated on demand by the user in the knowledge base or conference Q&A interface, and is not automatically triggered in the preview or finalization call chain of real-time sentence segmentation translation, thus not increasing the end-to-end latency of real-time translation;

[0050] The recall results are returned to the front end via an interface and visualized by the user interface in the form of pop-ups or drawers, making it easy for users to compare the original text, translation, and reference materials.

[0051] The above operations are based on local knowledge base retrieval (RAG). By providing the large language model with accurate relevant knowledge fragments (such as product manual clauses, legal provisions, and medical terminology explanations), the conference Q&A and translation proofreading results can conform to the context and fixed expressions of specific fields, reducing misunderstandings caused by a lack of background knowledge. The retrieval is invoked on demand, only executed when the user actively initiates a retrieval, and does not enter the real-time sentence segmentation and translation process, thus not affecting the low latency of real-time translation.

[0052] Prioritized, further execution is performed before each large language model translation call:

[0053] The computer device maintains a terminology list, and each entry in the terminology list includes source text, target text, entry type, group, and activation flag fields; the entry type is one of three: "must be translated as", "must not be translated as", or "preserve original text";

[0054] The computer device applies each entry simultaneously in both directions: "source language → target language" and "target language → source language", eliminating the need for the user to repeatedly enter the same term in both directions.

[0055] The computer device allows users to divide items into several groups according to "meeting topics" and select the active group set for the meeting before the start of each meeting;

[0056] The computer device will activate all entries in the set, group them by type, and assemble them into terminology instruction blocks, which will then be injected into the system prompt words called by the translation big language model.

[0057] After the large language model returns the translation, the computer device scans each activated term entry in the translation to perform violation detection. A violation must be translated as such only if the source text appears in the original text but the target text does not. A violation will not be translated as such only if both the source text and the target text appear in the original text. A violation will be retained only if the source text appears in the original text but the translation does not directly copy it. Detected violations are recorded in a structured format and their locations are marked with underlines / tooltips in the front-end caption bubble.

[0058] The computer equipment does not perform string replacement or post-processing rewriting on the violations in order to maintain the naturalness of the translated syntax.

[0059] The above operations influence the model through "soft hint injection" rather than "hard post-processing replacement," allowing the model to understand terminological constraints while still freely organizing syntax and changing word forms (such as singular / plural and tense), thus outputting translations that are both grammatically correct and fluent. "Two-way rules" halve the user's configuration workload and intelligently cover both parties in the conversation; the "group activation" mechanism allows users to quickly load corresponding professional terminology sets for different meetings, much like switching "skill packs"; front-end visual markers (underline / tooltips) make terminology violations immediately apparent, allowing users to quickly locate problems without comparing sentence by sentence, significantly improving proofreading and QA efficiency.

[0060] By forcefully injecting user-defined deterministic rules (must / never / reserved) into the LLM inference process through system prompts, the "black box" model output becomes controllable and predictable to a certain extent. The verification and labeling mechanism is equivalent to providing a "terminology compliance audit report" for each translation, making the model's "performance" transparent. This enhances users' trust in AI translation results, especially in fields with extremely high accuracy requirements such as law and medicine.

[0061] ◆The computer device automatically calls the speech synthesis adapter to synthesize audio for each sentence that enters the "final state" according to the user's preset three reading modes: "off / local speaker / virtual microphone". The audio is then routed to the output device corresponding to the mode through the device binding method of the browser audio interface.

[0062] By utilizing a user-space virtual audio driver (such as VB-AudioVirtualCable) to create a "virtual microphone" visible in the system's audio device list, and using the browser's standard API setSinkId(), the audio output target can be dynamically switched to this virtual device at runtime. This achieves pure software-based, configuration-free, cross-application audio redirection. Users do not need to manually adjust settings in system sound settings or third-party virtual mixing software; it can be done with a single click, greatly reducing the barrier to entry and improving reliability.

[0063] Furthermore, the multiple sealing predicate specifically includes at least two of the following predicates:

[0064] Predicate 1: A sentence-final punctuation mark appears at the end of the sentence;

[0065] Predicate 2: The difference between the last update time and the current time is ≥2000 milliseconds;

[0066] Predicate 3: The sentence duration is ≥6 seconds and the interval between the most recent speech activity is ≥300 milliseconds;

[0067] Predicate 4: Sentence duration ≥ 10 seconds;

[0068] Predicate 5: The sentence is generated by forced truncation and has an idle time of more than 8 seconds;

[0069] Meanwhile, the computer device schedules the wall clock beat in memory every 200 milliseconds, and uses the difference between the wall clock time and the last update time as a fallback estimate of the voice activity interval.

[0070] Ideally, in streaming ASR, the ASR engine pushes text (partial or final) whenever the speaker has new speech. We can estimate the "speech activity interval" by subtracting the "time of receiving the last ASR text" from the "time of receiving the current ASR text". If this interval becomes longer, it indicates that the speaker may be thinking or has finished speaking.

[0071] A special case exists in real-world scenarios: ASR (Automatic Speech Retrieval) may not send any data at all during silent periods. For example, a speaker might pause for 3 seconds without any ASR output. In this case, the system receives no timestamp updates and therefore cannot calculate the actual silence duration by comparing the "ASR push time." The sealing predicates that rely on this (especially the "interval between the most recent speech activity" condition in predicate two "silent sealing" and predicate three "soft upper limit sealing") will be "paralyzed" due to lack of data and cannot be triggered, causing the sentence to be delayed in sealing.

[0072] The system starts an independent timer with a period of 200 milliseconds ("wall clock beat"). This timer is independent of whether the ASR has data; it runs steadily like the ticking of a clock. At each "beat," the system checks the current real time on the wall clock and then subtracts the sentence's "last ASR update time" from the "current wall clock time," using this difference as an estimate of the "voice activity interval." Even if the ASR is "dead" during a silent period and does not push any data, this independent metronome can continuously and periodically "sense" the passage of time. Once the calculated silence duration reaches a threshold (such as 1000 milliseconds for predicate two), the corresponding sealing condition is met, and the sentence can be sealed normally.

[0073] Furthermore, the sentence assembler further executes in the "pending" state:

[0074] For each incoming partial text, the longest common prefix is ​​calculated between it and the previous partial text. Only the part with the common prefix is ​​used as the stable text to drive subsequent judgments. For example, if the previous partial text was "I am going to the," and the new partial text received this time is "I am going to the store," then the LCP is "I am going to the." If the next new text becomes "I am going to the market," the LCP with the old text "I am going to the store" is still "I am going to the." Only this LCP is considered "stable text," used to update the display and trigger subsequent judgments. This is equivalent to adding a "stabilization filter" to the ASR's guessing. No matter how the ASR changes the following words, the confirmed prefix will not be backed up, thus completely eliminating the "word replacement" jitter in the subtitles and ensuring visual stability.

[0075] When the previous audio segment is forcibly truncated and the newly arrived text in this round has a maximum of six normalized word-level overlaps with the end of the confirmed buffer (excluding capitalization and punctuation), the overlapping headers in the current round's text are stripped off by word-level alignment before being written into the buffer. For example, if the end of the confirmed buffer is "...theprojectdeadline", after forced truncation, the newly arrived text is "deadlineistomorrow". After normalization and word-level alignment, the header word "deadline" is found to overlap. Therefore, the assembler removes this overlapping header word, appending only "istomorrow" to the buffer, ultimately resulting in "...theprojectdeadlineistomorrow". This perfectly avoids the repetition of the word "deadline", thus fixing the semantic boundary uncleanliness problem caused by the ASR model's behavioral defects and ensuring sentence coherence.

[0076] A short-delay translation of the stabilized text is triggered in parallel by a preview translation coroutine. When the preview result is returned, it is compared with the sentence identifier recorded at the time of the request and the currently active sentence identifier of the assembler. If they do not match, the result is discarded. To achieve low latency, preview translation needs to be started during the sentence "pending processing" stage. However, when the asynchronous translation returns, the original sentence may have been sealed, and a new sentence may have started. If the returned translation result is directly used for the currently active sentence, it will lead to "misapplication," that is, the translation of the previous sentence will contaminate the subtitles of the new sentence. After calculating the "stabilized text," an asynchronous "preview translation" task is immediately started, and the sentence identifier of the sentence at the time of startup (i.e., the globally monotonically increasing ID) is recorded and bound to the translation request. When the preview translation result is returned, it is not applied directly, but a key check is performed first: comparing whether the "sentence identifier bound to the returned result" is consistent with the "sentence identifier currently active in the assembler." If they are consistent, it means that the translation result is "on the right track," the currently active sentence is still the original sentence, the result is valid, and it can be used to update the preview subtitles; if they are inconsistent, it means that the original sentence has been sealed and the active sentence is now a new sentence. The translation result is now "outdated" and is discarded to avoid contaminating the content of new sentences. Through a simple "generational verification" mechanism, this invention perfectly solves the common data race and misalignment problems in asynchronous parallel processing at extremely low cost, ensuring the accuracy of real-time preview subtitles.

[0077] Furthermore, the single decision function further includes:

[0078] The language fusion parsing submodule is called and returns the final source language according to the following priority: if the text language recognition result falls within the set of source language basic code and target language basic code, the text language recognition result is adopted; otherwise, the acoustic recognition result is adopted. When the acoustic recognition also fails, the text language recognition result is adopted again. If all the above methods fail, the language is downgraded to the source language configured by the user.

[0079] The decision object is saved as a route snapshot of the sentence by the sentence assembler in an immutable form. All subsequent events of the sentence are forcibly read from this snapshot to fill the display panel's home, source language, and target language fields.

[0080] When the system is in dual-source mode, the microphone path corresponds to speaker number 0, the display panel is assigned to the right, the source language is the real-time recognition result, and the target language is the user-configured source language; the system playback path corresponds to speaker number 1, the display panel is assigned to the left, the source language is the real-time recognition result, and the target language is the user-configured target language.

[0081] When the system is in single-source mode and the detected source language base code is equal to the configured target language base code, the speaker number is 1, the display panel belongs to the right, and the target is the user-configured source language; when the detected source language base code is equal to the configured source language base code or is not in the set of both, the speaker number is 0, the display panel belongs to the left, and the target is the user-configured target language.

[0082] Furthermore, the engines requiring local inference in the large language model adapter, automatic speech recognition adapter, and speech synthesis adapter are hosted in independent subprocesses by the computer device. This is the most important isolation design. Each engine requiring local inference (such as a large language model or a speech recognition model) runs in a completely independent Python subprocess, rather than within the main process. This brings three core advantages: stability—a crash in one engine (such as memory overflow or CUDA error) will not cause the entire main program to crash; the engine management module can detect the crash and restart it; resource isolation—avoiding multiple AI frameworks (PyTorch, TensorFlow) competing for global resources such as CUDA context and memory allocator in the main process, fundamentally solving the problems of "memory leaks and initialization conflicts"; and independence—each engine can be started, stopped, and restarted independently without interference.

[0083] The main process and the child process communicate via a binary frame protocol consisting of 4 bytes of big-endian length + 1 byte of type + N bytes of payload. The child process and the main process need to exchange data (such as audio, text, and commands) through "inter-process communication". This invention does not use conventional HTTP or fragmented JSON lines, but instead designs a custom, efficient binary protocol.

[0084] ◇4-byte big-endian length: Defines the length of the entire subsequent data frame so that the receiver can read it accurately.

[0085] ◇1-byte type: The type of this frame is identified by a one-byte enumeration value (such as 0x01=AUDIO, 0x02=CONFIG, 0x03=RESULT, 0x0B=SHUTD-OWN) to achieve fast routing.

[0086] ◇N-byte payload: The actual data content.

[0087] This protocol is efficient, unambiguous, and easy to parse. It avoids the complex encoding / decoding problems of text protocols based on newline characters when transmitting binary data (such as audio), and also avoids the "incomplete message" trap caused by improper buffering or packet splitting, making communication extremely reliable.

[0088] When the engine switches during runtime, the computer device performs:

[0089] For each load request, a signature is calculated. If the new signature equals the existing effective signature and the role is currently READY or STARTING, the process returns immediately without further action. This is a performance optimization that avoids unnecessary restarts caused by users repeatedly clicking the same configuration.

[0090] Each time the signature changes, the reload sequence number is incremented by 1 and recorded as the expected sequence number for this reload. After each asynchronous wait, the current expected sequence number is compared with the latest reload sequence number. If they do not match, the current process is stopped and any started child processes are cleaned up. This is an ingenious concurrency control design that ensures only the last load request succeeds, achieving the effect of "idempotent overriding." The user always gets the model they last selected, preventing "race conditions" and resource leaks caused by slow operations.

[0091] Before switching, the session epoch module is invoked to mark the current epoch as deprecated and immediately create a new epoch, giving a maximum window of 10 seconds for active sessions to end naturally. Upon receiving a switch request, the current epoch is immediately marked as deprecated, and a new epoch is created simultaneously. Afterward, new translation sessions begin based on the new epoch (new engine), while old, ongoing sessions continue running under the old epoch (old engine) unaffected. The system asynchronously waits for the reference count of the old epoch to drop to 0 (i.e., all old sessions end naturally) and sets a maximum wait time (e.g., 10 seconds). If all old sessions end within the 10-second window, the switch is successful. If sessions still exist after the timeout (e.g., the user keeps talking), the reload process is forcibly resumed. A few stubborn old sessions may fail, but this ensures the system does not wait indefinitely. This achieves elegant, user-insensitive, zero-downtime hot updates. The translation service is uninterrupted throughout, and users do not need to restart the entire meeting because of adjusting a TTS tone or prompt, which greatly enhances the product's professionalism and usability.

[0092] Furthermore, the step of automatically invoking the speech synthesis adapter for each sentence that transitions to the "final state" further includes:

[0093] In the front-end responsive state system, subscribe to the set of finalized sentences of each display panel. When the number of sets changes, skip the panel with the mode "closed". Use the recorded recently read sentence identifier to filter out new sentences that have not yet been read. Determine whether to extract the original text or the translation based on the target text selection of the panel. Enqueue each item to be read into the serial playback queue.

[0094] The playback coroutine sequentially retrieves the items to be read from the queue and submits a synthesis request to the main process. After the main process returns the binary audio, the front end creates an audio element. If the setting is "local speaker", the playback method of the audio element is called to play it on the default output device. If the setting is "virtual microphone", setSinkId is first called to bind the output of the audio element to the virtual microphone device ID, and the output target of this audio element is redirected to the virtual microphone device. After playback, the sound will flow into the virtual device and be picked up by the third-party conferencing software.

[0095] Before requesting playback, the stored virtual microphone device ID is used to perform self-healing by calling the parsing submodule: if the device ID still exists in the browser enumeration results, it is used directly; otherwise, it performs case-insensitive matching on the tags of all output devices according to a predefined keyword set. If a match is found, the old ID in the database is replaced with the newly discovered device ID. The predefined keyword set is a list of a series of characteristic strings predefined in the code, used to identify virtual audio devices. This mechanism achieves seamless self-recovery of the function. Users do not need to know that the device ID has changed, nor do they need to manually reselect the device; the software automatically "finds" the correct device, ensuring the extremely high reliability of the "one-click reload" function.

[0096] In "Local Speaker" mode, the played TTS audio may be picked up by the microphone of the same computer and sent back to ASR. To address this, this invention relies on browser echo cancellation enabled by the front-end audio acquisition to suppress this feedback and provides an echo risk warning when the user switches to the local speaker mode. In "Virtual Microphone" mode, the audio is output through an independent virtual device and will not be physically picked up by the local microphone, thus naturally avoiding this problem.

[0097] Furthermore, the computer device only installs the core dependency packages upon initial startup; when the user enables the local inference engine, a standard installation pipeline consisting of at least ten stages is executed, which in turn include: hardware requirements and external dependency checks, CUDA version PyTorch verification, optional pre_pip hook, overlay dependency lock file-based installation, optional post_install hook, model weight dual-source sequential rollback download (HuggingFace first, ModelScope rollback), in-process real import verification, engine-specific compliance checks, optional post_verify hook, and atomic writing to the installation manifest; the installation result is recorded in the install.json installation manifest.

[0098] The computer device performs the following each time it starts up:

[0099] Read the basic dependency list, check for the existence of the corresponding release information directory in the base site directory for each dependency, and if it does not exist, call pip to install it with "--upgrade-strategyonly-if-needed";

[0100] For each installed overlay engine, compare the current dependency directory declaration with the dependency snapshot in the installation manifest, install only the newly added difference set, and immediately overwrite the installation manifest with the new snapshot.

[0101] This invention provides two complementary mechanisms:

[0102] ◇Initial installation mechanism: When a user first enables a local AI engine, a complex, ten-stage installation process is executed.

[0103] ◇Startup maintenance mechanism: Each time the software starts, a lightweight and fast "health check and synchronization" process is executed to ensure that the installed environment is consistent with the software requirements.

[0104] This invention constructs a complete lifecycle management system from "initial installation" to "continuous maintenance." The "ten-stage pipeline" solves the reliability problem of deploying in complex AI environments through fine-grained decomposition and status recording; while the "startup alignment" mechanism solves the environment synchronization problem after software upgrades through intelligent dependency detection and a minimal change strategy. The combination of the two achieves the core goals of minimizing installation packages, ensuring reliable deployment, and automating upgrades.

[0105] Preferably, there are strong constraints on the lock file format: For each overlay deployment mode, the local inference engine explicitly declares the lock file path pointed to by the 'overlay_lockfile' field in the engine metadata, and each non-empty, non-comment line in the lock file must match either the strict equals sign form of "package name == version number" (e.g., transformers == 4.36.0) or the remote archive form of "package name @ remote archive URL" (usually containing a hash value, such as package @ https: / / ... / package-1.2.3.tar.gz#sha256=...);

[0106] Preferably, critical dependency pollution is prohibited: the lock file must not contain dependencies starting with torch, torchaudio, or nvidia-*; during startup, the audit function iterates through the lock file pointed to by the 'overlay_lockfile' field and performs the above-mentioned checks and prohibitions. Those that fail the check are recorded in the unified log library at the warning level for developers to fix.

[0107] Preferred, automated auditing and feedback: The above lockfile locking mode ensures that the overlay dependencies of the same software version distributed to different users have the exact same version combination, and prevents torch family dependencies from being repeatedly loaded into the overlay, which could cause ABI conflicts or disk usage bloat.

[0108] The aforementioned "file locking + auditing" mechanism solves a series of deep-seated engineering problems affecting stability and user experience in the distribution, deployment, and maintenance of desktop AI applications. Its technical effects are multi-dimensional and significant:

[0109] ◇Ensuring absolute reproducibility of the deployment environment: By mandating that all Overlay dependencies be locked to exact versions or hashed archives, the risk of "dependency drift" caused by PyPI index updates or the release of new versions of upstream packages is completely eliminated. For the same version of software distributed to any user globally, the combination of Python package versions in all Overlay directories will be completely consistent. This means that the behavior that the software passes testing on the developer's machine can be 100% reproduced in the user's environment, fundamentally eliminating the classic "it works on my machine" problem and greatly improving the reliability of software distribution.

[0110] ◇ It eliminates dependency conflicts and resource waste, improving the system stability and resource utilization efficiency of complex AI applications (running multiple local engines simultaneously).

[0111] To prevent ABI conflicts: Duplicate declarations of core libraries such as torch are explicitly prohibited for overlays, ensuring the uniqueness of low-level binary interfaces like the CUDA runtime and cuDNN within a process. Multiple overlay engines installing different versions of PyTorch, or even different CUDA versions, can easily lead to unpredictable crashes or computational errors. This rule eliminates such conflicts at the source.

[0112] Saves disk space and memory: Avoids repeatedly downloading and storing PyTorch libraries of several gigabytes in base_venv and multiple overlay directories, significantly reducing the overall size of the software and runtime memory usage.

[0113] ◇ Mechanize and shift dependency management to the left: Transform dependency compliance checks from "human review" and "debugging after problems occur" to "mechanical rules that are automatically executed at startup." The audit function acts like a static code analysis tool, identifying issues early in the development and integration phases. Improved development efficiency: Developers receive immediate, clear, and automatic compliance feedback when adding or modifying overlay dependencies, eliminating reliance on vague experience or lengthy cross-environment testing. Reduced maintenance costs: Potential runtime compatibility issues are detonated before compilation / distribution, avoiding user complaints and emergency fixes caused by environmental problems after software release.

[0114] ◇ Seamless integration with existing architecture and enhanced observability: Audit results are logged as "warnings" in the log repository maintained by the unified log bus module. This allows dependency compliance issues to be covered by existing diagnostic interfaces, monitoring dashboards, and the "ironclad self-checking SQL paradigm." The health status of system dependencies is incorporated into the global observability system. Operations personnel can understand the lock file compliance status of all Overlay engines by querying logs, much like querying translation latency, achieving unified management from an operations perspective.

[0115] Furthermore, the specified process audio capture channel based on the operating system process audio loopback application interface is implemented by the computer device by performing the following sub-steps:

[0116] Call IMMDeviceEnumerator to enumerate all active audio sessions, and record the PID and process name of each session for the front end to select;

[0117] After selecting the PID, construct the AUDIOCLIENT_ACTIVATION_PARAMS structure with the activation type AUDIOCLIENT_ACTIVATION_TYPE_PROCESS_LOOPBACK and the mode PROCESS_LOOPBACK_MODE_INCLUDE_TARGET_PROCESS_TREE.

[0118] ActivateAudioInterfaceAsync asynchronously, and use COMcompletionhandler to wait for the result;

[0119] The WASAPI data packet is read in the acquisition thread, downmixed to mono, resampled to 16 kHz, and quantized to 16-bit PCM. Each frame of PCM is sent back to the Node.js main thread via N-API's ThreadSafeFunction and then pushed to the rendering process via IPC.

[0120] This invention outlines a technology stack that captures audio from a specific process at the operating system level, processes it in a standardized manner, and then securely delivers it to the application layer processing pipeline. It combines the advanced feature of PROCESS_LOOPBACK provided by Windows to achieve process-level audio capture, which was previously difficult to do.

[0121] This invention also protects a real-time two-way simultaneous interpretation device based on multi-source audio and multi-model collaboration, the device being configured in a computer device for performing the above-described method, the device comprising:

[0122] The audio acquisition module is used to capture audio frames from the user-selected audio source and encode them into standard PCM.

[0123] The translation strategy module is used to call a single decision function for each frame of automatically recognized speech text that arrives, and output an immutable decision object containing four fields: source language, target language, speaker number, and display panel affiliation. The translation strategy module also includes: a language fusion and parsing submodule, used to output the final source language according to a preset priority; and a decision scheduling submodule, used to generate the decision object based on the dual-source / single-source flag and the speaker number.

[0124] The sentence assembly module is used to maintain a sentence assembler for each audio source and atomically transfer each sentence between the three states of "to be processed - sealed - finalized" according to the decision object and the multiple sealing predicate;

[0125] The final state module is used to asynchronously call the large language model adapter for correction and translation after the sentence enters the "sealed" state, and ensure the final state is closed according to the predefined degradation strategy;

[0126] The automatic reading module is used to send the target text of the finalized sentence to the speech synthesis adapter according to the three reading modes of "off / local speaker / virtual microphone", and route it to the corresponding output device according to the mode; the automatic reading module also includes: virtual device self-healing parsing submodule and echo suppression submodule;

[0127] The engine management module is used to host the local inference engine in an independent subprocess and maintain its lifecycle; the engine management module also includes: a signature fast path submodule, an overloaded sequence number guard submodule, and a subprocess crash monitoring submodule;

[0128] The installation management module is used to perform a minimal core installation on the first boot, execute a ten-stage standard installation pipeline when the user enables the local engine, and execute a dependency alignment process on each boot. The installation management module also includes: a dual-GPU detection submodule, a memory budget pre-check submodule, an installation list submodule, and a drift detection submodule.

[0129] The session protection module is used to ensure that engine switching does not interrupt the session by using session epochs and reference counting.

[0130] Furthermore, the device also includes:

[0131] The knowledge base management module is used to implement on-demand knowledge base retrieval (RAG).

[0132] The terminology management module is used to implement bidirectional rules for terminology and soft prompts + UI tags;

[0133] The LAN API sharing module is used to provide an external, AI-compatible entry point.

[0134] The unified log bus module is used to achieve end-to-end latency observability;

[0135] The license base module is used to implement multiple license levels for desktop software.

[0136] Furthermore, the LAN API sharing module exposes the local area network API sharing method for multiple large language model engines configured on the local machine using the OpenAI compatible protocol, including:

[0137] Mount routes with the same URL paths and request body schemas as the OpenAI Chat Completions API and Embeddings API in the FastAPI application of the main process; aggregate multiple large language model providers already configured on the local machine under the same entry point in the form of "virtual model name" mapping, route to the corresponding backend adapter according to the model field of the request body, and return the backend response after converting it back to the OpenAI standard schema;

[0138] Several self-signed API tokens are maintained in the local business database. All external interface calls must include a valid token in the HTTP Authorization Header. Users can revoke any token with one click in the user interface.

[0139] All external interface calls are recorded and statistically analyzed by dimensions such as call token name, backend engine type routed to, call date, usage (character or token estimate), and failure count; error messages in external responses are anonymized according to preset keyword rules;

[0140] Provide call statistics views in a separate pop-up window, categorized by Token / Engine / Model / Time Period.

[0141] This "LAN API Sharing" solution is not simply port forwarding, but a systematic project aimed at solving real-world team collaboration pain points. It completely changes the cumbersome and high-risk model where each device within a team needs to repeatedly and independently configure multiple LLM API keys. Administrators only need to configure and bind all AI engines on a single host, and other team members can seamlessly use them via the LAN. When adding new members or devices to the team, there is no need to distribute and configure keys, achieving "out-of-the-box" functionality. By consolidating the storage of sensitive API keys from all terminal devices to a single point (the shared host), the risk of key leakage due to device loss, system reinstallation, or improper storage is significantly reduced. The unified revocation mechanism also facilitates emergency response.

[0142] Client developers only need to use one SDK (OpenAI) to call all aggregated models, without having to write adaptation code for each vendor; switching, upgrading, and failover of the backend engine are completely transparent to the client. Administrators can replace or load balance the backend at the gateway level without notifying or updating all clients.

[0143] Through mandatory and comprehensive logging and visual statistics, the system provides unprecedented transparency and control over the use of AI services by the team. It can clearly track the usage and costs of different tokens, projects, and time periods, facilitating internal accounting or billing. Real-time monitoring of call frequency and token consumption helps to provide timely warnings before exceeding the budget. The anonymized logging meets internal audit requirements while protecting user privacy and trade secrets to a certain extent.

[0144] The gateway layer's "virtual model name" mapping allows administrators to implement complex routing strategies, routing requests to different backends based on request characteristics (such as content and priority). For example, a low-cost model can be used for simple tasks, while a high-performance model can be used for complex tasks. When a backend service (such as a specific cloud vendor's API) becomes unavailable, requests can be automatically routed to a backup engine, ensuring the overall availability of the team's services.

[0145] Furthermore, the unified log bus module maintains an SQLite log library independent of the business database. The log library stores logs in a single table named logs, with fields including timestamp, category, component name, level, source process, session identifier, trace identifier, message, and load. The category is one of 10 closed enumeration values: system / install / engine / audio / asr / llm / tts / session / ui / rag.

[0146] Background tasks clean up old logs every 5 minutes according to their level (debug level logs are kept for 1 hour, info level logs for 6 hours, and warn and error level logs for 7 days).

[0147] A Pydantic model is defined as the load schema for each of the 10 categories, and all logs must pass validation before being written.

[0148] For each sentence event traversing the system, four timestamps are embedded: asr_yielded, event_emit, ws_received, and seg_rendered. These correspond to the moment when the automatic speech recognition adapter outputs the text, the moment before the sentence assembly module delivers the event to the WebSocket sending channel, the moment when the front-end WebSocket on Message callback entry is reached, and the moment when the DOM pixels have been updated after the front-end Vue view is refreshed.

[0149] A pre-defined SQL paradigm that can be directly executed on the log database is used to verify two architectural rules: "the original text arrives before the translation" and "the large language model does not block the automatic speech recognition output". The former is achieved by comparing the event timestamps of the same sentence identifier to determine the order of the source / target fields, while the latter is achieved by associating the adjacent intervals of the asr_yielded event with the large language model calls during the same period.

[0150] This observability solution equips the entire real-time translation system with a "full-body CT scan" and an "autonomous driving diagnostic system," with far-reaching technical effects covering the entire lifecycle of development, testing, operation and maintenance, and architecture evolution.

[0151] ◇ Achieve measurable, traceable, and debuggable internal states of complex systems: Transform the originally black-box, multi-component asynchronous pipeline into a white-box data stream with a precise timeline. Any subtitle delay anomaly can be quickly identified by querying its four timestamp links, pinpointing whether the bottleneck occurs in ASR, LLM, network, or rendering stages; this greatly reduces troubleshooting costs, enabling vague user feedback such as "latency glitches" and "subtitle stuttering" to be rapidly and accurately translated into locatable code module or configuration issues.

[0152] ◇Provides data-driven verification methods for architecture design and iteration: The "Ironclad Self-Check SQL Paradigm" transforms the qualitative principles set by architects (such as "non-blocking" and "original text priority") into automated, continuously run quantitative tests. After each code commit, configuration change, or dependency upgrade, these SQL statements can be run to verify that core architectural constraints have not been violated. This is equivalent to establishing a continuous integration testing protection network for the system architecture, preventing performance regressions or architectural decay during rapid iterations, and ensuring the long-term health of the system.

[0153] ◇Supports the establishment of performance baselines and capacity planning: Structured end-to-end logs enable the establishment of baselines for various performance metrics (P50, P95, P99 latency), allowing analysis of performance under different scenarios (such as long sentence translation, technical terminology) and different engines (local model vs. cloud API). This provides a scientific data basis for capacity planning, resource allocation, and user experience optimization, enabling system optimization to move from "gut feeling" to "data analysis."

[0154] ◇Enhancing the proactiveness and predictability of system operations and maintenance: Through continuous monitoring and analysis of logs, abnormal patterns can be detected in advance, such as a slow increase in the call time of a specific engine or an increase in the failure rate of a certain type of request, triggering alarms before affecting a large number of users. This transforms reactive firefighting into proactive operations and maintenance, improving the overall availability and stability of the system.

[0155] ◇Enhanced governance capabilities of modular architecture: 10 closed log categories force all modules to output logs according to specifications, and a unified load model ensures the quality of log data. This allows the behavior of modules developed from different teams or at different times (such as audio acquisition, ASR adapter, LLM facade) to be uniformly observed and understood; reduces the cognitive burden of system complexity and enhances the maintainability of large software projects.

[0156] Furthermore, the authorization base module maintains a method for licensing management of computer software, the method comprising:

[0157] The system receives and stores a digital authorization certificate issued by an authorization server. The digital authorization certificate includes a long-term activation code and a short-term verification code. The long-term activation code is a readable string consisting of a prefix, a plan abbreviation, and a three-segment four-character random short code, where the random short code is taken from a predefined character set excluding easily confused characters. The short-term verification code is obtained by the computer device from the authorization server in exchange for the long-term activation code. It uses an Edwards curve EdDSA signature, and the payload includes at least: an authorization identifier, a device identifier information field, an authorization capability list field, and a validity period field. The device identifier information field is an array of device identifiers used to bind one digital authorization certificate to multiple client devices. The authorization capability list field is a set of one or more capability identifiers.

[0158] Based on the matching verification of the device identification information and the locally calculated device fingerprint, it is determined whether the current client device has been authorized. The determination of whether the current client device has been authorized includes: verifying whether the locally calculated device fingerprint is included in the device identification array; wherein, the locally calculated device fingerprint is generated based on combining at least two types of inherent information of the following device hardware or operating system and calculating their hash values: machine globally unique identifier, central processing unit identification information, and motherboard identification information;

[0159] During the validity period of the digital authorization credential, access control is performed on the client's software functions based on the authorization capability information contained therein; the access control on the client's software functions includes: when a user attempts to use any software function, checking whether the capability identifier corresponding to that function exists in the authorized capability list;

[0160] When the digital license is about to expire, a renewal request is sent to the license server to obtain a new digital license; in the offline state where the license server cannot be connected, the software function is allowed to continue to be used within the grace period specified in the digital license; if the renewal cannot be completed after the grace period, the software function is locked.

[0161] Identify the authorization plan type in the digital authorization certificate; if it is identified as a lifetime authorization type, skip the check on the validity period field, but continue to perform device binding verification and authorization revocation checks;

[0162] Periodically retrieve and verify a digitally signed list of license revocations from the authorization server; if the identifier of the current digital authorization credential exists in the verified list of license revocations, immediately lock the software functionality;

[0163] Record the time when the license revocation list was last successfully obtained; if the client device remains offline for more than the preset maximum number of offline days since that time, the software functions will be locked and a new network connection verification will be required even if a new license revocation list cannot be obtained.

[0164] This licensing management approach is not a collection of isolated features, but an organic whole that collectively addresses the systemic challenges of traditional desktop software licensing in terms of security, flexibility, usability, and business sustainability.

[0165] ◇ Achieves secure, flexible, and user-friendly device management: Utilizing a DIDS (Device Identifier Array) + composite device fingerprint mechanism, it strikes an optimal balance between security and convenience. Secure Binding: Fingerprints based on multiple hardware information are difficult to forge, effectively preventing the unauthorized spread of authorization on unauthorized devices. Legitimate Multi-Device Support: Supports users to legally use the same authorization on multiple of their own devices, aligning with modern users' multi-device work habits and greatly enhancing user experience and purchase value. Flexible Management: The server can manage the device binding list by updating the DIDS array, without needing to change the activation code.

[0166] ◇A fine-grained, dynamically adjustable feature licensing model has been built: feature unlocking is decoupled from the licensing plan and controlled through a feature capability list. Fine-grained licensing: Different feature combinations can be configured for different users or subscription tiers, enabling differentiated product pricing and services. Dynamic upgrades: Service providers can dynamically unlock new features for users by renewing credentials containing new features without changing the user's authorization credentials, supporting flexible operational strategies.

[0167] ◇Ensuring high availability and a graceful offline experience: The state machine design of "heartbeat renewal + offline grace period" balances the real-time nature of online verification with the practical needs of offline use. Business continuity: Users can continue to use the software even during temporary network interruptions or while traveling, avoiding business interruptions caused by brief network outages. Controllable lifecycle: The grace period provides a reasonable time window, after which mandatory online verification ensures the eventual consistency of the authorization status and prevents indefinite offline use.

[0168] ◇ Overcame industry challenges in license revocation and anti-piracy: The combination of a "signed revocation list (RL) + forced online connection after offline time" constitutes a powerful proactive revocation system. Instant revocation: Licenses can be remotely revoked almost in real-time upon detection of license leaks or abuse, significantly reducing piracy losses and response time. Defense and evasion: By setting a "maximum offline days" and forcing online connection, it effectively combats piracy techniques such as "using the device offline to evade revocation," significantly enhancing anti-piracy robustness.

[0169] ◇ Achieving a balance between long-term trustworthiness of business promises and technological controllability: The "lifetime license" bypasses expiration checks but is still subject to device binding and revocation lists. Fulfilling the promise: It truly delivers on the "one-time purchase, lifetime use" promise. Even after the service terminates in the future, activated lifetime users can continue to use the service, establishing strong user trust and product reputation. Retaining control: It does not relinquish all control; the service provider can still handle extreme situations (such as large-scale key leaks) through the revocation mechanism, protecting its own legitimate rights while respecting user rights.

[0170] The present invention also protects a computer device, the device including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus;

[0171] The memory is used to store computer programs;

[0172] When the processor executes the program stored in the memory, it implements the steps of the real-time bidirectional simultaneous interpretation method based on multi-source audio and multi-model collaboration as described above.

[0173] Beneficial effects of this invention:

[0174] 1. Stable and jitter-free subtitles: The sentence assembler uses a three-state atomic transfer of "pending processing - sealed - finalized", combined with multiple sealed predicates and the longest common prefix stabilization algorithm and overlap stripping algorithm, to fundamentally solve the problems of "infinite bubble growth" and "back-off jitter" in streaming ASR, and ensures that the subtitles are always closed through the asynchronous finalized "timeout degradation" path;

[0175] 2. Panel and translation direction are consistent: A single decision function simultaneously calculates four fields: source language, target language, speaker number, and display panel affiliation, eliminating the problem of inconsistent subtitle display position with the target language; in dual-source mode, the physical channel is used as the anchor point to ensure that the panel does not shake in bilingual scenarios;

[0176] 3. Model can be switched hot during runtime: Independent subprocesses carry and sign fast paths, and overloaded sequence number guarding mechanisms make switching AI models during runtime both reliable and uninterrupted to the meeting; binary frame protocol avoids the "line feed / encoding / buffering" trap;

[0177] 4. Automatic reading can be reloaded with one click: The "three reading modes" combined with browser audio interface device binding and device ID self-healing parsing enable the translation results to be regarded as microphone input by third-party conferencing software; the local speaker mode relies on browser echo cancellation to suppress the TTS output from being picked up by its own ASR, and the virtual microphone mode audio is output through an independent virtual device and naturally does not have this feedback.

[0178] 5. User-friendly for those without GPUs: The "on-demand minimal installation" and "dual-GPU detection + memory budget pre-check + hardware gating" mechanisms keep the size of the distribution installation package to around 300MB, so pure cloud users do not need to download local models;

[0179] 6. Zero-intervention upgrade: The "ten-stage standard installation pipeline" combined with the "basic dependency alignment + overlay dependency drift detection" mechanism at startup automatically converges the differences between the installation states of the new and old versions;

[0180] 7. Precise audio source selection: Based on the process audio loopback application interface of the operating system process, the process audio capture mechanism allows the translation range to be precisely locked to a specific application, avoiding the mixing of noise such as system notification sounds and other videos;

[0181] 8. Session Uninterrupted Protection: A session protection mechanism based on session epoch reference counting ensures that runtime configuration changes will not interrupt the current meeting, improving the enterprise-level user experience;

[0182] 9. Decoupling of original text correction and final translation: The design of splitting the sentence final state into two independent large language model calls allows the correction of proper nouns, personal names, numbers, etc. at the original text level and the final translation to be filled into the subtitles in the same atomic final state. Moreover, the failure of one call will not affect the other (the first call failure uses the archived original text, and the last call failure uses the preview translation).

[0183] 10. Consistency of professional terminology can be guaranteed: The four-in-one terminology management method of "two-way rules + soft prompts + verification + UI tags" ensures the consistency of professional terminology without compromising the naturalness of the translation;

[0184] 11. Knowledge base retrieval can assist in translation proofreading and conference Q&A: The on-demand knowledge base retrieval method of "dedicated query keyword extraction + embedding similarity Top-K recall + front-end visualization" allows users to compare external domain knowledge background without affecting real-time translation delay;

[0185] 12. Multi-device shared host configuration: The LAN API sharing method based on OpenAI compatible agent enables the LLM configuration on a host to be reused by other devices in the local area network, and has the ability to de-identify and audit.

[0186] 13. End-to-end latency observability and self-checking: The observability method of "independent log library + 10 types of indexes + hierarchical TTL + Pydantic strongly typed payload + four timestamp end-to-end tracking + ironclad self-checking SQL paradigm" enables any architectural change to be verified within 5 minutes through SQL self-checking without introducing regression;

[0187] 14. Multi-tiered Authorization Management Method: A mechanism combining a device identifier array (dids) and a composite device fingerprint achieves an optimal balance between security and convenience. Function unlocking is decoupled from the authorization plan, controlled through a feature list. Different feature combinations can be configured for different users or subscription tiers, enabling differentiated product pricing and services. Service providers can dynamically unlock new features for users by renewing credentials containing new features without changing user authorization credentials, supporting flexible operational strategies. The state machine design of "heartbeat renewal + offline grace period" balances the real-time nature of online verification with the practical needs of offline use. The combination of "signed revocation list (RL) + forced online connection during offline duration" constitutes a powerful proactive revocation system. Attached Figure Description

[0188] Figure 1 This is a data flow diagram of the real-time bidirectional simultaneous interpretation method based on multi-source audio and multi-model collaboration disclosed in this invention;

[0189] Figure 2 This is the sentence three-state lifecycle state machine of Example 1;

[0190] Figure 3 This is a flowchart of the single-decision-point translation routing method in Example 2;

[0191] Figure 4 This is a flowchart of the three-level automatic reading routing process in Example 3;

[0192] Figure 5 This is the state transition diagram of the engine hot overload state machine in Example 5;

[0193] Figure 6 This is a flowchart illustrating the ten-stage standard installation pipeline execution process in Example 7.

[0194] Figure 7 This is a timing diagram for session epoch protection and audio frame time-division multiplexing in Example 8;

[0195] Figure 8 This is a schematic block diagram of the computer device in Embodiment 10.

[0196] Explanation of reference numerals in the attached figures

[0197] 500. Computer equipment; 501. Communication bus; 502. Processor; 503. Storage medium; 5031. Operating system; 5032. Computer program; 504. Internal memory; 505. Communication interface. Detailed Implementation

[0198] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The embodiments of the present invention are given for illustrative and descriptive purposes only, and are not intended to be exhaustive or to limit the invention to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described to better illustrate the principles and practical application of the invention, and to enable those skilled in the art to understand the invention and design various embodiments with various modifications suitable for a particular purpose.

[0199] In this specification, "computer device" refers to any electronic device with computing capabilities, including but not limited to desktop computers, laptops, workstations, and other devices capable of running the Windows 10 / 11 operating system; "automatic speech recognition adapter" refers to a software component that provides transcribe_stream / transcribe_batch interfaces externally and internally encapsulates a specific ASR model implementation; "large language model adapter" and "speech synthesis adapter" have similar meanings; "main process" refers to the Python process that carries the FastAPI backend and is responsible for overall scheduling; "child process" refers to an auxiliary Python process that carries a specific local inference engine and runs independently outside the main process, hereinafter also referred to as the overlay child process; "WebSocket interface" refers to the bidirectional real-time communication channel opened by the main process to the frontend; "frontend" refers to the Vue3 single-page application running within the Electron rendering process.

[0200] Figure 1 This is a data flow diagram of the real-time bidirectional simultaneous interpretation method based on multi-source audio and multi-model collaboration disclosed in this invention. To clarify the different improvements of this invention, the functional modules of the real-time bidirectional simultaneous interpretation method based on multi-source audio and multi-model collaboration disclosed in this invention will be described separately below through different embodiments.

[0201] Example 1: A Streaming Subtitle Assembly Method Based on a Three-State Sentence Lifecycle Model

[0202] This embodiment provides a streaming caption assembly method based on a three-state sentence lifecycle model, executed by the computer device. This method can be performed by a "sentence assembly module" in the computer device.

[0203] In this embodiment, the sentence assembly module maintains an independent assembler instance for each audio source (e.g., microphone path mic and system playback path sys). Each assembler instance holds only one active sentence in the "PENDING" state at any given time and maintains a dictionary of "sealed sentences awaiting finalization". See the sentence three-state lifecycle state machine for details. Figure 2 As shown.

[0204] Step S101, Sentence allocation: When the computer device receives the first frame of non-empty text for a certain audio source from the automatic speech recognition adapter, it calls the global sentence counter to allocate a globally unique and monotonically increasing integer identifier (hereinafter referred to as "sentence ID") for the sentence, and creates a data structure in memory as the state carrier of the sentence, with the initial state set to "PENDING", that is, pending processing.

[0205] In this embodiment, the global sentence counter is shared between the two audio sources (mic / sys) to ensure that the sentence ID is strictly monotonically increasing throughout the translation session. The sentence ID is not only used as a field in the event schema, but also serves as an anti-pollution anchor in the asynchronous preview translation sub-process (see step S108 for details).

[0206] Step S102, Receiving in the pending state: When in the pending state, the computer device maintains two text buffers—an acknowledged buffer and a temporary buffer; the acknowledged buffer stores "final" type text pushed by the ASR adapter; the temporary buffer stores "partial" type text pushed by the ASR adapter. Each time the computer device receives a partial text, it executes the longest common prefix algorithm, using only the common prefix portion with the previous partial text as "stable text" to drive subsequent judgments, thereby absorbing partial text backoff from ASR.

[0207] Step S103, overlap stripping: When the previous speech segment is forcibly cut and the newly arrived part of the text in this round overlaps with the end of the confirmed buffer by a maximum of 6 standardized words after removing capitalization and punctuation, the computer device strips the overlapping head of the text in this round and writes it into the buffer by comparing the length k by k, so as to avoid the repetition of boundary words.

[0208] Step S104, sealing condition determination: The computer device executes step S105 when any of the following predicates is triggered:

[0209] Predicate 1 (punctuation mark sealing): The sentence ends with a punctuation mark (period, question mark, exclamation mark, Chinese period, Chinese question mark, Chinese exclamation mark, ellipsis, etc.);

[0210] Predicate 2 (Silent Sealing): The difference between the last update time and the current time is ≥2000 milliseconds;

[0211] Predicate 3 (soft upper limit sealing): Sentence duration ≥ 6 seconds, and the interval between the most recent speech activity ≥ 300 milliseconds;

[0212] Predicate 4 (hard upper limit sealing): Sentence duration ≥ 10 seconds;

[0213] Predicate 5 (Stale Filler): The sentence is generated by "forced truncation" and has a pause of more than 8 seconds.

[0214] Furthermore, to prevent the predicate from being unevaluable when the ASR does not send any part of the text during the silent segment, the computer device additionally schedules a "wall-clock tick" in memory every 200 milliseconds, using the difference between the wall-clock time and the last update time as a fallback estimate of the voice activity interval.

[0215] Step S105, Sealing and Event Issuance: Simultaneously with the status switching to "SEALED", the computer device executes the following sub-steps:

[0216] S105a: Freeze the sentence text field so that subsequent ASR outputs will not be appended;

[0217] S105b: Send a "seg_sealed" event to the front end via the WebSocket interface. The event carries: sentence ID, source language, target language, display panel, sealed original text, and preview translation.

[0218] S105c: Scrolls the sentence assembler of the audio source and assigns a new sentence ID to enter a new round of PENDING;

[0219] S105d: Add the sealed sentence to the "Dictionary of Pending Finalization".

[0220] Step S106, Asynchronous Finalization with Secondary Decoupling: The computer device asynchronously schedules the finalization task locally. Unlike the traditional "single LLM call + multi-tag parsing" path, this embodiment breaks down the finalization into two independent, parallel, and separately degradeable large language model calls:

[0221] Step S106a, ASR Error Correction Call (First Call): The computer device initiates the first call to the large language model adapter using the `asr_correct` prompt template. The input is the archived source text and the context of the N most recent finalized source texts in the same language (default N=8). The system prompt specifically requires "only output the corrected source text in the same language, without any translation." The string returned by this call is temporarily stored as the `corrected_source` field of the target sentence, serving as the input for the second translation call. This call has an independent timeout threshold (default 4 seconds, half of the total timeout budget of 8 seconds for finalization) and a degradation path—if the timeout occurs, the uncorrected archived source text is used.

[0222] Step S106b, Last Call: Immediately after the First Call returns, the computer device initiates a second call serially using the `translate` prompt template. The input is the corrected source text (or the archived source text if the First Call times out or fails), and the K most recent finalized source-translation pairs (K=3 by default, only if the source and target languages ​​are different) are appended as paragraph context. The string returned by this call is written to the `final_translation` field. This call has an independent timeout threshold (default 4 seconds, half of the total finalization timeout budget of 8 seconds) and an independent degradation path—if the timeout occurs, the preview translation generated in the PENDING stage is reused.

[0223] Step S106c: The trace_id of the two calls are associated: First Call and Last Call share the same sentence identifier, but each is assigned an independent `call_kind` field ("finalizer_correct" and "finalizer_translate" respectively, and "preview" for the preview call in the PENDING stage) which falls into the structured log, making it easy to track the time consumption, error cause and input / output of the two calls in the diagnostic interface.

[0224] Step S106d, cancellation semantics of two calls: If the sentence is still in progress in either First Call or Last Call, and is determined by "retroactive merge (see step S109)" to be merged with a newly archived sentence, the computer device cancels the asynchronous tasks corresponding to the two calls and releases the corresponding resources to avoid polluting the fields of the subsequent merged sentence by the expired calculation results of the previous sentence.

[0225] Step S107, Final State Atomic Upgrade: After both the First Call and Last Call return, the computer device performs an atomic "Sentence State Field Update," writing the `corrected_source` and `final_translation` fields together, and atomically upgrading the state from SEALED to FINALIZED. If the First Call times out or parsing fails, the uncorrected, sealed original text is used as a fallback for `corrected_source`; if the Last Call times out or parsing fails, the PENDING stage preview translation is used as a fallback for `final_translation`. This atomic upgrade is still performed even if any or all of these states are downgraded, ensuring that the front-end subtitles can be completed under any failure.

[0226] Step S108, Asynchronous Preview Anti-Pollution: In the PENDING stage, the computer device triggers the translation of "stable text" with a short delay (default 200 milliseconds for jitter) in parallel through the preview translation coroutine. When the preview result is sent back, the sentence ID recorded at the time of request is compared with the currently active sentence ID of the assembler. If they do not match, the result is discarded, thereby avoiding the translation of the previous sentence from polluting the subtitle bubble of the new sentence.

[0227] Step S109, Retroactive Merge: Before a new sentence is transferred to SEALED, the computer device compares whether the previous adjacent SEALED sentence is still in a "pending finalization" state, whether the difference between their timestamps falls within the merge window (default 10 seconds), and whether the first character of the new sentence presents a lowercase "continuation of the previous sentence" signal. If all three conditions are met, the asynchronous First Call / Last Call task dispatched by the previous sentence is cancelled, the two original sentences are merged, a new First Call / Last Call is dispatched again with the merged sentence identifier, and a "segment discarded" event is sent to the front end to clean up the merged old bubbles. This mechanism specifically solves the problem that some ASR engines incorrectly insert punctuation at the end of long sentences, causing the preceding and following sentences to be mis-segmented.

[0228] In this embodiment, the method achieves real-time generation of semantically complete, stable, and non-backtracking subtitles on a continuously arriving speech stream; by using parallel judgment of five types of sealed predicates, the length of the segment to be processed is converged multiple times, fundamentally solving the problem of "infinite bubble growth"; the longest common prefix stabilization algorithm absorbs the backtracking of ASR text; the overlap stripping algorithm solves the problem of boundary word repetition at forced truncation points; the asynchronous finalization with secondary decoupling (two independent LLM calls, First Call / Last Call) and the "timeout degradation" path ensure that the subtitles have a certain closing point; the monotonically increasing sentence ID + preview result alignment verification prevents asynchronous translation misalignment; and the backtracking merging mechanism corrects adjacent sentences that are miscut by ASR.

[0229] The design of "decoupling the original text correction and translation into two calls" in this embodiment has the following essential advantages over the traditional "single LLM call + multi-tag parsing of `[CORRECTED] / [TRANSLATION]`": 1. Failure of one call will not affect the other, improving overall availability (the archived original text is used when the First Call fails, and the previewed translation is used when the Last Call fails); 2. Each call task is singular after splitting, avoiding the translation truncation problem that occurs on some models when using mixed prompts; 3. Each of the two calls has independent timeout and degradation thresholds, and independent structured logs (distinguished by call_kind), making it easier to locate the performance bottlenecks and errors in the "correction" and "translation" stages respectively.

[0230] Example 2: A bidirectional streaming speech translation routing method based on a single decision point

[0231] This embodiment provides a "single decision point" translation routing method executed by the computer device. The method can be implemented by a "translation strategy module" within the computer device. See the flowchart for the single decision point translation routing method. Figure 3 .

[0232] In this embodiment, the translation strategy module consists of two sub-modules: a language fusion parsing sub-module (a pure function without side effects) and a decision scheduling sub-module.

[0233] Step S201, Language Fusion and Parsing: The computer device calls the language fusion and parsing submodule, with the following inputs: the language code given by acoustic recognition, the ASR text, the user-configured source language base code, and the user-configured target language base code. The submodule returns the final source language according to the following priority:

[0234] If the language returned by the text language identification falls within the set of [source language basic code, target language basic code], then the text language identification result is adopted.

[0235] Otherwise, the acoustic recognition result will be adopted (if it is valid);

[0236] Otherwise, the text language recognition result will be adopted (if valid);

[0237] Otherwise, it will be downgraded to the source language configured by the user.

[0238] Furthermore, when comparing languages, the basic code (such as "zh" instead of "zh-CN") is used to eliminate dialect suffix differences.

[0239] Step S202, Single Decision Entry: The computer device calls a unique decision function `decide(...)` for each frame of arriving ASR text. The input parameters include: acoustic language, ASR text, whether it's a dual-source mode flag, and the fixed speaker number corresponding to the current audio source. The decision function returns an immutable "decision object" containing the following fields:

[0240] src_lang: The source language confirmed in this identification;

[0241] tgt_lang: The target language for this test;

[0242] speaker: Speaker ID (0 indicates microphone / our side, 1 indicates system / other side);

[0243] panel: Displays which panel belongs to ("left" or "right");

[0244] reason: A parsable string representing the reason for the decision, such as "mono-tgt-flip,src=en,tgt=zh".

[0245] Step S203, Dual-source branch: When the system is in "microphone + system playback" dual-source mode, the computer device directly uses the physical channel as the speaker anchor point:

[0246] Microphone path (speaker=0) is fixed with panel="right", source language is the real-time recognition result, and target is the user-configured source language;

[0247] The system playback path (speaker=1) is fixed with panel="left", the source language is the real-time recognition result, and the target language is the user-configured target language.

[0248] In this embodiment, this branch is not affected by the acoustic / text language recognition results, thus avoiding panel shaking caused by language fluctuations in bilingual users.

[0249] Step S204, Single-source branch: When the system is in single-source mode, the computer device determines the route according to the detection language:

[0250] If the detected source language base code is equal to the configured target language base code, it is determined that "the user speaks in the opposite direction", speaker=1, panel="right", target=user configured source language, reason="mono-tgt-flip,src=...,tgt=...";

[0251] If the detected source language base code is equal to the configured source language base code, it is determined to be "speak according to configuration", speaker=0, panel="left", target=user configured target language, reason="mono-src-accept,src=...,tgt=...";

[0252] If both are not equal (out-of-scope), then translate uniformly to the target language configured by the user, speaker=0, panel="left", reason="mono-out-of-scope, src=..., tgt=...".

[0253] Step S205, Decision Transmission and Log: The decision object is saved in an immutable form by the sentence assembly module as a "routing snapshot" of the sentence. All subsequent events of the sentence (sealing, finalization) are forcibly read from this snapshot to populate the panel / src_lang / tgt_lang fields, avoiding inconsistencies in event routing for the same sentence due to language misjudgment after large language model correction. Simultaneously, the computer device writes the reasoning, panel, speaker, and source / target language of the decision object into a structured log database (SQLite) for querying by the diagnostic interface.

[0254] In this embodiment, the method solves the problem of inconsistent subtitle display position with the target language; the dual-source branch uses physical channels as anchor points to prevent the subtitle panel from shaking in bilingual scenarios; the single-source branch uniformly translates out-of-scope text to the target language instead of discarding it, improving the user experience; the decision object is immutable and can be safely passed between async tasks, avoiding race conditions.

[0255] Example 3: Co-transmission outgoing method based on three-level automatic reading routing: "Off / Local speaker / Virtual microphone"

[0256] This embodiment provides a "three-level automatic reading routing" output method executed by the computer device. The method can be performed by the "automatic reading module" in the computer device. See the flowchart for the three-level automatic reading routing process. Figure 4 .

[0257] Step S301, Mode Enumeration and State Management: The computer device maintains a reading mode state variable for each display panel, with a value of one of the following three levels:

[0258] "off": Off mode, the computer device does not perform any TTS synthesis;

[0259] "local": Local speaker mode, where TTS audio is played through the computer device's default audio output device;

[0260] "vmic": Virtual microphone mode, where TTS audio is played through a user-mode virtual audio device and captured by third-party conferencing software as "microphone input".

[0261] In this embodiment, the modes of the two panels and the target text (source / translation) are persisted independently in local storage; the user-mode virtual audio device includes, but is not limited to, VB-AudioVirtualCable, Voicemeeter, etc.

[0262] Step S302, Automatic detection of new sentences: The computer device subscribes to the "finalized sentence set" of each panel in the front-end responsive state system. Once the number of sentences in the set changes, the "new sentence detection" process is triggered.

[0263] S302a: Skip panels in "off" mode;

[0264] S302b: Use the recorded "Recently Read Sentence ID" to filter out new sentences that have not yet been read aloud;

[0265] S302c: Determine whether to extract the source text or the translation based on the "Target Text Selection" in this panel;

[0266] S302d: Enqueue each item to be read aloud into the serial playback queue and move the "most recently read sentence ID" to the front;

[0267] S302e: Start the playback coroutine consumption queue.

[0268] Step S303, Synthesis Request: The playback coroutine sequentially retrieves the items to be read from the queue and submits an HTTP POST request to the / tts / speak interface of the computer device's main process. The request body contains text and the target language code. The main process returns binary audio in WAV or MP3 format according to the bound speech synthesis adapter. If the TTS engine is still in the cold start phase, the main process will block and wait for a maximum of 150 seconds.

[0269] Step S304, Output Device Routing: The computer device uses the browser audio interface at the front end to create an HTMLAudioElement instance, and executes different branches according to the current audio level:

[0270] If the setting is "local", the element's play() method is called directly to play the content on the default output device.

[0271] If the setting is "vmic", first call setSinkId(virtual microphone device ID) to bind the output of this element to the virtual microphone device, and then call play().

[0272] Step S305, Virtual Device Self-Healing Resolution: Before requesting playback, the computer device calls the "Virtual Microphone Resolution Submodule":

[0273] S305a: The submodule calls the browser enumeration interface to obtain all audio output devices;

[0274] S305b: If the device ID stored in the database still exists in the enumeration result, use it directly;

[0275] S305c: If not present, perform case-insensitive matching of all device tags according to the predefined keyword set (including "cableinput", "vb-audio", "voicemeeter", "loopback", "virtualcable", etc.);

[0276] S305d: Upon a successful hit, replace the old ID in the database with the newly discovered device ID and notify the main process to persist the update.

[0277] In this embodiment, since the audio output device ID in the browser / Electron may change each time the device is re-enumerated, the self-healing resolution mechanism makes the device ID drift transparent to the user.

[0278] Step S306, Echo Suppression: In the "local" mode, TTS playback via the local speaker may be re-picked by the local microphone and sent to ASR. This invention relies on browser echo cancellation enabled by front-end audio acquisition (getUserMedia constraints) and provides an echo risk warning when the user first switches to the "local" mode to suppress this feedback. The "vmic" mode naturally does not have this problem because the audio is output via an independent virtual device and is not physically picked up by the local microphone. It should be noted that the mute / unmute control frame mechanism on the main process side is used for recording streaming scenarios in the author panel (see step S803 in embodiment eight), and is not used for echo suppression during local TTS playback.

[0279] Step S307, Manual routing in the writing field: When the user clicks the send button in the writing field, the computer device executes steps S303-S305, but skips the automatic detection step S302; and routes the output according to the current reading mode of the panel (off / local speaker / virtual microphone) to "speak" the sentence typed by the user to the other party in the meeting.

[0280] In this embodiment, the method avoids modification of system-level audio routing configuration by combining the browser standard setSinkId with user-space virtual drivers; front-end responsive monitoring and serial playback queues enable "instant playback"; device ID self-healing resolution makes device ID drift transparent to the user; and software-level echo suppression avoids infinite loops caused by TTS output being picked up by its own ASR.

[0281] Example 4: A Multi-Task Large Language Model Translation Facade Method Based on Shallow Copy Injection and Directory-Based Registry

[0282] This embodiment provides a method for invoking the "translation facade + prompt word registration center" executed by the computer device.

[0283] Step S401, Registration Center Directory Declaration: The computer device maintains a "prompt word directory" in memory, which consists of several "prompt word definition objects".

[0284] Each defined object contains the following fields:

[0285] ① Prompt Word Identifiers: In this embodiment, a total of 12 identifiers are declared in the directory, namely `translate` (real-time ASR translation LLM call), `asr_correct` (ASR error correction call in First Call in Embodiment 1), `interpret` (compose-bar translation in simultaneous interpretation mode), `dict` (dictionary mode in text translation page, also used in vocabulary definition scenarios), `translate_text` (paragraph mode in text translation page), `summary` (meeting minutes generation), `meeting_qa` (question-answer prompt words based on meeting context, used for debriefing questions in the "Meeting Minutes Panel"), `rag_query_extract` (query keyword extraction for knowledge base retrieval in Embodiment 9), `speaking_coach` (speaking coach comments), `learning_subtitle` (batch translation of foreign language learning subtitles), `learning_asr_correct` (ASR error correction in foreign language learning mode), and `learning_passive_expand` (passive subtitle explanation and expansion).

[0286] ②Persistent key: The key name used in the SQLitesettings table;

[0287] ③ Default text dictionary: Includes both Chinese and English versions;

[0288] ④ Variable metadata list: Variable name + internationalization key, used for front-end display of template variable hints;

[0289] ⑤ Auxiliary field metadata list: such as temperature, keywords, text box / number box type, used for front-end automatic rendering configuration form;

[0290] ⑥ Internationalized grouping and tag keys: used for front-end navigation categorization.

[0291] Step S402, Runtime Parsing and Caching: When a caller needs a prompt word, the computer device queries the SQLitesettings table for the user-defined value corresponding to the key. If it exists, it is used; otherwise, the text matching the current interface language from the default text dictionary is used. The result is written to a memory cache with the key (prompt word identifier, interface language), and the cache is set to a 5-second time-to-live (TTL). When the user modifies the prompt word on the front end and saves it, the computer device clears the relevant cache entries so that the new text is immediately available for the next call.

[0292] Step S403, Unified Facade Interface Call: The computer device implements a Translator facade object, exposing three interfaces: translate_stream (streaming translation, returning an asynchronous iterator), translate_batch (batch translation, returning a complete string), and translate_text (a general interface, automatically selecting one of the above two). The key parameters common to the three interfaces include: source text, source language, target language, prompt word identifier and system prompt word string (one of the two must be passed), and call context. This facade object is shared by all callers, including First Call (asr_correct) and Last Call (translate) in Implementation Example 1, compose-bar simultaneous interpretation calls in Implementation Example 3, keyword extraction from the knowledge base query in Implementation Example 9, text translation pages, and historical retranslations, thereby ensuring that all LLM calls follow the same audit and degradation path.

[0293] Step S404, shallow copy equivalent keyword injection: When the facade interface is called, the computer device obtains the system prompt word for the current call from the registry center, and then passes the prompt word and the original text together to the corresponding method of the underlying large language model adapter in the form of "keyword parameter", without modifying any member variables of the adapter.

[0294] In this embodiment, the mechanism is equivalent to "shallow copy of prompt scope by call", but the cost is only one function parameter passing, and it is inherently thread-safe; it avoids the huge memory and performance overhead caused by the traditional solution of copy.deepcopy(provider) and then modifying the field, and also avoids the concurrency race state caused by directly modifying the adapter member variables and then restoring them.

[0295] Step S405, Dual-segment structured logging: For each facade call, the computer device sends an "llmcallstart" log before the call and an "llmcalldone" log after the call ends (regardless of success, timeout, or exception). Both logs record fields such as call type, provider, model ID, system prompt, user input, output, decision reason, error content, and timeout in milliseconds using structured payload (LlmPayload).

[0296] Step S406, Zero-Modification Front-End Extension: When a new prompt word needs to be added, the developer only needs to append a prompt word definition object to the directory. After the front-end configuration page obtains the JSON representation of the directory by calling the / prompts / catalog interface, it automatically renders the left navigation groups, the main editing area text box, and the auxiliary field input boxes, without requiring any modification to the front-end code.

[0297] Example 5: Engine Hot Reload Method Based on Subprocess Isolation and Sequence Number Guarding

[0298] This embodiment provides an engine hot reload method based on subprocess isolation and sequence number guards, executed by the aforementioned computer device. See the engine hot reload state machine state transition diagram. Figure 5 .

[0299] Step S501, Subprocess Engine Hosting: For each local inference engine, the computer device starts an independent Python subprocess to host it, outside of the main process. The subprocess uses a unified boot script (boot.py) to refactor sys.path, placing the overlay site directory and the server source code directory at the beginning of the path, and then executes the engine's runner script using runpy.

[0300] Step S502, Binary Frame IPC Communication: The main process and the child process establish bidirectional communication through the child process's standard input and standard output. All data is transmitted in units of "frames," with a frame structure of 4 bytes big-endian length + 1 byte type + N bytes payload. Type enumeration includes AUDIO, CONFIG, RESULT, TTS_REQUEST, TTS_AUDIO, ERROR, PING, PONG, AUDIO_END, TRANSLATE_REQ, SHUTDOWN, STREAM_START, etc. (Engine readiness is indicated by the RESULT frame's type=ready subtype; there are no independent READY frames). The maximum frame length is set to 64 megabytes as a defensive upper limit.

[0301] Step S503, Role Lifecycle Machine: The computer device maintains an engine lifecycle machine for five parallel roles (automatic speech recognition, translation, speech synthesis, pronunciation evaluation, and general utility class). Each role's lifecycle object has the following states: IDLE → STARTING → READY or FAILED; READY / STARTING → STOPPING → IDLE. Each state transition is completed atomically under the protection of a lock.

[0302] Step S504, Signature Fast Path: The computer device calculates a signature for each "load request" obtained by regularizing the engine identifier and configuration dictionary serialization. If the new signature is equal to the already effective signature and the role is currently READY or STARTING, it returns directly without triggering any restart.

[0303] Step S505, Reload Sequence Number Guard: With each signature change, the computer device increments the "Reload Sequence Number" by 1 and records it as the "Expected Sequence Number" for this reload, then asynchronously starts a new subprocess loading process. After each critical asynchronous wait, the loading process compares the current expected sequence number with the latest reload sequence number. If they are inconsistent, it means that a new reload request has arrived during this period. The current process immediately stops and cleans up the started subprocesses, without updating the final state.

[0304] Step S506, Crash Self-Detection and Diagnostic Collection: The standard error output of the child process is continuously read by a background task of the main process and its tail buffer is retained; the main process simultaneously listens for the end-of-file signal of the frame channel. When the end-of-file signal arrives without requesting closure, it is considered that the child process has exited unexpectedly, triggering a "crash handling callback". The callback switches the engine state to FAILED and records a structured diagnostic entry, including the stage name (ready_handshake / runtime_crash, etc.), the exception call stack, the stderr tail, the child process PID, the signal number, etc. The diagnostic entry can be read by the / engines / diagnosticsREST interface and displayed as a visual diagnostic page on the front end.

[0305] Step S507, Session Expiration Protection: Before engine switching, the computer device calls the "Session Epoch" module (see Embodiment 8) to mark the current epoch as deprecated, giving a window of up to 10 seconds to allow the ongoing session to end naturally before starting a new engine; if the timeout occurs, it will be forced to continue to avoid the user being suspended for a long time.

[0306] Example 6: Hardware Gated On-Demand Installation Method Based on Dual-GPU Detection and Memory Budget Pre-Check

[0307] This embodiment provides a method for "dual-GPU detection + memory budgeting + on-demand installation" executed by the computer device.

[0308] Step S601, Dual-path detection: The computer device invokes the "GPU capability detection" module, which attempts two methods in the following order:

[0309] S601a: Prioritizes enumerating the list of graphics cards and computing power through the PyTorch interface (torch.cuda.is_available / torch.cuda.get_device_properties);

[0310] S601b: If S601a fails, it will downgrade to using the output of the nvidia-smi command-line tool to obtain the graphics card information.

[0311] In this embodiment, the module returns a unified dictionary, including: whether CUDA is detected, number of graphics cards, index / name / megabytes of video memory / computing power of each card, maximum megabytes of video memory, whether the minimum requirements are met (default threshold 8 gigabytes), and human-readable reasons for non-compliance.

[0312] Step S602, Hardware Gating: Before attempting to load the local engine, the computer device calls the detection module. If it returns "Not Satisfied," the missing engine is recorded in the "Missing Engine Table," containing a reason string and the missing type (e.g., "hardware:gpu<8192MB"), and loading is skipped. The front-end queries the missing engine table in the engine selection dropdown menu, automatically grays out the unsatisfied engines, and displays the reason through tooltips.

[0313] Step S603, Memory Budget Pre-check: For local engines controlled by hardware gating, the computer device further invokes the "Memory Budget Manager". The manager internally maintains the following variables: currently allocated memory, currently reserved memory, and a dictionary of reserved memory amounts recorded by the engine. Pre-check process:

[0314] S603a: Query the min_vram_mb in the engine metadata;

[0315] S603b: Calculate the current available video memory = total video memory - allocated video memory - reserved video memory;

[0316] S603c: If the available video memory is less than min_vram_mb, return (failure, reason string);

[0317] S603d: If successful, min_vram_mb is atomically added to the reserved dictionary and the reserved video memory is updated.

[0318] Step S604, Minimal Installation on Demand: Upon initial startup, the computer device only installs the "core dependency package" (approximately 100 megabytes), containing only the FastAPI backend, frontend, public libraries, and API engine adapters (such as OpenAI and Edge-TTS). Local engine dependency packages are maintained independently as "overlay packages," and are only downloaded and installed when the user explicitly enables a local engine in the UI and clicks "One-Click Install." Each overlay package is installed in %APPDATA% / CrossMeet / overlays / <engine_id> Located in the / directory, isolated from the basic environment.

[0319] Step S605, the front-end on-demand switch is disabled by default: the local engine option is set to disabled by default in the front-end engine binding drop-down box of the computer device. If the GPU is not detected in step S601, the local engine remains disabled and the reason is displayed; if the detection is successful in step S601 but the overlay is not installed, the local engine option is clickable, but a "one-click installation" dialog box pops up after clicking, requiring the user to actively confirm before triggering the download and installation process.

[0320] Example 7: Local Inference Engine Installation Method Based on Ten-Stage Pipeline and Upgrade Idempotent Self-Alignment

[0321] This embodiment provides a "ten-stage standard installation pipeline + upgrade idempotent self-alignment" method executed by the computer device. See the flowchart for the ten-stage standard installation pipeline execution. Figure 6 .

[0322] Step S701, Ten-Stage Standard Pipeline: The computer device executes ten stages for each local inference engine in the order shown in the table below. Failure of any stage immediately terminates the process and writes the context (stage name, error, child process stderr tail) into a diagnostic entry; the output of any successful stage will not be rolled back due to subsequent stage failures (i.e., "forward idempotency"), allowing the user to skip completed stages when retrying.

[0323] |Stage|Name|Content performed by the computer device|

[0324] |------|------|--------|

[0325] |0|precheck|Hardware requirement check (GPU / VRAM) + external dependency (e.g., ffmpeg) detection|

[0326] |1|verify_pytorch| Run the verification script via a subprocess to confirm that the CUDA version of PyTorch in the base environment is ready (if necessary, lock the PyTorch version in the overlay using the official index at this stage).

[0327] |2|hook:pre_pip| An optional engine hook that allows for environment customization before pip installation|

[0328] |3|install_overlay_deps|Installs the declared overlay dependency using the "--targetoverlay--no-deps" lock file|

[0329] |4|hook:post_install| Optional hook for the engine, allowing customized processing such as gitclone / source code patching / directory copying|

[0330] |5|download_weights|Download model weights sequentially from two sources (HuggingFace + ModelScope) (HuggingFace first, ModelScope if failure occurs), and write to the file-level manifest|

[0331] |6|verify_import| Performs actual module import verification in the overlay child process|

[0332] |7|verify_check_installed| Calls the check function declared in the engine metadata, returning (compliance status, list of missing items) for each selected model ID.|

[0333] |8|hook:post_verify| An optional hook for the engine, allowing post-verification processing such as generating sample sound prompts|

[0334] |9|stamp| Write operation records to install.json installation manifest via atomic file|

[0335] Step S702, Sandbox sys.path bootstrapping: The verify_import step in stage 7 reconstructs the module lookup path of the child process through the bootstrapping script with "sys.path.insert(0,overlay)" and "sys.path.insert(0,server)", bypassing embedded Python's ignoring of PYTHONPATH; the probe script is executed with runpy, and any ImportError is returned through stderr, which is then captured by the main process and thrown as an installation error.

[0336] Step S703, install manifest: The install.json file in phase 10 records the following fields: Engine ID, Display Name, Deployment Mode, pip_requires (list of main engine packages), overlay_deps (snapshots of specs for all overlay dependencies), models (list of downloaded model IDs), cuda_target (CUDA wheel tag), installed_at (ISO8601 timestamp), and installed_via (execution module name). A separate manifest.json file is saved in the model weights directory, recording the size and modification time of each file for verification during network outages.

[0337] Step S704, Basic Dependency Alignment at Startup: Each time the software starts, the computer device reads server / requirements.txt and executes the following for each dependency:

[0338] S704a: Extract package name (process version specifiers and hyphens / underscores);

[0339] S704b: Check if the corresponding dist-info directory exists in the basic site directory;

[0340] S704c: If it does not exist, use pip to install it with "--upgrade-strategyonly-if-needed".

[0341] Step S705, Overlay Dependency Alignment at Startup: For each installed overlay engine, the computer device performs the following:

[0342] S705a: missing_overlay_deps checks if the dist-info of the overlay_deps declared in the manifest exists;

[0343] S705b: overlay_deps_drift compares the current catalog declaration with the overlay_deps snapshot of the last install.jsonstamp, and only returns the difference set of the new set (without reinstalling the entire set).

[0344] S705c: If there are differences, call install_overlay_deps to install only the missing items;

[0345] S705d: After successful installation, the overlay_deps field in install.json is immediately overwritten with a new snapshot to avoid repeated triggering on the next startup.

[0346] Step S705e, overlaylockfile locking mode (ADR-22): For each overlay deployment mode, the computer device explicitly declares the `overlay_lockfile` field in the engine metadata catalog of the local inference engine. This field is a relative path to a lock file within the native source tree (e.g., `engines / faster-whisper / overlay.lock`). Each line of the lock file declares a pinned version constraint that the overlay depends on, in one of the following two allowed forms: (i) strict equality form ` <package> == <version>(ii) Remote archiving format <package> @ <url>The URL contains a hash checksum. The lock file is prohibited from containing incomplete pinned forms such as `>=`, `~=`, or no version constraints.

[0347] Step S705f, Overlay Lockfile Consistency Self-Check and Torch Family Pollution Prohibition: The computer device calls a lockfile audit function during the engine service initialization process during application startup. The audit function traverses all overlay and base_venv mode engines and performs the following checks on each engine: (i) The lock file pointed to by the `overlay_lockfile` field must exist in the source code tree; otherwise, the engine is recorded as "no available lockfile" and skipped; (ii) Each non-empty, non-comment line in the lock file must match one of the two pinned forms mentioned above; otherwise, "non-pinned form" violation is recorded and a line number is attached; (iii) The lock file must not contain lines starting with `torch==` / `torchaudio==` / `nvidia-*`, etc. - The torch family dependency must be uniformly managed by the base environment (`server / .venv`). Overlay redeclaration is prohibited to avoid the CUDA runtime occupying several GB of video memory being repeatedly loaded into the overlay directory, and to avoid runtime ABI conflicts caused by torch version drift between different overlay engines. The audit function returns a list of errors; the startup log outputs errors at the warning level (not fatal). Developers can manually run `python-mscripts.audit_engine_lockfiles` for local review.

[0348] Step S705g, the technical effects of the lockfile mode compared to the general dependency declaration: The lockfile mode has the following three technical effects compared to the traditional approach of "developers declaring overlay_deps in the catalog in a scattered form and taking a relatively loose version number according to the requirements": (i) Consistency and reproducibility - when distributed to different users for installation, the lock file ensures that the exact version of each package in the overlay is exactly the same, avoiding the same software version from having different behaviors on different user machines due to the drift of the upstream version of PyPI; (ii) Isolation and auditing - the audit function can mechanically detect patterns such as torch family pollution with hard rules in the SQL paradigm, without requiring developers to re-examine every time they add an overlay dependency; (iii) Upgrade idempotency guarantee - the startup reconcile logic can use the lock file as the "expected state" baseline and perform a difference operation with the dependency snapshot in the installed overlay's `install.json`, only installing the new items instead of reinstalling the whole group, so that the size of the overlay directory is strictly converged in the long-term evolution.

[0349] Example 8: A Session Protection Method Based on Session Epoch Reference Counting and Cross-Process Audio Frame Time Division Multiplexing

[0350] This embodiment provides a method for "session protection + audio time-division multiplexing + process audio capture" executed by the computer device. See the timing diagram for session epoch protection and audio frame time-division multiplexing. Figure 7 .

[0351] Step S801, Session Epoch Creation: The computer device maintains a "Current Epoch" object in memory. This object includes: Epoch ID (UUID), a snapshot of the current engine configuration, an active reference count, and a deprecation flag. When a new session enters the translation process, it enters the epoch through the asynchronous context manager `protect_translation_session`, incrementing the reference count by 1; it decrements the reference count by 1 upon exiting. The epoch itself is not immediately destroyed; it waits for the reference count to reach zero before triggering a cleanup callback.

[0352] Step S802, Epoch Deprecation and Drain: When the engine manager receives a reload request, the computer device:

[0353] S802a: Call deprecate_current_epoch to mark the current epoch as deprecated and immediately create a new epoch;

[0354] S802b: Sessions already in the old epoch continue to run using the old engine; newly entered sessions acquire the new epoch;

[0355] S802c: The main caller waits for the old epoch references to be zeroed out using wait_for_epoch_drain(old_epoch,timeout=10s);

[0356] S802d: Force the reload process to continue even if there are still active sessions after the timeout;

[0357] S802e: Asynchronous cleanup after old epoch references are zeroed out.

[0358] Step S803, Borrowing and Returning Data in the Writing Panel: When the user starts recording in the writing panel, the computer device performs the following:

[0359] S803a: Calling translation.pauseAudio() causes the main WebSocket to send a mute control frame, and the main chain stops consuming from the microphone source;

[0360] S803b: The writing bar link exclusively uses this microphone source and sends the audio to a separate writing bar ASR for one-time transcription;

[0361] S803c: When a user stops recording, calling translation.resumeAudio() sends an unmute control frame, and the main chain resumes consumption.

[0362] Step S804, Operating System API Process Audio Capture: The computer device implements process-level audio capture using the ProcessLoopback capability provided by Windows 10 2004+.

[0363] S804a: Call IMMDeviceEnumerator to enumerate all active audio sessions and record the PID and process name of each session for the front end to select;

[0364] S804b: After selecting the PID, construct AUDIOCLIENT_ACTIVATION_PARAMS with activation type AUDIOCLIENT_ACTIVATION_TYPE_PROCESS_LOOPBACK and mode PROCESS_LOOPBACK_MODE_INCLUDE_TARGET_PROCESS_TREE.

[0365] S804c: Call ActivateAudioInterfaceAsync to activate asynchronously, and use COMcompletionhandler to wait for the result;

[0366] S804d: Reads WASAPI data packets in the acquisition thread, downmixes to mono, resamples to 16kHz, and quantizes to 16-bit PCM;

[0367] S804e: Each PCM frame is sent back to the Node.js main thread via N-API's ThreadSafeFunction, and then pushed to the audio pipeline of the rendering process via IPC.

[0368] Step S805, Unify audio source abstraction: The three sources mentioned above—microphone, system playback, and process audio—are uniformly presented as AudioSource on the front end, and ultimately all enter the source node of the browser's AudioContext and flow into the same ASR pipeline; the back end only needs to distinguish them through the audio_source field (such as "mic" / "system" / "process:zoom_pid_12345"), which is used for log classification and strategy branching.

[0369] Example 9: A Real-Time Translation Knowledge Base RAG Injection Method Based on Embedding Similarity and Query Semantic Extraction

[0370] This embodiment provides a knowledge base retrieval (RAG) method executed by the computer device, which can be performed by the "knowledge base management module" in the computer device.

[0371] Step S901, Multi-knowledge base management and question-and-answer entries: The computer device organizes entries in the local business database with "knowledge base" as the top-level container. Each knowledge base consists of several "question-and-answer entries". Each entry contains question text, answer text, keywords, and floating-point vector, embedding model identifier and dimension returned by the computer device calling the embedding adapter.

[0372] Step S902, OpenAI-compatible Embedding Adapter: The computer device implements an embedding adapter, which uses the ` / v1 / embeddings` interface compatible with the OpenAIChatCompletionsAPI as the external contract abstraction and supports multiple providers including OpenAI, Alibaba Cloud Bailian, and local compatibility services; the embedding adapter has a caching layer for the same text to avoid duplicate requests and the ability to retry in case of network anomalies.

[0373] Step S903, Query Keyword Extraction for Retrieval: When a user initiates a knowledge base retrieval request, the computer device retrieves the most recent context from the current session, calls the `rag_query_extract` prompt registered in step S401, and initiates a lightweight call to the large language model adapter to extract the retrieval keywords, proper nouns, technical terms, etc., from the context in a structured form as a "query string". The query string is intentionally distinguished from the original text itself. Directly using the original text as a query often results in high recall noise due to too many stop words, while the specially extracted query keywords are more focused on the domain terms to be retrieved.

[0374] Step S904, Embedded Similarity Recall: The computer device sends the query string into the embedding adapter to obtain the query vector, and then calculates the similarity of all segment vectors in the currently activated knowledge base using cosine similarity. The segments with the Top-K (default K=3) and similarity higher than the threshold (default 0.5) are taken as the recall results.

[0375] Step S905, Recall Result Return: The computer device organizes the recalled Top-K items (including questions, answers, similarity scores, and sources) into a result set and returns it to the front end through an interface for users to refer to in translation proofreading or conference Q&A scenarios; This method does not automatically inject the recall results into the prompt words of real-time translation calls to ensure low latency and determinism in the real-time translation link.

[0376] Step S906, On-demand triggering: The knowledge base retrieval is an on-demand call initiated by the user (knowledge base Q&A / conference Q&A panel), and is not automatically triggered in the real-time translation chain such as PENDING preview, First Call or Last Call, so as not to increase the real-time translation delay and not to conflict with the ironclad rule of "the original text arrives before the translation".

[0377] Step S907, RAG Recall Fragment Front-end Visualization: The computer device displays the recall fragment triggered by the current sentence in the panel (in the form of a pop-up or drawer). Users can click to view metadata such as the document to which the fragment belongs, score, and original text location, and can manually copy the fragment content when proofreading the translation.

[0378] In this embodiment, the method reduces similarity recall noise by extracting query keywords using dedicated prompt words, acquires domain knowledge through Top-K recall using cosine similarity, and makes the recall results controllable and credible for users through front-end visualization, providing on-demand knowledge base background support for translation proofreading and conference Q&A.

[0379] Example 10: A Terminology Management Method Based on Two-Way Rules, Soft Hints, and UI Tags

[0380] This embodiment provides a terminology management method executed by the computer device without modifying the LLM output. This method can be performed by the "terminology management module" in the computer device.

[0381] Step S1001, Three-category entry model: The computer device maintains a "terminology list" in memory, which consists of several "entries". Each entry contains the following fields: source text, target text, entry type, group, activation flag, and remarks. The entry type enumeration includes three types: ① force-to (must be translated to), that is, the target text must be output when the source text appears; ② forbid (never translate to), that is, the source text is never allowed to be translated into the target text (used to exclude common mistranslations); ③ keep (keep the original text), that is, the source text is directly copied into the translation when it appears (used for brand names, codes, etc.).

[0382] Step S1002, Bidirectional Validation Default: The computer device applies both src→tgt and tgt→src to each entry simultaneously to prevent users from repeatedly entering the same term in both directions. Specifically, for force-to type entries, the forward validation is "the translation must contain the target text when the source text appears in the original text," and the reverse validation is "the translation must contain the source text when the target text appears in the original text."

[0383] Step S1003, Group Activation Mechanism: The computer device allows users to divide items into several groups according to the "meeting topic" (such as "Product A Terminology", "Customer B Jargon", "Technical Vocabulary"). Before each meeting starts, users can select the active group set for this meeting on the UI (multiple selections are allowed); items outside the active set are not effective for the current session. When the active set is persisted, it simultaneously records the "last used set (last)" and "default set (default)" labels to facilitate users' quick reuse of history.

[0384] Step S1004, Soft prompt injection: Before each LLM call translation, the computer device groups all entries in the activation set by type into a term instruction block and injects it into the system prompt (not user prompt) of the LLM call translation, in the format of "must be translated as: {src1}→{tgt1};{src2}→{tgt2};...; never translated as: {srcA}↛{tgtA};...; retain the original text: {src},{src},...".

[0385] Step S1005, Translation Verification and UI Marking: After the Last Call (translate) returns the translation, the computer device scans each active term entry in the translation to perform violation detection. A force-to violation occurs only if the source text appears in the original text but the target text does not; a forbid violation occurs only if both the source text and the target text appear in the original text; a keep violation occurs only if the source text appears in the original text but the translation does not copy it verbatim. Detected violations are recorded in a structured format as "Violation Type + Affected Entries" in the segment context, and the violation location is marked in the front-end caption bubble with an underline / tooltip.

[0386] Step S1006, Do not automatically rewrite LLM output: The computer device intentionally does not perform string replacement or post-processing rewriting on violating entries. The reason is that simple string replacement will destroy the naturalness of the translation syntax (such as tense, singular / plural, inflectional morphology); and some LLMs will output rigid translations after being hard-constrained by terminology. The combination of "soft hints + validation + UI markers" allows users to manually judge whether correction is needed during review, which respects the naturalness of the translation more than hard rewriting.

[0387] Step S1007, CSV Batch Management: The computer device allows users to import / export terminology entries in batches using CSV files. Each row includes fields such as source text, target text, entry type, group name, activation flag, and remarks, facilitating the migration of terminology lists across multiple devices / scenes.

[0388] In this embodiment, the method solves the terminology consistency problem in professional field translation in a four-in-one approach of "two-way rules + soft prompts + verification + UI tags". Compared with "users manually pasting terms to prompts each time", it reduces operation costs; compared with "post-processing string replacement", it preserves the naturalness of the translation; and compared with "no feedback", it makes violations visible.

[0389] Example 11: Local Area Network API Sharing Method Based on Multi-Source Fusion Proxy

[0390] This embodiment provides a local area network (LAN) API sharing method executed by the computer device to expose an OpenAI-compatible entry point. The method can be implemented by a "LANAPI sharing module" within the computer device.

[0391] Step S1101, OpenAI Compatibility Entry Exposure: The computer device mounts a set of routes with the same URL paths and request body schemas as OpenAIChatCompletionsAPI (` / v1 / chat / completions`) and EmbeddingsAPI (` / v1 / embeddings`) in the FastAPI application of the main process, so that any third-party client that already supports the OpenAI compatibility protocol can access the local machine without additional development.

[0392] Step S1102, Multi-backend engine aggregation: The computer device allows multiple LLM providers (OpenAI, AnthropicClaude, Ollama, Alibaba Cloud Bailian, local LLM, etc.) configured on the local machine to be aggregated under the same entry point in the form of "virtual model name" mapping; the user specifies the virtual model name in the `model` field of the request body, and the computer device routes to the corresponding backend adapter according to the preset mapping, and returns the backend response after converting it back to the OpenAI standard schema.

[0393] Step S1103, Token authentication and revocation: The computer device maintains several self-signed APITokens in the local business database. Each token is associated with a readable name, creation time, last usage time, and revocation flag. All external interface calls must carry a valid token in the HTTPAuthorizationHeader. Users can revoke any token with one click on the UI, making it immediately invalid.

[0394] Step S1104, call statistics record: The computer device records all external interface calls in an independent call statistics table according to the dimensions of "call Token name, backend engine category to which it is routed, call date, usage (character or Token estimated value), and failure count"; error information in external responses is desensitized according to preset keyword (APIKey segment, email, mobile phone number, etc.) rules to avoid sensitive information being displayed.

[0395] Step S1105, Statistics pop-up: The computer device provides a statistical view of the call in the form of an independent pop-up on the UI, according to dimensions such as "Token / Engine / Model / Time Period", so as to facilitate users to audit the sharing and usage.

[0396] In this embodiment, the method achieves the capability of "configuring one host and sharing with multiple devices" at minimal cost—no third-party agent software required, no APIKey distribution required for each device, no modification to third-party clients required, and it also has de-identification and auditing capabilities to reduce the risk of key leakage.

[0397] Example 12: A Real-Time Translation End-to-End Delay Observability Method Based on Four Timestamps and Ironclad Self-Check SQL Paradigm

[0398] This embodiment provides a latency decomposition and ironclad self-checking method covering the entire audio→ASR→LLM→rendering chain, executed by the computer device. This method can be performed by the "Unified Log Bus Module" in the computer device.

[0399] Step S1201, Independent Log Library and 10 Types of Indexes: The computer device maintains a "unified log library" in `%APPDATA% / CrossMeet / logs.db` as an independent SQLite file, separate from the business database (crossmeet.db), to avoid high-frequency log writes affecting business transactions. The log library contains a single table `logs`, with fields including timestamp, category (a closed enumeration: system / install / engine / audio / asr / llm / tts / session / ui / rag, a total of 10 categories), component name, level, source process, session ID, trace ID, message, and payload (JSON string).

[0400] Step S1202, Hierarchical Rolling TTL: The log table is a rolling window with hierarchical TTL by level - the computer device cleans up old logs by level every 5 minutes as a background task (debug level is retained for 1 hour, info level for 6 hours, and warn and error levels for 7 days) to avoid the log library from expanding indefinitely after long-term operation.

[0401] Step S1203, Pydantic strongly typed payloadschema: The computer device defines 10 Pydantic models as payloadschemas for each of the 10 categories (e.g., `AsrPayload`, `LlmPayload`, `SessionPayload`). Each schema declares the field types, default values, and ranges that logs may carry under that category. All logs must pass Pydantic validation before being written to avoid spelling errors or type drift in payload fields.

[0402] Step S1204, stdlibbridge: The computer device maps all logs from the Python standard `logging` module to the corresponding category by the logger name prefix (e.g., `adapters.llm.*→llm`, `core.session.*→session`), so that any module can also be automatically logged to the log library directly via `logger.info(...)`.

[0403] Step S1205, Four-timestamp full-link tracking: The computer device embeds four timestamps for each sentence event traversing the system's key nodes, namely:

[0404] (i) `asr_yielded`: The moment the ASR adapter outputs partial or final text, an event is recorded by the `core.session.pipeline` component;

[0405] (ii) `event_emit`: The event is recorded by the `event_emit` component just before the sentence assembly module delivers the event to the WebSocket sending channel;

[0406] (iii) `ws_received`: The moment the front-end WebSocketonMessage callback entry point is reached, the `client_trace` component writes back the record from the front-end to the back-end via HTTP.

[0407] (iv) `seg_rendered`: The `client_trace` component writes back to the backend record the instant the DOM pixels have been updated after the frontend Vue `nextTick()`.

[0408] Step S1206, Ironclad Self-Check SQL Norms: The computer device has pre-configured SQL norms in its documentation and diagnostic tools that can be directly executed on the log database, and verifies the two architectural ironclad rules respectively:

[0409] (i) Ironclad Rule #1 self-check (original text arrives before translation): Using the event_emit log with the same sentence_id as the unit, group by detail.has_src / detail.has_tgt fields to find the earliest timestamp. If the earliest has_tgt timestamp of any sentence is earlier than the earliest has_src timestamp, it is considered to violate Ironclad Rule #1.

[0410] (ii) Ironclad Rule #2 self-check (LLM does not block ASR): For logs with component='asr_yielded', calculate the adjacent intervals according to the time series. If a window with a value much higher than partial_interval_s appears locally and there are LLM call logs in the same period, it is considered that Ironclad Rule #2 may be violated.

[0411] Step S1207, One-click diagnostic script: The computer device is equipped with a command-line diagnostic script (such as `python-mscripts.analyze_latency--since5\minutes`), which automatically reconstructs the four timestamp links of any sentence from the log library, outputs a readable "end-to-end delay decomposition table", and highlights sentences that violate the ironclad rule.

[0412] Step S1208, Front-end latency indicator: The computer device displays the end-to-end latency (in milliseconds) of the current session in real time in the front-end meeting mode status bar. The latency is calculated as the median of `seg_rendered−asr_yielded` of the most recent N sentences, allowing users to directly perceive the latency bottleneck on the UI.

[0413] In this embodiment, the method solves the long-standing problem of "difficulty in locating latency bottlenecks and difficulty in detecting violations of ironclad rules" in real-time translation software by integrating "independent log library + strongly typed payload + four timestamp tracking + SQL paradigm self-check + command line diagnosis + front-end indicator". This allows any architectural change (adding an engine, adjusting the prompt, modifying sentence segmentation rules, etc.) to be verified by ironclad SQL within 5 minutes without introducing regressions.

[0414] Example 13: A Method for Multi-Level Authorization and Device Fingerprint Binding of Desktop Software Based on First-Time Activation and Short-Term Credential Heartbeat

[0415] This embodiment provides a multi-tiered distribution licensing method for desktop real-time translation software, executed by the computer device. As shown in the figure, this method can be performed by the licensing dock module in the computer device.

[0416] The proposed method addresses the shortcomings of the following three types of existing desktop software licensing schemes: ① Purely offline signature licensefile schemes cannot be revoked, cannot be subscribed to by time, and have zero user dissemination costs; ② Purely online verification schemes have zero tolerance for network jitter and frequently report usage behavior to the authorization server, which violates privacy; ③ Account-based SaaS schemes must be equipped with an account system and force users to log in and be redirected when they switch devices.

[0417] Step S1301, Two-layer voucher design:

[0418] The first layer is the long-life license_key, which is a readable string consisting of "prefix + plan abbreviation + three random short codes" (e.g., `meetai-friend-1y-A3B8-K9DC-X2NF`). The prefix is ​​a preset fixed string to visually distinguish it from other authorized platform credentials. The three random short codes are taken from a predefined character set and exclude easily confused characters such as 0 / O / 1 / I / L. Each segment has 4 characters, for a total of 12 characters of random entropy. The license_key only serves as an activation credential and does not carry authorization information or participate in any client signature verification.

[0419] The second layer is the short-lived entitlement_jwt, which is obtained by the computer device by calling the authorization server in exchange for the license_key. It is signed using Edwards curve EdDSA (Ed25519), and the payload is fixed to contain twelve fields: iss, sub, dids (device fingerprint hash string array), plan, feat array, iat, exp, lic_exp, grace_until, device_quota, ki (keyindex, public key rotation index), and protocol version number v; the protocol version number v is frozen to 2 after the first batch of distributions.

[0420] Furthermore, the dids field is a string array, where each element is a valid device fingerprint (a 32-character hexadecimal SHA-256 hash calculated in step S1304) that is legally bound to the license_key. When the client verifies the signature, it is considered that the local machine has been bound as long as any element of `computeMachineId()∈dids` matches, so that a long-life license_key can be legally bound to multiple devices (subject to the device quota limit).

[0421] This design has three advantages over "one license bound to only one device": ① Authorized users can use the license synchronously across the primary, backup, and test machines without the need for "activation-release-reactivation" operations; ② When the authorization server stops issuing new tokens in the future, the multi-device lifetime tokens already written to disk can still work on the bound devices, which is more user-friendly; ③ Attackers copying license.dat to an unbound fourth machine will still be rejected by the client due to `computeMachineId()∉dids`, so security is not compromised.

[0422] Furthermore, the ki field is an integer key index, and the client maintains a public key ring in the source code (e.g., `ED25519_PUBLIC_KEYS:Record`).<number,string> The signature verification is performed by looking up the corresponding public key using `ki`. When the authorization server needs to rotate the signing key in emergency scenarios such as private key leakage, the server only needs to switch the issuing key to `ki=new index`. The client already has the new public key embedded, so the transition can be smooth without bumping the protocol version number v or forcing all clients to upgrade. During the transition period when the old and new versions coexist (the default maximum wait time is 21 days, overriding exp+grace_until), the old `ki` credential will expire naturally, and the old public key can be removed from the new version client to complete the rotation.

[0423] Step S1302, initial stimulation + heartbeat + grace period three-state combination:

[0424] When the computer device first starts up and there is no license_key file locally, it automatically calls the "trial registration" interface of the authorization server to exchange its local fingerprint hash for a trial entitlement_jwt and writes it to disk. On each subsequent startup, it prioritizes reading the written entitlement_jwt for local verification. When the entitlement_jwt is less than one heartbeat cycle threshold (default 24 hours) away from its exp field, the computer device triggers a "heartbeat" call to exchange the locally written entitlement_jwt for a new entitlement_jwt with the same license_id. If the user is offline and the heartbeat cannot be completed, the computer device allows the user to continue using the device until the offline grace period indicated by the grace_until field of the entitlement_jwt expires (default 14 days after exp). During the offline grace period, the computer device attempts a heartbeat each time it starts up, but failure is not fatal. After the offline grace period expires, the computer device enters a "locked, network required" state.

[0425] Step S1303, Capability Whitelist (feat array): The feat field in the entitlement_jwt payload is a string array listing the specific functional capabilities unlocked in this license (e.g., `interpret`, `vmic`, `rag`, `lan_share`, `export`, `glossary_unlimited`, `quick_phrases_unlimited`). All feature gating in the client code queries the feat array, not the plan field. This design allows the licensee server to flexibly grant different capability combinations to different purchasers within the same plan (e.g., an internal test license can enable only `interpret` and not `vmic`), and adding new plans on the server side does not require modification of the client code.

[0426] Step S1304, Composite machine fingerprint algorithm:

[0427] The computer device generates its own fingerprint without relying on a single hardware identifier. Instead, it combines three types of signals: ① a global machine identifier in the operating system registry (Windows registry `HKLM\SOFTWARE\Microsoft\Cryptography\MachineGuid`); ② a CPU manufacturer and model string obtained through a system management infrastructure query command; and ③ a motherboard serial number string obtained through a system management infrastructure query command. These three signals are then concatenated in lexicographical order, subjected to a secure hash algorithm (SHA-256), and the first 32 characters are truncated into hexadecimal to form the final fingerprint. This design offers the following resistance to erosion: replacing the hard drive does not affect the fingerprint; replacing the memory does not affect the fingerprint; reinstalling the operating system triggers a fingerprint change (identifying the device as new, consistent with user intuition); and it can distinguish between devices in virtual machine cloning scenarios due to differences in motherboard serial numbers or processor models.

[0428] Step S1305, Operating system level credential protection storage:

[0429] The computer device encrypts the entitlement_jwt using an operating system-level secure storage application interface (DPAPI on Windows, Keychain on macOS, and libsecret on Linux) and stores it in the license.dat file in the user's configuration directory. This encryption method binds the key to the current operating system user's master key, ensuring that even if an attacker copies the license.dat file to another computer or another operating system user, the credentials cannot be decrypted. This design eliminates the low-cost piracy path of "using a single license across multiple machines by sharing the license.dat file."

[0430] Step S1306, Apply archive integrity self-verification:

[0431] During the packaging phase, the computer device calculates the SHA-256 hash of the application archive (Electronasar) and writes it to the asar_hash.txt file, which is distributed with the archive but located outside the archive. Upon startup (in packaged mode), the computer device calls the authorization base module to verify whether the actual hash of the current archive file matches the expected hash in asar_hash.txt. If either does not match, the application is considered tampered with, the authorization status is directly set to "integrity failed," and the user is prevented from accessing the main functionality. This design prevents users from bypassing authorization by unpacking the archive, modifying the authorization check code, or repackaging the archive.

[0432] Step S1307, Clock rollback defense:

[0433] Upon each successful heartbeat or initial activation, the computer device persists the IAT timestamp recorded by the server when issuing the entitlement_jwt as a "known time lower bound" to the local DPAPI-protected clock_guard.dat file. Upon subsequent startups, the computer device compares its local system time with this known time lower bound. If the local time is significantly earlier than the known time lower bound (the default threshold is 1 hour), it is considered that the user is attempting to bypass the expiration judgment of the entitlement_jwt by rolling back the system clock. The computer device will immediately trigger a forced heartbeat to reset the known time lower bound. If the heartbeat fails due to network issues, the authorization status is locked.

[0434] Step S1308, Multi-point signature verification anti-patch:

[0435] The computer device does not verify the entitlement_jwt signature only once during main process startup, but also performs signature verification again at at least four independent code paths: ① after the archive integrity self-verification during application startup; ② before the rendering process loads the main route; ③ at the entry point of each high-value functional capability (simultaneous interpretation mode startup, virtual microphone sending startup, knowledge base retrieval call startup, LAN sharing startup, etc.), the feat array in entitlement_jwt is verified locally again; ④ when the background heartbeat callback receives a new entitlement_jwt, it is immediately verified before being written to disk. This design prevents attackers from compromising the entire authorization gating system by modifying a single function (such as bypassing startup signature verification).

[0436] Step S1309, Equipment Quota and Migration Frequency Limits:

[0437] The authorization server maintains an active device table for each license_key. Each record includes a local fingerprint hash prefix, first activation time, most recent heartbeat time, and release time. The authorization server imposes a limit on the number of active devices for the same license_key (device_quota, determined by the plan, e.g., 2 for individual subscriptions and 5 for internal lifetime subscriptions). Users can release a device by calling the "Release Device" interface, but the release operation is limited in frequency (maximum 3 times within 30 days by default) to prevent "borrowing a device for a period of time and then releasing it" exploitation. The heartbeat interface returns a specific error code when it detects that the device corresponding to the entitlement_jwt has been released, triggering the client to immediately lock the device.

[0438] Step S1310, Initially confirm the user agreement and implement trial anti-fraud measures:

[0439] During the NSIS installation phase, the computer device presents the user with a "30-day free trial + partial lock of high-value features after the trial + privacy policy statement (machine fingerprint hash reporting but not the content used) + intellectual property and anti-reverse engineering clauses" in the form of a local end-user license agreement (EULA). The user must check "I agree" to continue the installation. When the trial registration interface is called for the first time, the authorization server records the trial in the trial_registrations table with machine_id_hash as the unique constraint, so that the same machine cannot bypass the 30-day limit by uninstalling and reinstalling. At the same time, the authorization server performs risk scoring on the registration of different machines by the same IP range multiple times within 24 hours. If the threshold is exceeded, the trial period is automatically shortened or the registration is rejected.

[0440] Step S1311, the email-based automatic approval channel for extension appeals:

[0441] When a user triggers the "This machine trial has already been used" error for a legitimate reason (such as replacing hardware or reinstalling the system), the computer device provides an extension appeal entry on the expiration interface. After the user fills in their email address and the reason, the authorization server queries the history of the same email address: if the email address has never been automatically approved before, the first automatic extension of 7 days will be approved and a new entitlement_jwt will be returned immediately; otherwise, the appeal will be transferred to the manual review queue. This design strikes a balance between anti-fraud and a user-friendly experience.

[0442] Step S1312: The protocol version number is permanently frozen while the public key can be rotated.

[0443] Before the computer equipment is officially distributed for the first time, the following contract items will be considered permanently frozen: license_key character set and grouping format, entitlement_jwt payload field name and order (including dids array type / kikeyindex field type / v=2 integer value, signature algorithm (Ed25519EdDSA), default offline grace period (14 days), relative path of the authorization server interface and request / response field name (including the V5 addition `POST / api / v1 / revocation_list`), error code string value (such as trial_exhausted, device_quota_reached, invalid_signature, etc.).

[0444] Protocol version number v is frozen at 2. Any renaming of a field (e.g., `dids` → `device_ids`), change of field type (e.g., `dids:string[]` → `dids:{hash:string,label:string}[]`), field deletion, changes to the `iss` string, changes to the Lifetimeplan keyword definition, changes to the RL`signature` algorithm, or extension of the `MAX_OFFLINE_DAYS` value (shortening is allowed) requires bumping the protocol version number to v=3 and sending a new client. Permissible changes within v=2 include: adding a plan name (without violating lifetime detection rules); adding a featflag; adding optional fields; adding a new ki item to `ED25519_PUBLIC_KEYS` (public key rotation).

[0445] The Ed25519 public key embedded in the client is selected by looking up the table through the `ki` field index. In emergency scenarios where the private key is leaked, rotation can be completed simply by the server changing the ki value used for issuance, without breaking the protocol version number v or requiring all users to upgrade the client.

[0446] Step S1313, Front-end activation credential UI:

[0447] The computer device's rendering end uses a global dialog box (rather than a full-screen routing page) to carry out the activation process. The dialog box is triggered by the global status controller from any location in the application (the "Activate Authorization" button on the About page, the authorization status badge in the sidebar, the "Activate Now" button on the paid feature gate card, the trial expiration notification click, etc.). When the user enters the license_key, the dialog box formats the input in real time (automatically converting to uppercase and automatically grouping by 4-4-4) and identifies the authorization type based on "prefix + plan abbreviation" and displays it in real time. After successful activation, the dialog box presents the authorization type, issuance date, expiration date, number of device slots, list of unlocked features and capabilities, and a copyable authorization code in the form of a certificate, and automatically closes the dialog box after 5 seconds to return the user to the application context before activation, avoiding disruption to the user's original workflow.

[0448] Step S1314: Replace scattered prompts with unified front-end payment access control cards:

[0449] For each function in the aforementioned capability whitelist, the computer device uses a uniform payment gate component (available in mini, card, and full-page sizes) at its entry control. This payment gate component is rendered with an emerald accent color and visual anchor points (a solid round stamp in wax seal style, a semi-transparent paper-textured background, and a large serif italic title), ensuring visual consistency between the payment prompt and the "Activate Authorization" call button. Users can directly access the activation dialog box with a single click from any locked function entry point. This design addresses the common user experience flaw of "users being prompted for Pro but unable to find the activation entry point."

[0450] Step S1315, Sidebar Authorization Status Badge:

[0451] The computer device displays the current authorization status as a constantly visible badge at the bottom of the sidebar of the main interface. The badge color is mapped according to the following rules: green for trial with more than 7 days remaining; yellow for trial with 7 to 3 days remaining; red for trial with 3 days remaining or within the offline grace period; purple for lifetime plans; blue for activated time-based subscriptions; and red for expired, revoked, or integrity-failed subscriptions, which are entirely unclickable and serve only as a notification. This design allows users to know the authorization status at any time without opening any menus.

[0452] Step S1316, Actively Cancel List RL:

[0453] In addition to the passive revocation path of "heartbeat check device table" in step S1309, this embodiment introduces a revocation list mechanism periodically issued by the authorization server, which takes effect in the following scenarios: ① The entitlement_jwt that has been disk-written but revoked by the server can still be verified by the client before the exp expiration; ② An attacker deliberately avoids heartbeat calls to extend the lifetime of the revoked credential. Specifically:

[0454] S1316a (Revocation list data structure): The authorization server maintains a list of all currently revoked license_ids (all rows in the `licenses` table of the database where `status='revoked'`), and encapsulates it in the following closed mode:

[0455] ```json

[0456] {

[0457] "count": <integer>,

[0458] "ts": <Unix integer seconds, the current signing time>,

[0459] "rlist": ["100", "256", "1234",...],

[0460] "ki": <integer, the key index used for signing>,

[0461] "signature": "<Base64-encoded Ed25519 signature>"

[0462] }

[0463] ```

[0464] Where `signature = Ed25519_sign(privKey[ki], canonicalJSON({count, ki, rlist, ts}))`; The `canonicalJSON` function is defined as: The fields are sorted alphabetically (`count` / `ki` / `rlist` / `ts`), the elements in `rlist` are sorted lexicographically as strings, there is no space separator, and it is UTF-8 encoded.

[0465] S1316b (Endpoint exposure): The authorization server exposes the revocation list at the `POST / api / v1 / revocation_list` endpoint without authentication (public data).

[0466] S1316c (Client-side fetching and caching): The computer device invokes the endpoint at the following two times: ① during the application startup bootstrapLicense process; ② during each heartbeat scheduling tryHeartbeatIfDue process. After each successful fetch, the computer device uses operating system-level secure storage (DPAPI / Keychain / libsecret) to encrypt and write `{ts,rlist,ki,signature,last_update_ms}` to the `revocation.dat` file in the user configuration directory, where `last_update_ms` is the client's local timestamp used to calculate "how long has it been since the last successful fetch of the revocation list".

[0467] S1316d (Signature Verification and Revocation Check): After the computer device retrieves the revocation list, it immediately checks the following process: ① `pubKey=ED25519_PUBLIC_KEYS[ki]`, and selects the public key by looking up `ki` in the table; ② Verify the signature using `Ed25519_verify(pubKey,canonicalJSON({count,ki,rlist,ts}),signature)`; ③ If the signature fails, discard the current response and fallback the previous cache; ④ If the signature succeeds, check `currentEntitlement.sub∈rlist`, and if it is, atomically set the client's authorization status to "tampered" and clear the license.dat file.

[0468] S1316e (Offline Grace Timeout Forced Network Connection): Each time the computer device starts, in addition to the above checks, the following is executed: `if(now-last_update_ms>MAX_OFFLINE_DAYS*86400*1000)→status is tampered`, where `MAX_OFFLINE_DAYS` is synchronized with the offline grace period `grace_until`, which defaults to 14 days. This mechanism prevents attackers who deliberately circumvent heartbeat calls from extending the life of revoked credentials beyond 14 days—they must connect to the network and obtain a new RL whenever they start the application, otherwise it will be locked.

[0469] S1316f (Revocation Cascade Effect): When the authorization server executes the `license:revoke` command, in addition to setting the `status` row of the `licenses` table to `revoked`, it also releases all active devices associated with that license_key (the `released_at` row of the `devices` table is set to the current time). The client thus enters a double defense: ① The server returns a `license_revoked` error code on the next heartbeat, triggering an immediate lock; ② Even if the client stops heartbeating, it will still be locked the next time it initiates a pull RL due to `ent.sub∈rlist`.

[0470] Step S1317, Lifetime Immortality Rules:

[0471] In the `classifyState` function, the computer device directly returns the `kind:'lifetime'` status for `entitlement_jwt` containing any of the following keywords in the `plan` field, skipping the checks on the three time fields: `exp`, `grace_until`, and `lic_exp`.

[0472] S1317a (Keyword Enumeration): The lifetime keyword includes, but is not limited to, `'pro-lifetime'` and `'internal-life'`, and uses "containing the substring 'life'" as a wildcard rule, so that future extended names such as `'friend-lifetime'` and `'early-lifetime'` will automatically conform to the rule without requiring client protocol upgrades.

[0473] S1317b (Meaning of Immortality): The core meaning of this design is that once a lifetime user obtains a valid entitlement_jwt and writes it to disk, they can continue to use all the unlocking functions of the computer device even if the authorization server is permanently shut down. This design has the following two implications in enterprise software: ① It promises lifetime availability to lifetime plan purchasers, ensuring that the service provider will not break its promise due to business adjustments; ② The software distributor strategically possesses a graceful sunset channel—if maintenance ceases in the future, the sold lifetime tokens will continue to function.

[0474] S1317c (Immortality does not equal exemption from authorization): Lifetime users are still subject to the dual constraints of `dids` verification (step S1304) and revocation list RL check (step S1316)—copying license.dat to an unbound machine will still be rejected because `computeMachineId()∉dids`; the license_id will still be locked if it enters the revocation list. Only "server off → client still usable" maintains long-term use; it is neither exemption from authorization nor exemption from compliance.

[0475] In this embodiment, the method employs a 17-in-1 approach—including dual-layer credentials, first activation and heartbeat, offline grace period, capability whitelist, composite machine fingerprint, operating system-level credential protection, application archive integrity self-verification, clock rollback defense, multi-point signature verification, device quota migration frequency limit, email appeal extension, protocol version permanent freeze, global activation dialog box, unified payment gating, sidebar badges, revocation list (RL), and Lifetime immortality rules—to provide desktop real-time translation software with a modern authorization platform that allows subscription, revocation, tiered access, team sharing, elegant sunset, and independence from account systems. This contrasts with the common "pure offline signature" approach in existing desktop software. Compared to the "name license" or "pure online account" solutions, this embodiment simultaneously achieves multiple technical benefits, including offline friendliness (a 14-day grace period is sufficient to cover business travel and air travel scenarios), privacy friendliness (only heartbeats are reported for machine fingerprint hashes, not usage behavior or translated content), strong anti-cracking capabilities (DPAPI encryption + device binding + integrity verification + clock rollback defense + proactive revocation list), flexible distribution (feat whitelist + plan enumeration + multiple device bindings can be arbitrarily combined), good user experience (the UI never leaves the current context), and elegant operation and maintenance (public key rotation does not break the protocol + lifetime user sunset compatibility).

[0476] Example 14: A Real-Time Two-Way Simultaneous Interpretation Device Based on Multi-Source Audio and Multi-Model Collaboration

[0477] This embodiment provides a real-time two-way simultaneous interpretation device based on multi-source audio and multi-model collaboration. The device can be configured in a desktop computer device and is used to execute any of the embodiments of the aforementioned method embodiments.

[0478] The device includes: an audio acquisition module, a translation strategy module, a sentence assembly module, a finalization module, an automatic reading module, an engine management module, an installation management module, a session protection module, a knowledge base management module, a terminology management module, a LANAPI sharing module, a unified log bus module, and an authorization base module.

[0479] The audio acquisition module is used to capture audio frames from user-selected audio sources (including at least one of microphone devices, system playback loopback devices, and specified process audio capture channels based on the operating system process audio loopback API) and encode them into standard PCM format.

[0480] The translation strategy module is used to call the single decision function for each arriving frame of ASR text and output the immutable decision object. The translation strategy module further includes: a language fusion parsing submodule, used to output the final source language according to a preset priority; and a decision scheduling submodule, used to generate the decision object based on dual-source / single-source flags and speaker numbers.

[0481] The sentence assembly module is used to maintain a sentence assembler for each audio source, and atomically transfers each sentence between three states based on the decision object and the multiple encapsulation predicate. The sentence assembly module further includes: a sentence allocation submodule, a pending state receiving submodule, an overlap stripping submodule, an encapsulation condition judgment submodule, an event sending submodule, and a preview anti-contamination submodule.

[0482] The finalization module is used to asynchronously call the large language model adapter for correction and translation after the sentence enters the "sealed" state, and ensures the final state is closed according to a predefined degradation strategy. The finalization module further includes: an ASR error correction call submodule (executes First Call / asr_correct), a final translation call submodule (executes Last Call / translate), a call cancellation and backtracking merging submodule (handles the two call cancellation semantics during backtracking merging), and an atomic state promotion submodule (handles the two-step push of "original text backfilling first, translation text backfilling later" or vice versa).

[0483] The automatic reading module is used to send the target text of "finalized" sentences to the speech synthesis adapter according to the three reading modes, and route it to the corresponding output device according to the mode. The automatic reading module further includes: a mode state management submodule, a new sentence detection submodule, a synthesis request submodule, an output device routing submodule, a virtual device self-healing parsing submodule, and an echo suppression submodule.

[0484] The engine management module is used to host the local inference engine in an independent subprocess and maintain its lifecycle. The engine management module further includes: a subprocess hosting submodule, a binary frame IPC submodule, a 5-role lifecycle machine submodule, a signature fast path submodule, an overloaded sequence number guard submodule, and a crash self-detection and diagnosis collection submodule.

[0485] The installation management module performs a minimal core installation upon first startup, executes a ten-stage standard installation pipeline when the user enables the local engine, and performs a dependency alignment process on each subsequent startup. The installation management module further includes: a dual-GPU detection submodule, a hardware gating submodule, a memory budget pre-check submodule, a ten-stage pipeline submodule, an installation manifest submodule, a startup-time basic dependency alignment submodule, a startup-time overlay dependency alignment submodule, and an overlaylockfile auditing submodule (performs strict pinned form validation and torch family prohibited item validation on the `overlay_lockfile` field of each overlay engine, with the results logged in a unified log library at the warning level).

[0486] The session protection module is used to ensure that engine switching does not interrupt the session through session epochs and reference counting. The session protection module further includes: a session epoch submodule, an epoch discarding and emptying submodule, a composer borrowing-returning submodule, and a process audio capture submodule.

[0487] The knowledge base management module is used to implement the on-demand knowledge base retrieval (RAG) described in Embodiment 9. The knowledge base management module further includes: a multi-knowledge base container submodule, an OpenAI-compatible embedding adaptation submodule, a query keyword extraction submodule, an embedding similarity recall submodule, a recall result return submodule, an on-demand trigger control submodule, and a recall fragment front-end visualization submodule.

[0488] The terminology management module is used to implement the bidirectional rules and soft hints + UI tags for the terminology described in Embodiment 10. The terminology management module further includes: a three-category entry model submodule, a bidirectional verification submodule, a group activation submodule, a soft hint injection submodule, a translation verification and UI tagging submodule, and a CSV batch management submodule.

[0489] The LAN API sharing module is used to implement the external OpenAI compatible entry point described in Example 11. The LAN API sharing module further includes: an OpenAI compatible entry point submodule, a multi-backend engine aggregation submodule, a Token authentication and revocation submodule, a source IP and prompt anonymization submodule, and a statistics pop-up window submodule.

[0490] The unified log bus module is used to implement the end-to-end latency observability described in Example 12. The unified log bus module further includes: an independent log library submodule (10 types of indexes + hierarchical rolling TTL), a Pydantic strongly typed payload submodule, a stdlibbridge submodule, a four-timestamped data entry submodule, a self-checking SQL normalization submodule, a command-line diagnostic script submodule, and a front-end latency indicator submodule.

[0491] The authorization base module is used to implement the desktop software multi-level authorization method of first activation + heartbeat and short-lived certificate as described in Embodiment Thirteen. The authorization base module further includes: a two-layer credential submodule (long-life license_key issuance / short-life entitlement_jwtv2 issuance and renewal / dids multi-device binding), a machine fingerprint submodule (MachineGuid + processor + motherboard serial number combined SHA-256), an EdDSA verification submodule (embedded public key ring (lookup table by ki index) + jose verification + protocol version number guard v=2), an operating system-level credential protection submodule (DPAPI / Keychain / libsecret encrypted license.dat), an application archive integrity self-verification submodule (asarSHA-256 verification), a clock rollback defense submodule (last_known_time persistence and comparison), a multi-point verification submodule (startup / rendering / function entry / heartbeat callback four points), a heartbeat scheduling submodule (exp 24-hour trigger + 14-day offline grace period + forced immediate heartbeat), a device quota and migration frequency limit submodule (device_quota + 30-day 3-time release limit), and a single automatic mailbox function. The module includes: Approval Appeal Submodule, Front-end Activation Dialog Submodule (globally triggerable dialog + real-time input formatting + credential-based success panel + 5-second automatic close), Front-end Payment Gated Card Submodule (unified visual payment prompts in mini / card / full-page sizes), Sidebar Authorization Status Badge Submodule (constantly visible + color-mapped six states), Revocation List RL Retrieval and Caching Submodule (retrieves `POST / api / v1 / revocation_list` + Ed25519 signature verification canonicalJSON + DPAPI encrypted cache `revocation.dat` + `last_update_ms` compared with 14-day MAX_OFFLINE_DAYS, forcing network connection), and Lifetime Perpetual Life Rule Submodule (`classifyState` skips exp / grace_until / lic_exp checks and returns `kind:'lifetime'` for tokens in the plan containing the substrings `pro-lifetime` / `internal-life` / `life`.

[0492] Example 15: A computer device

[0493] See Figure 8 The computer device includes a processor 502, a memory, and a communication interface 505 connected via a communication bus 501. The memory may include a storage medium 503 and internal memory 504. The storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it causes the processor 502 to perform the steps described in any embodiment of the method of the present invention.

[0494] The processor 502 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0495] The communication interface 505 is used to connect a microphone and a speaker; in addition, in this embodiment, the computer device should be equipped with an NVIDIA GPU (minimum 8GB of video memory) to support the local inference engine; when the computer device is not equipped with a GPU, it can still run using only the API engine.

[0496] Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art and related fields based on the embodiments of the present invention without inventive effort should fall within the scope of protection of the present invention.< / url> < / package> < / version> < / package>

Claims

1. A real-time bidirectional simultaneous interpretation method based on multi-source audio and multi-model collaboration, characterized in that, This method, when applied to computer devices, includes the following steps: The computer device captures audio frames from an audio source selected by the user and encodes them in a standard Pulse Code Modulation (PCM) format; the audio source includes at least one of the following: the microphone device of the computer device, the system playback loopback device of the computer device, and a specified process audio capture channel based on the operating system process audio loopback application programming interface. The computer device calls the automatic speech recognition adapter to generate text from the audio frame, and calls a single decision function for each frame of text; the single decision function integrates the acoustic language recognition result and the text language recognition result, and outputs an immutable decision object containing four fields: source language, target language, speaker number, and display panel affiliation. The computer device maintains a sentence assembler in memory for each audio source and assigns a globally monotonically increasing sentence identifier to each sentence; the sentence assembler atomically transfers each sentence between the three states of "pending processing - sealed - finalized" based on the decision object and the multiple sealed predicate; After each sentence enters the "sealed" state, the computer device asynchronously and independently calls the large language model twice. The first call outputs the corrected original text in the same language with ASR error correction prompts, and the second call outputs the final target language translation with translation prompts. Each of the two calls has an independent timeout and degradation path. The two calls are executed serially, and after both return, an atomic final state update fills the front-end subtitle with the corrected original text in the same language and the final target language translation. The computer device automatically calls the speech synthesis adapter to synthesize audio for each sentence that enters the "final state" according to the user's preset three reading modes: "off / local speaker / virtual microphone". The audio is then routed to the output device corresponding to the mode through the device binding method of the browser audio interface.

2. The real-time bidirectional simultaneous interpretation method based on multi-source audio and multi-model collaboration according to claim 1, characterized in that, The multiple sealing predicate specifically includes at least two of the following predicates: Predicate 1: A sentence-final punctuation mark appears at the end of the sentence; Predicate 2: The difference between the last update time and the current time is ≥2000 milliseconds; Predicate 3: The sentence duration is ≥6 seconds and the interval between the most recent speech activity is ≥300 milliseconds; Predicate 4: Sentence duration ≥ 10 seconds; Predicate 5: The sentence is generated by forced truncation and has an idle time of more than 8 seconds; Meanwhile, the computer device schedules the wall clock beat in memory every 200 milliseconds, and uses the difference between the wall clock time and the last update time as a fallback estimate of the voice activity interval.

3. The real-time bidirectional simultaneous interpretation method based on multi-source audio and multi-model collaboration according to claim 1, characterized in that, The sentence assembler continues execution in the "pending" state: For each incoming portion of text, calculate the longest common prefix with the previous portion of text, and use only the portion with the common prefix as the stable text to drive subsequent judgments; When the previous audio segment is forcibly truncated and the newly arrived text in this round has a maximum of 6 words overlapping at the end of the confirmed buffer after removing capitalization and punctuation, the overlapping headers in the text in this round are stripped off according to the word level comparison and then written into the buffer. The translation is triggered in parallel by the preview translation coroutine with a short delay in translating the stabilized text. When the preview result is sent back, the sentence identifier recorded at the time of request is compared with the sentence identifier of the assembler's current active sentence identifier. If they do not match, the result is discarded.

4. The real-time bidirectional simultaneous interpretation method based on multi-source audio and multi-model collaboration according to claim 1, characterized in that, The single decision function further includes: The language fusion parsing submodule is called and returns the final source language according to the following priority: if the text language recognition result falls within the set of source language basic code and target language basic code, the text language recognition result is adopted; otherwise, the acoustic recognition result is adopted. When the acoustic recognition also fails, the text language recognition result is adopted again. If all the above methods fail, the language is downgraded to the source language configured by the user. The decision object is saved as a route snapshot of the sentence by the sentence assembler in an immutable form. All subsequent events of the sentence are forcibly read from this snapshot to fill the display panel's home, source language, and target language fields.

5. The real-time bidirectional simultaneous interpretation method based on multi-source audio and multi-model collaboration according to claim 4, characterized in that, The single decision function further includes: When the system is in dual-source mode, the microphone path corresponds to speaker number 0, the display panel is assigned to the right, the source language is the real-time recognition result, and the target language is the user-configured source language; the system playback path corresponds to speaker number 1, the display panel is assigned to the left, the source language is the real-time recognition result, and the target language is the user-configured target language. When the system is in single-source mode and the detected source language base code is equal to the configured target language base code, the speaker number is 1, the display panel belongs to the right, and the target is the user-configured source language; when the detected source language base code is equal to the configured source language base code or is not in the set of both, the speaker number is 0, the display panel belongs to the left, and the target is the user-configured target language.

6. The real-time bidirectional simultaneous interpretation method based on multi-source audio and multi-model collaboration according to claim 1, characterized in that, The step of asynchronously and independently calling the large language model twice further includes: The first call is initiated to the large language model adapter with an ASR error correction prompt. The input is the archived original text and the context of the most recent several finalized original texts in the same language. The system prompt is limited to "only output the corrected original text in the same language, without any translation". The first call has a first timeout threshold. If the timeout occurs, the uncorrected archived original text will be used. The second call is initiated to the large language model adapter with translation prompts. The second call is initiated serially after the first call returns. The input is the corrected original text of the first call. If the first call times out or fails, the input is the archived original text, and several recently finalized source-translation pairs are added as paragraph context. The second call has a second timeout threshold. If the timeout occurs, the preview translation generated in the pending state stage is used. The first call and the second call share the same sentence identifier, and each is assigned an independent call category identifier that falls into the structured log; If the sentence is backtracked and merged when either of the two calls is still in progress, and the decision is that it should be merged with a newly archived sentence, the computer device cancels the asynchronous tasks corresponding to the two calls and releases the corresponding resources. After both calls return, the computer device performs an atomic sentence state field update, writes the corrected original text field and the final translated text field together, and promotes the state from sealed atomic to final state.

7. The real-time bidirectional simultaneous interpretation method based on multi-source audio and multi-model collaboration according to claim 1, characterized in that, The method also includes a knowledge base retrieval method provided on demand by the computer device, including: The computer device calls the query keyword extraction prompt words and initiates a lightweight call to the large language model adapter to extract the search keywords, proper nouns, and technical terms from the most recent context of the current session into a query string; The computer device sends the query string into the embedding adapter to obtain the query vector, and calculates the similarity of all question and answer item vectors in the currently activated knowledge base using cosine similarity. The top-K items with similarity higher than a preset threshold are taken as the recall results. The computer device returns the recall results to the front end in the form of a result set, which is used by the user for comparison and reference in translation proofreading or conference Q&A scenarios, without automatically inserting the translation word by word; The knowledge base retrieval is initiated on demand by the user in the knowledge base or conference Q&A interface, and is not automatically triggered in the preview or finalization call chain of real-time sentence segmentation translation, thus not increasing the end-to-end latency of real-time translation; The recall results are returned to the front end via an interface and visualized by the user interface in the form of pop-ups or drawers, making it easy for users to compare the original text, translation, and reference materials.

8. The real-time bidirectional simultaneous interpretation method based on multi-source audio and multi-model collaboration according to claim 6, characterized in that, The method is further executed before each large language model translation call: The computer device maintains a terminology list, each entry of which includes source text, target text, entry type, group, and activation flag fields; the entry type is one of three: "must be translated as", "must not be translated as", or "preserve original text"; The computer device applies each entry simultaneously in both directions, "source language → target language" and "target language → source language", eliminating the need for the user to repeatedly enter the same term in both directions. The computer device allows users to divide items into several groups according to "meeting topics" and select the active group set for the current meeting before the start of each meeting; The computer device will activate all entries in the set, group them by type, and assemble them into terminology instruction blocks, which will then be injected into the system prompt words called by the translation big language model. After the large language model returns the translation, the computer device scans each activated term entry in the translation to perform violation detection. A violation must be translated as such only if the source text appears in the original text but the target text does not. A violation will not be translated as such only if both the source text and the target text appear in the original text. A violation will be retained only if the source text appears in the original text but the translation does not directly copy it. Detected violations are recorded in a structured format and their locations are marked with underlines / tooltips in the front-end caption bubble. The computer equipment does not perform string replacement or post-processing rewriting on the violations in order to maintain the naturalness of the translated syntax.

9. The real-time bidirectional simultaneous interpretation method based on multi-source audio and multi-model collaboration according to claim 1, characterized in that, For each sentence that transitions to the "final state," the step of automatically invoking the speech synthesis adapter further includes: In the front-end responsive state system, subscribe to the set of finalized sentences for each display panel. When the number of sets changes, skip the panel with the mode "closed". Use the recorded recently read sentence identifier to filter out new sentences that have not yet been read. Determine whether to extract the original text or the translation based on the target text selection of the panel. Enqueue each item to be read into the serial playback queue. The playback coroutine sequentially retrieves the items to be read from the queue and submits a synthesis request to the main process. After the main process returns the binary audio, the front end creates the audio element. If the setting is "local speaker", the playback method of the audio element is called to play it on the default output device. If the setting is "virtual microphone", setSinkId is called first to bind the output of the audio element to the virtual microphone device ID before playback. Before requesting playback, the stored virtual microphone device ID is called to perform self-healing by calling the parsing submodule: if the device ID still exists in the browser enumeration results, it is used directly; if it does not exist, the labels of all output devices are matched in case-insensitively according to the predefined keyword set. If a match is found, the old ID in the database is replaced with the newly discovered device ID. The predefined keyword set is a list of a series of characteristic strings that are predefined in the code and used to identify virtual audio devices. In the "Local Speaker" setting, TTS audio played through the local speaker, which relies on echo cancellation enabled during front-end audio acquisition to suppress the audio, is re-picked up by the microphone; and an echo risk warning is given when the user first switches the panel to the local speaker setting.

10. The real-time bidirectional simultaneous interpretation method based on multi-source audio and multi-model collaboration according to claim 1, characterized in that, The engines requiring local inference in the large language model adapter, automatic speech recognition adapter, and speech synthesis adapter are carried by the computer device in independent sub-processes. The main process and the sub-process communicate via a binary frame protocol consisting of 4 bytes big-endian length + 1 byte type + N bytes payload. When the engine switches during runtime, the computer device executes: Calculate a signature for each load request. If the new signature is equal to the already effective signature and the role is currently READY or STARTING, return directly. Each time the signature changes, the reload sequence number is incremented by 1 and recorded as the expected sequence number for this reload. After each asynchronous wait, the current expected sequence number is compared with the latest reload sequence number. If they do not match, the current process is stopped and the started child processes are cleaned up. Before switching, the session epoch module is invoked to mark the current epoch as deprecated and immediately create a new epoch, giving a window of up to 10 seconds to allow active sessions to end naturally.

11. The real-time bidirectional simultaneous interpretation method based on multi-source audio and multi-model collaboration according to claim 1, characterized in that, The computer device only installs the core dependency packages upon first startup; When a user enables the local inference engine, a standard installation pipeline consisting of at least ten stages is executed. These ten stages include, in sequence: hardware requirements and external dependency checks, CUDA version PyTorch verification, optional pre_pip hook, overlay dependency lock file-based installation, optional post_install hook, dual-source sequential rollback download of model weights, real import verification within a subprocess, engine-specific compliance checks, optional post_verify hook, and atomic writing to the installation manifest. The dual-source sequential rollback download of model weights includes HuggingFace priority and ModelScope rollback. The installation result is recorded in the install.json installation manifest. The computer device performs the following each time it starts up: Read the basic dependency list, check if the corresponding release information directory exists in the base site directory for each dependency, and if not, call pip to install it with "--upgrade-strategyonly-if-needed"; For each installed overlay engine, compare the current dependency directory declaration with the dependency snapshot in the installation manifest, install only the newly added difference set, and immediately overwrite the installation manifest with the new snapshot.

12. The real-time bidirectional simultaneous interpretation method based on multi-source audio and multi-model collaboration according to claim 11, characterized in that, For each overlay deployment mode, the local inference engine explicitly declares the lock file path pointed to by the 'overlay_lockfile' field in the engine metadata, and each non-empty, non-comment line in the lock file must match one of the following formats: "package name == version number" (strict equals sign) or "package name @ remote archive URL" (remote archive URL). The lock file must not contain dependencies starting with torch, torchaudio, or nvidia-*; during startup, the audit function iterates through the lock file pointed to by the 'overlay_lockfile' field and performs the above-mentioned checks and prohibitions. If the check fails, it will be recorded as a warning in the unified log library for developers to fix. The aforementioned lockfile locking mode ensures that the overlay dependencies of the same software version distributed to different users have exactly the same version combination, and prevents torch family dependencies from being repeatedly loaded into the overlay, which could cause ABI conflicts or disk usage bloat.

13. The real-time bidirectional simultaneous interpretation method based on multi-source audio and multi-model collaboration according to claim 1, characterized in that, The specified process audio capture channel based on the operating system process audio loopback application interface is implemented by the computer device by performing the following sub-steps: Call IMMDeviceEnumerator to enumerate all active audio sessions, and record the PID and process name of each session for the front end to select; After selecting the PID, construct the AUDIOCLIENT_ACTIVATION_PARAMS structure with the activation type AUDIOCLIENT_ACTIVATION_TYPE_PROCESS_LOOPBACK and the mode PROCESS_LOOPBACK_MODE_INCLUDE_TARGET_PROCESS_TREE. ActivateAudioInterfaceAsync asynchronously, and use COMcompletionhandler to wait for the result; The WASAPI data packet is read in the acquisition thread, downmixed to mono, resampled to 16 kHz, and quantized to 16-bit PCM. Each frame of PCM is sent back to the Node.js main thread via N-API's ThreadSafeFunction and then pushed to the rendering process via IPC.

14. A real-time bidirectional simultaneous interpretation device based on multi-source audio and multi-model collaboration, characterized in that, The apparatus is configured in a computer device for performing the real-time bidirectional simultaneous interpretation method based on multi-source audio and multi-model collaboration as described in any one of claims 1-13, the apparatus comprising: The audio acquisition module is used to capture audio frames from the user-selected audio source and encode them into standard PCM. The translation strategy module is used to call a single decision function for each frame of automatically recognized speech text that arrives, and outputs an immutable decision object containing four fields: source language, target language, speaker number, and display panel affiliation. The sentence assembly module is used to maintain a sentence assembler for each audio source and atomically transfer each sentence between the three states of "to be processed - sealed - finalized" according to the decision object and the multiple sealing predicate; The final state module is used to asynchronously call the large language model adapter for correction and translation after the sentence enters the "sealed" state, and ensure the final state is closed according to the predefined degradation strategy; The automatic reading module is used to send the target text of the finalized sentence to the speech synthesis adapter according to the three reading modes of "off / local speaker / virtual microphone", and route it to the corresponding output device according to the mode. The engine management module is used to host the local inference engine in an independent subprocess and maintain its lifecycle. The installation management module is used to perform a minimal core installation on the first boot, execute a ten-stage standard installation pipeline when the user enables the local engine, and perform a dependency alignment process on each boot. The session protection module is used to ensure that engine switching does not interrupt the session by using session epochs and reference counting.

15. The real-time bidirectional simultaneous interpretation device based on multi-source audio and multi-model collaboration as described in claim 14, characterized in that, The device further includes: The knowledge base management module is used to enable on-demand knowledge base retrieval of RAGs. The terminology management module is used to implement bidirectional rules for terminology and soft prompts + UI tags; The LAN API sharing module is used to provide an external, AI-compatible entry point. The unified log bus module is used to achieve end-to-end latency observability; The license base module is used to implement multiple license levels for desktop software.

16. The real-time two-way simultaneous interpretation device based on multi-source audio and multi-model collaboration as described in claim 15, characterized in that, The LAN API sharing module exposes local area network API sharing methods for multiple large language model engines configured on the local machine using OpenAI-compatible protocols, including: Mount routes with the same URL paths and request body schemas as the OpenAI Chat Completions API and Embeddings API in the FastAPI application of the main process; aggregate multiple large language model providers already configured on the local machine under the same entry point in the form of "virtual model name" mapping, route to the corresponding backend adapter according to the model field of the request body, and return the backend response after converting it back to the OpenAI standard schema; Several self-signed API tokens are maintained in the local business database. All external interface calls must include a valid token in the HTTPAuthorization Header. Users can revoke any token with one click in the user interface. All external interface calls are recorded and statistically analyzed by dimensions such as call token name, backend engine type routed to, call date, usage (character or token estimate), and failure count; error messages in external responses are anonymized according to preset keyword rules; Provide call statistics views in a separate pop-up window, categorized by Token / Engine / Model / Time Period.

17. The real-time bidirectional simultaneous interpretation method based on multi-source audio and multi-model collaboration according to claim 15, characterized in that, The unified log bus module maintains an SQLite log library independent of the business database. The log library stores logs in a single table named logs, with fields including timestamp, category, component name, level, source process, session identifier, trace identifier, message, and load. The category is one of 10 enclosed enumeration values: system / install / engine / audio / asr / llm / tts / session / ui / rag. Background tasks clean up old logs every 5 minutes according to their level: debug logs are kept for 1 hour, info logs for 6 hours, and warn and error logs for 7 days. A Pydantic model is defined as the load schema for each of the 10 categories, and all logs must pass validation before being written. For each sentence event traversing the system, four timestamps are embedded: asr_yielded, event_emit, ws_received, and seg_rendered. These correspond to the moment when the automatic speech recognition adapter outputs the text, the moment before the sentence assembly module delivers the event to the WebSocket sending channel, the moment when the front-end WebSocket on Message callback entry is reached, and the moment when the DOM pixels have been updated after the front-end Vue view is refreshed. A pre-defined SQL paradigm that can be directly executed on the log database is used to verify two architectural rules: "the original text arrives before the translation" and "the large language model does not block the automatic speech recognition output". The former is achieved by comparing the event timestamps of the same sentence identifier to determine the order of the source / target fields, while the latter is achieved by associating the adjacent intervals of the asr_yielded event with the large language model calls during the same period.

18. The real-time two-way simultaneous interpretation device based on multi-source audio and multi-model collaboration as described in claim 15, characterized in that, The authorization base module maintains a method for authorization management of computer software, the method including: The system receives and stores a digital authorization certificate issued by an authorization server. The digital authorization certificate includes a long-term activation code and a short-term verification code. The long-term activation code is a readable string consisting of "prefix + plan abbreviation + three segments of four-character random short code," where the random short code is taken from a predefined character set excluding easily confused characters. The short-term verification code is obtained by the computer device from the authorization server in exchange for the long-term activation code, using an Edwards curve EdDSA signature. The payload includes at least: an authorization identifier, a device identifier information field, an authorization capability list field, and a validity period field. The device identifier information field is an array of device identifiers used to bind one digital authorization certificate to multiple client devices. The authorization capability list field is a set of one or more capability identifiers. Based on the matching verification of the device identification information and the locally calculated device fingerprint, it is determined whether the current client device has been authorized. The determination of whether the current client device has been authorized includes: verifying whether the locally calculated device fingerprint is included in the device identification array; wherein, the locally calculated device fingerprint is generated based on combining at least two types of inherent information of the following device hardware or operating system and calculating their hash values: machine globally unique identifier, central processing unit identification information, and motherboard identification information; During the validity period of the digital authorization credential, access control is performed on the client's software functions based on the authorization capability information contained therein; the access control on the client's software functions includes: when a user attempts to use any software function, checking whether the capability identifier corresponding to that function exists in the authorized capability list; When the digital license is about to expire, a renewal request is sent to the license server to obtain a new digital license; in the offline state where the license server cannot be connected, the software function is allowed to continue to be used within the grace period specified in the digital license; if the renewal cannot be completed after the grace period, the software function is locked. Identify the authorization plan type in the digital authorization certificate; if it is identified as a lifetime authorization type, skip the check of the validity period field, but continue to perform device binding verification and authorization revocation checks.

19. The real-time two-way simultaneous interpretation device based on multi-source audio and multi-model collaboration according to claim 18, characterized in that, The authorization base module maintains a method for licensing management of computer software, the method further comprising: Periodically retrieve and verify a digitally signed list of license revocations from the authorization server; if the identifier of the current digital authorization credential exists in the verified list of license revocations, immediately lock the software functionality; Record the time when the license revocation list was last successfully obtained; if the client device remains offline for more than the preset maximum number of offline days since that time, the software functions will be locked and a new network connection verification will be required even if a new license revocation list cannot be obtained.

20. A computer device (500), characterized in that, The device includes a processor (502), a communication interface (505), a memory, and a communication bus (501). The processor (502), the communication interface (505), and the memory communicate with each other through the communication bus (501). The communication interface (505) is used to connect a microphone and a speaker. The memory includes a storage medium (503) and an internal memory (504), wherein the storage medium (503) is used to store the operating system (5031) and computer programs (5032). When the processor (502) executes the computer program (5032) stored in the memory, it implements the steps of the real-time bidirectional simultaneous interpretation method based on multi-source audio and multi-model collaboration as described in any one of claims 1-4.

Images