Multi-language barrier-free conference room and implementation method thereof
By using a unified AI model server and streaming technology, combined with asynchronous message queues, low-latency and high-efficiency translation for cross-language conferences was achieved, solving the problems of high latency and high resource consumption in existing technologies, and improving conference efficiency and system stability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENZHEN YINNUO INTELLIGENT EQUIPMENT CO LTD
- Filing Date
- 2026-02-05
- Publication Date
- 2026-05-01
AI Technical Summary
Existing technical solutions suffer from high processing latency, high resource consumption, and poor real-time interactivity in cross-language conferences, making it difficult to support smooth and natural dialogue and affecting conference efficiency.
A unified AI model server is used to perform model inference in parallel to generate multilingual translations. By combining streaming technology and asynchronous message queues to decouple various processing services, real-time parallel translation and display of audio data can be achieved.
It enables low-latency, high-efficiency multilingual conferencing, reduces computational overhead and timing latency, improves system throughput and resilience, and supports large-scale concurrent meetings.
Smart Images
Figure CN121963745A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multilingual barrier-free meeting room and its implementation method. Background Technology
[0002] With the deepening of global collaboration, the demand for cross-language conferences is growing. Existing technical solutions typically employ a sequential processing model: first, automatic speech recognition (ASR) converts complete sentences in one language into text; then, multiple independent monolingual machine translation (MT) engines are sequentially invoked for translation; finally, the translation is presented to the user via text-to-speech (TTS). This approach has significant drawbacks: high processing latency, as translation cannot begin until the complete sentence is finished; high resource consumption, as multiple calls to the translation model increase computational overhead; and poor real-time interactivity, making it difficult to support fluent and natural dialogue, severely impacting conference efficiency.
[0003] Therefore, there is an urgent need in this field for a real-time multilingual conferencing solution that can achieve low latency, high efficiency, and high accuracy. Summary of the Invention
[0004] Based on this, it is necessary to address the technical problems of high translation delay and low efficiency in existing technologies by proposing a multilingual barrier-free conference room and its implementation method.
[0005] Firstly, a method for implementing a multilingual, barrier-free conference room is provided, the method comprising: The method is executed collaboratively by an application server, an AI model server, and multiple user terminals, and includes the following steps: The application server creates a meeting room instance and generates a corresponding access link. The user terminal connects to the application server through the access link and sends its selected target language information to the application server. The application server receives audio data streams from user terminals in real time and segments them into continuous audio stream segments. The application server sends the audio stream segment to the AI model server; The AI model server performs speech recognition on the received audio stream segments and generates corresponding source language text segments. The AI model server, based on the target language list provided by the application server, translates the source language text fragment into translated text fragments in multiple target languages in parallel through a single model inference. The AI model server sends the translated text fragment to the application server; The application server selects the corresponding language from the translated text fragments based on the target language selected by each user terminal, and pushes it to the corresponding user terminal for display in real time.
[0006] Preferably, the application server and the AI model server exchange data through an asynchronous message queue to decouple the request and processing processes.
[0007] Preferably, both the source language text fragment and the translated text fragment are associated with corresponding sentence IDs and timestamp information.
[0008] Preferred options also include: The application server combines the source language text fragments and their corresponding multiple translated text fragments to form meeting record texts in different languages, and stores them in the meeting record database; Upon detecting the end of the meeting, the application server retrieves the meeting record text from the meeting record database and instructs the AI model server to generate meeting minutes in different languages based on the meeting record text; The application server distributes the generated meeting minutes to the corresponding user terminals.
[0009] Preferably, the meeting record text consists of sentences arranged in chronological order, with each sentence containing the original sentence, translations in multiple languages, a sentence ID, and a timestamp.
[0010] Preferably, the generation request sent by the application server to the AI model server includes a constraint instruction, which restricts the AI model server to generate meeting minutes only from the meeting record text.
[0011] Preferred options also include: The meeting minutes generated by the AI model server are output in a hierarchical Markdown format; The application server renders the Markdown format meeting minutes into a meeting minutes mind map and distributes it to user terminals.
[0012] Preferably, the AI model server performs speech recognition on the received audio stream segments to generate corresponding source language text segments: The AI model server receives audio stream segments.
[0013] The encoder encodes the current audio stream segment based on the encoding state of the previous audio stream segment and updates the encoding state. The decoder generates source language text segments in real time based on the encoding results of the current audio stream segment and the context of the preceding source language text segments.
[0014] Preferred options also include: When the application server sends the audio stream segment to the AI model server, it also sends a list of target languages required for this translation task. The target language list is determined by the application server based on the language selected by all user terminals in the current conference room.
[0015] Secondly, a multilingual accessible conference room system is provided for implementing the multilingual accessible conference room implementation method as described in any of the above claims, including: Multiple user terminals are used to connect to the application server via an access link and send their respective selected target language information to the application server. The application server is used to create meeting room instances and generate corresponding access links; It receives audio data streams from user terminals in real time and segments them into continuous audio stream segments; The audio stream segment is sent to the AI model server; The system receives translated text fragments from an AI model server, selects the corresponding language from the translated text fragments, and pushes them to the corresponding user terminals for display in real time. The AI model server is used to translate the source language text fragment into multiple target languages in parallel through a single model inference, based on the target language list provided by the application server. The translated text fragment is sent to the application server.
[0016] Beneficial effects: This application invites participants via a unique QR code or link, enabling "one-click access" and significantly reducing the complexity of meeting organization and the barrier to entry, while improving meeting startup efficiency. Employing streaming technology, audio is processed as it arrives, eliminating the need to wait for complete sentences and drastically reducing end-to-end processing time from the source. Using a unified AI model server, multilingual translations are generated in parallel during a single model inference, avoiding the significant computational overhead and latency associated with multiple calls to independent models in traditional serial solutions, achieving high efficiency and low resource consumption. Asynchronous message queues decouple various processing services, preventing inter-service blocking, improving system throughput and resilience, and enabling stable support for large-scale concurrent meetings. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] in: Figure 1 This is an application environment diagram of a method for implementing a multilingual accessible conference room in one embodiment. Figure 2 A flowchart of a method for implementing a multilingual accessible meeting room in one embodiment; Figure 3 This is a schematic diagram of a meeting minutes in Markdown format in one embodiment; Figure 4 This is a mind map diagram of meeting minutes in one embodiment; Figure 5 This is a schematic diagram of a multilingual accessible conference room system in one embodiment. Detailed Implementation
[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0020] The method for implementing a multilingual, barrier-free meeting room provided in this invention can be applied to, for example... Figure 1In this application environment, user terminal 110 communicates with application server 120 via a network. The application server creates a conference room instance and generates a corresponding access link. The user terminal connects to the application server through the access link and sends its selected target language information to the application server. The application server receives audio data streams from the user terminals in real time and segments them into continuous audio stream segments. The application server sends the audio stream segments to the AI model server. The AI model server performs speech recognition on the received audio stream segments and generates corresponding source language text segments. Based on the target language list provided by the application server, the AI model server translates the source language text segments into translated text segments in multiple target languages in parallel through a single model inference. The AI model server returns the translated text segments to the application server. The application server, based on the target languages selected by each user terminal, selects the corresponding language from the translated text segments and pushes it to the corresponding user terminal for display in real time. In this invention, a unified AI model is employed for multilingual conference room scenarios. This model generates multilingual translations in parallel during a single model inference, avoiding the significant computational overhead and timing delays caused by multiple calls to independent models in traditional serial solutions. This achieves high efficiency and low resource consumption. The user terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The application server can be implemented using a standalone server or a server cluster. The unified AI model mounted on the AI model server adopts a Transformer-based encoder-decoder architecture. The encoder is a streaming speech encoder used for incremental encoding of input audio stream segments. The decoder is a multilingual parallel decoder, whose output layer corresponds to a vocabulary containing all supported languages. During inference, the target language list provided by the application server is converted into a series of language identifiers, which are used as prefixes along with the encoder's output and input to the decoder. During a single forward propagation, the decoder generates the corresponding target language lexical probability distributions in parallel at the output layer based on these language identifiers, thereby achieving a one-time parallel generation from audio stream segments to multilingual translated text segments. The invention will be described in detail below through specific embodiments.
[0021] Please see Figure 2 As shown, Figure 2 A flowchart illustrating the method for implementing a multilingual accessible meeting room according to an embodiment of the present invention includes the following steps: S1: The application server creates a conference room instance and generates a corresponding access link. User terminals connect to the application server through the access link and send their respective selected target language information to the application server.
[0022] Specifically, user terminals refer to the hardware devices participating in the meeting, such as smartphones, personal computers, and conference tablets, on which the meeting client software runs. The application server is the control center and scheduling hub of the entire system. It is a service cluster composed of multiple key service components, responsible for receiving, coordinating, and distributing data, ensuring the reliable and orderly execution of the entire real-time translation process. It implements WebSocket gateway services, acting as a bidirectional real-time communication bridge between the server and all user terminals (such as mobile phones and computers), and is responsible for maintaining a stable, long-term connection with each participant's device. Simultaneously, it serves as a buffer and communication channel between various processing services (such as speech recognition and translation services), ensuring that the data flow is not blocked due to the processing speed of any link. It is responsible for handling business logic, such as meeting management, user management, routing decisions, and coordinating other services such as AI models and databases. A meeting room instance refers to an independent virtual space that carries all data and states of a specific meeting. An access link is the link bound to a meeting room instance; it is a unique URL (Uniform Resource Locator) pointing to a specific virtual meeting room, automatically generated by the meeting creator / organizer when creating the meeting room in the system backend, and is a unique resource address. This link is usually associated with a unique QR code, which can be scanned to automatically redirect to the meeting. The target language refers to the language (e.g., Chinese, English, Japanese) that the participant personally chooses and wishes to receive the meeting content in.
[0023] For example, Zhang San, the meeting organizer, uses a personal computer to access the meeting webpage through a browser. Dr. Smith uses a smartphone with the system's dedicated meeting app installed. Sato uses a meeting tablet and accesses the meeting link through its built-in browser. After Zhang San creates a meeting room instance, "2024 Global Product Technology Summit," in the system backend, the system automatically generates a unique resource URL: https: / / a11y-meeting.com / room / global-tech-summit-2024-A1B2C3. This URL is simultaneously encoded as a QR code image, which participants can scan to join the meeting. After successfully entering the virtual meeting room interface, the system presents a language selection dropdown menu, allowing each user to choose their preferred language. Zhang San, as the speaker, keeps the system's default interface language set to "Chinese (Simplified)," and his speech will be recognized as the source language by the system. Dr. Smith selects "English" from the language list in the settings or personal information area of the interface. Sato does the same, selecting "Japanese." Once the user completes the language selection, the client will send the preference setting to the backend server and bind it to the user's session identity.
[0024] S2: The application server receives audio data streams from user terminals in real time and segments them into continuous audio stream segments.
[0025] Specifically, an audio data stream refers to the continuous audio signal generated when a user speaks. An audio stream segment refers to an audio data stream divided into segments in time sequence (e.g., fixed-length segments of 200-500 milliseconds). Incremental processing of audio stream segments through streaming avoids the high latency of waiting for the entire sentence to finish before processing. The application server, as the core hub, ensures that audio data can be reliably and orderly distributed to subsequent processing units (such as AI models).
[0026] S3: The application server sends audio stream segments to the AI model server.
[0027] Specifically, the AI model server is a software service cluster that hosts a unified, end-to-end streaming artificial intelligence system where multiple models work collaboratively; that is, it hosts multiple AI models. These AI models include streaming speech recognition services, unified multilingual translation, and meeting minutes generation services based on a retrieval-enhanced generative framework. It is responsible for converting continuous audio streams into text in real time, translating them, and summarizing meeting minutes.
[0028] Preferably, the AI model is further trained from mainstream models currently on the market, such as ChatGPT and Deepseek. Through local deployment, further domain-adaptive training and fine-tuning of specific capabilities are performed using professional corpora, thereby creating a customized model that combines general knowledge with advanced professional skills. With local deployment, all audio and text data are processed within the client's intranet environment, meeting the stringent data security requirements of scenarios such as finance, government, and law. Deploying the model on a local server eliminates latency and instability caused by public network fluctuations, ensuring a smooth meeting experience. In the local environment, deeper and continuous incremental training and optimization are performed based on the client's specific terminology, expression habits, and business domain, making the model increasingly "intellectual."
[0029] Specifically, existing models are customized and trained. The entire training process follows the paradigm of "pre-training - domain adaptation - incremental learning" to ensure that the model has both strong general-purpose capabilities and can achieve optimal performance in professional conference scenarios. Large-scale pre-training - building basic capabilities.
[0030] The first stage is pre-training. The speech recognition model is trained using speech-text pairing datasets ranging from tens of thousands to hundreds of thousands of hours in size, covering multiple languages, accents, and noisy environments. A streaming speech recognition model is trained using connectionist temporal classification or sequence-to-sequence pre-training objectives. This model initially acquires the ability to map input audio sequences to text sequences and learns mechanisms such as causal convolution and sliding window attention required for streaming processing. The multilingual translation model is pre-trained using a massive parallel corpus covering hundreds of language pairs. A unified multilingual translation model is trained through Transformer-based sequence-to-sequence pre-training. This model learns to internally construct a cross-lingual semantic space and can control the output language by adding specific language identifiers (such as [English]) before the input.
[0031] The second phase involves domain adaptation and supervised fine-tuning, aiming to integrate and fine-tune the pre-trained model into a domain expert model proficient in business and technology conference scenarios. First, a high-quality parallel corpus of professional domains is constructed, carefully collecting and cleaning high-quality speech-source-multilingual translation triplet data covering target domains such as business, technology, and finance. For example, a Chinese business speech audio, its corresponding Chinese text, and its high-quality English, Japanese, German, and other language translations. The corpus must ensure the accuracy of professional terminology translations and that sentence structures conform to formal written or spoken language. End-to-end supervised fine-tuning integrates the pre-trained streaming speech recognition model and multilingual translation model to construct an end-to-end unified AI model server. Its input is audio segments, and its output is parallel-generated multilingual text. The input is initially an audio segment plus a list of target languages (e.g., [English, Japanese]). During forward propagation, the audio passes through a speech encoder to obtain features, which, along with target language information, are input into the fused multilingual decoder. The model generates a text sequence for each target language. The cross-entropy loss between the output of each target language and the corresponding standard translation is calculated, and the losses of all languages are weighted and summed to obtain the total loss. Backpropagation is performed using the gradient descent algorithm to minimize the total loss and update all model parameters. This process forces the model to learn the ability to directly map audio features to multilingual translations, rather than simply "dictating" and then "translating".
[0032] The third phase achieves continuous evolution through online incremental learning and alignment. After model deployment, it enters a continuous online optimization phase to adapt to specific user groups and new expressions. Following user authorization and strict anonymization, the system collects user feedback data. The most important feedback comes from the "error correction mechanism" mentioned in your document: when users correct the model's recognition or translation results, the system records (audio clips, the corrected text) as high-quality supervisory data pairs. Simultaneously, the system anonymizes and stores meeting data (audio, the ultimately approved meeting transcript text) as positive samples. The system periodically (e.g., weekly) incrementally trains the deployed model using newly collected high-quality data. During training, a small learning rate is typically used to prevent catastrophic forgetting of existing knowledge.
[0033] A custom-trained AI model server is deployed as a local API service on a GPU server within the intranet. The application server communicates with the locally deployed AI model service instance via remote calls over the intranet. This intranet communication features extremely low latency, ample and stable bandwidth, avoiding interference from public network fluctuations on real-time audio transmission and translation.
[0034] For example, when participant A (in Chinese) speaks, their terminal device's microphone continuously captures audio. Before transmission or after reception by the application server, the system performs a series of preprocessing operations to improve subsequent recognition accuracy and transmission efficiency. Preprocessing includes: noise reduction, i.e., filtering out ambient noise such as background keyboard sounds and air conditioning noise; echo cancellation, i.e., eliminating the other party's voice played from the speaker to prevent it from being captured by the microphone again and forming an echo; silence detection, i.e., intelligently determining when it is valid speech and when it is a pause to avoid transmitting silent segments and save bandwidth; and using efficient audio codecs such as OPUS (Opus Interactive AudioCodec) to compress audio stream segments, reducing the amount of data transmitted over the network. The application server pushes the preprocessed audio stream segments (such as segments containing the word "we") to the AI model server through efficient communication protocols or message queues.
[0035] The application server maintains bidirectional real-time communication with each user terminal via a WebSocket long-lived connection. Internally, the application server employs an asynchronous message queue to decouple the processing units. Upon receiving an audio stream segment, the server encapsulates it as a message and publishes it to the speech recognition task queue. The AI model server, acting as a consumer, retrieves and processes the message from the queue, and then publishes the recognition and translation results to the results queue. The application server consumes the results queue and, based on the session ID and target language tag carried in the message, pushes the translation to the target user terminal via the corresponding WebSocket long-lived connection. This asynchronous, decoupled architecture ensures the system's high concurrency and reliability.
[0036] S4: The AI model server performs speech recognition on the received audio stream segments and generates corresponding source language text segments.
[0037] Specifically, source language text fragments refer to incomplete text blocks corresponding to the audio obtained by the speech recognition model in the AI model server after performing speech recognition on the input audio stream fragments, such as "we" or "next".
[0038] S5: The AI model server, based on the target language list provided by the application server, translates source language text fragments into multiple target language translated text fragments in parallel through a single model inference.
[0039] Specifically, the target language refers to the language selected by each user terminal when entering the meeting room. The target language list is a user-language mapping table generated by the application server based on the target languages selected by all participating users. The AI model server performs speech recognition on the audio stream segments to obtain source language text segments. Then, based on a single model inference, it translates the source language text segments into multiple target languages in parallel, resulting in translated text segments in multiple target languages. A single model inference refers to a complete forward propagation computation performed by the multilingual translation model in the AI model server. In this application, it specifically refers to the model simultaneously completing both speech recognition and multilingual translation tasks in a single computation. Parallel translation differs from traditional serial translation (first recognizing complete sentences, then sequentially calling independent translation models such as Chinese-English and Chinese-Japanese), meaning that the multilingual translation model can simultaneously output translations in multiple target languages.
[0040] The AI model server employs a streaming encoder architecture. The encoder incrementally encodes the current audio stream segment based on the encoding state of the previous audio stream segment and updates this encoding state for use in the next audio stream segment, thus achieving low-latency speech recognition. The AI model server further includes a multi-task decoder. During a single forward inference process, this decoder, based on the encoder's output, activates multiple output layers corresponding to the target language in parallel, thereby simultaneously generating translated text segments in multiple target languages. This 'one-time model inference, parallel multilingual output' mechanism fundamentally avoids the accumulated latency and resource overhead caused by serially calling multiple translation engines in traditional solutions.
[0041] For example, the "bottleneck" of the audio segment received by the AI model server and the context information "Next, we will evaluate this architecture" retrieved from the previous cache. Parallel output in one inference: Inside the model, it is processed through a unified multimodal large language model (such as a model based on the Transformer architecture). In a single inference, its decoder part activates multiple language output heads in parallel and synchronously generates: Source language (Chinese) text segment: "瓶颈"; Target language 1 (English) translation text segment: "bottleneck"; Target language 2 (Japanese) translation text segment: "ボトルネック". This process does not need to wait for the end of the whole sentence and is performed in real time for each incoming audio segment.
[0042] S6: The application server filters out the corresponding language from the translated text segments according to the target languages selected by each user terminal and pushes it to the corresponding user terminal for display in real time.
[0043] Specifically, real-time push means that the application server establishes a two-way real-time communication channel between the application server and the user terminal through long connection technologies such as WebSocket gateway service, enabling the application server to actively and immediately "push" data to the client without the client repeatedly "polling" for requests.
[0044] For example, the application server receives a text segment package containing translations in multiple languages from the AI model server. The application server extracts the corresponding translations from the package according to the user-language mapping table it maintains (A Chinese, B English, C Japanese). It pushes the Chinese text "瓶颈" to the device of user A. It pushes the English text "bottleneck" to the device of user B. It pushes the Japanese text "ボトルネック" to the device of user C. The conference client software on each user terminal receives and displays the pushed translation text in real time, which is basically synchronized with the speaker's voice.
[0045] This application invites participants through a unique QR code or link, achieving "one-click entry into the meeting", greatly reducing the organizational complexity and participation threshold of the meeting, and improving the meeting startup efficiency. By adopting streaming processing technology, audio is processed as it arrives without waiting for the end of a complete sentence, significantly reducing the end-to-end processing time from the source.
[0046] By using a unified AI model server, multiple language translations are generated in parallel in one model inference, avoiding the huge computational overhead and timing delay caused by multiple calls to independent models in traditional serial schemes, and achieving high efficiency and low resource consumption. By decoupling each processing service through an asynchronous message queue, it avoids service blocking, improves the throughput and anti-pressure ability of the system, enabling it to stably support large-scale concurrent meetings.
[0047] In some implementations, the application server and the AI model server exchange data through an asynchronous message queue to decouple the request and processing processes.
[0048] Specifically, an asynchronous message queue is a software component that acts as a temporary storage and forwarding buffer for messages. After an application server sends a message to the queue, it can continue processing other tasks without waiting for the AI model server to process it immediately. The AI model server then retrieves and processes messages from the queue according to its own processing capabilities. Decoupling refers to separating two services that would otherwise require direct, synchronous calls, so that they no longer directly depend on each other's real-time response capabilities, but instead communicate indirectly and asynchronously through middleware (message queues).
[0049] For example, after acquiring a user's audio stream segment, the application server converts it into a message (typically containing metadata such as session ID and segment ID), publishes it to a message queue named `audio_fragment_queue`, and then immediately returns, ready to receive the next segment. One or more AI model service instances continuously listen to `audio_fragment_queue`. Once a message arrives, an idle instance retrieves it. The AI model instances perform speech recognition and parallel translation on the audio segment. After processing, the results (source text and multiple translations) are published as another message to the `translation_result_queue`. The application server listens to `translation_result_queue`, obtains the results, and then pushes them to the corresponding user terminal.
[0050] The application server does not need to wait for the AI model server's processing results, allowing it to quickly handle more user requests and connections. The AI model server can also dynamically scale instances based on the number of messages backlogged in the queue to cope with traffic spikes. Even if the AI model server is temporarily unavailable due to a failure or update, messages are persistently stored in the queue and will not be lost. Once service is restored, messages in the queue can continue to be processed, achieving both peak shaving and valley filling, and fault recovery. The development, deployment, and scaling of the application server and AI model server can be carried out independently, reducing the complexity of the system architecture.
[0051] In some implementations, both the source language text fragment and the translated text fragment are associated with corresponding sentence IDs and timestamp information.
[0052] Specifically, a sentence ID is a globally unique identifier assigned by the system to each complete semantic sentence identified in a meeting. All audio stream segments, source language text segments, and their translated text segments in various languages that belong to the same sentence share the same sentence ID. A timestamp refers to the precise moment when each event (such as the start of an audio segment or the completion of sentence recognition) occurs. When the AI model server starts processing a new sentence, the system generates a unique sentence ID, such as SENT_MEETING_001. For all data packets generated during the processing of this sentence, whether intermediate segments or final results, they carry this sentence ID and the corresponding timestamp. The application server or client reorganizes the text segments according to the sentence ID and sorts them according to the timestamp, finally forming a coherent conversation flow.
[0053] For example, the user says: "We (segment 1) are going to (segment 2) evaluate (segment 3) the bottleneck of this architecture." These three audio segments and all the data generated during their recognition and translation will be marked with the same sentence ID: SENT_123. Even if the output of the English translation "bottleneck" (corresponding to segment 3) is 10 milliseconds later than the Chinese original "瓶颈", the system can correctly associate them to the same final sentence through the sentence ID SENT_123, showing as: "We are going to evaluate the bottleneck of this architecture, or Our next step is to evaluate the bottleneck of this architecture." Since speech recognition and translation are carried out in a streaming and segmented manner, there may be slight temporal differences in the output of translations in different languages. Through the sentence ID and timestamp, the system can accurately group all text segments in different languages that belong to the same sentence together, ensuring that what the user sees is a complete and contextually coherent sentence rather than a jumble of words. After the meeting, it is possible to quickly locate the discussion content at a certain time point in the meeting according to the timestamp, or accurately locate a certain speech and all its translations according to the sentence ID, which greatly facilitates the review and复盘 of the meeting content.
[0054] In some embodiments, it further includes: (1) The application server combines the source language text segments and their corresponding multiple translated text segments to form meeting record texts in different language types and stores them in the meeting record database; (2) When detecting the end of the meeting, the application server retrieves the meeting record text from the meeting record database and instructs the AI model server to generate meeting summaries in different language types based on the meeting record text; (3) The application server distributes the generated meeting summaries to the corresponding user terminals.
[0055] Specifically, meeting minutes refer to the complete, structured, multilingual set of meeting content automatically generated and saved by the system during the meeting. It is the sole source of fact for generating meeting minutes. The meeting minutes database is a database system used for persistently storing meeting minutes. Meeting minutes are highly summarized and structured summary documents generated by an AI model server based on the complete meeting minutes, containing key meeting points, discussion results, action plans, etc.
[0056] During the meeting, the application server combines source language text fragments and their corresponding multiple translated text fragments into complete sentences in real time, based on sentence IDs and timestamps, and stores them in the meeting record database. Each record includes the original text, translations in each language, and the speaking time. When the meeting host clicks "End Meeting" or the system detects that all users have left the meeting, the application server retrieves all the recorded text of the current meeting from the meeting record database. The application server sends the recorded text and a constraint instruction (e.g., "You are a meeting minutes assistant; please strictly generate a summary based on the following meeting minutes and do not add any unrecorded information.") to the AI model. The AI model server then generates a summary based on this. The application server sends the multilingual minutes (e.g., Chinese and English minutes) generated by the AI model server to the user terminals that selected the corresponding languages. The AI model server, which has the capability to generate meeting minutes, is part of a unified end-to-end model, obtained through localized deployment and customized training. Specifically, the meeting transcript generation service is a highly customized component based on mainstream large models (such as ChatGPT and DeepSeek), trained professionally for domain adaptation, and constrained by the strict RAG (Retrieval-Augmented Generation) framework. It significantly improves the efficiency and value of meeting information by transforming lengthy meeting dialogues into easily understandable structured summaries.
[0057] For example, after a meeting regarding a project review, the Chinese minutes generated by the AI model server might begin: Meeting Topic: Quarterly Review of Project XX. Decisions: 1. Establish an architecture evaluation team, led by Zhang San, to complete the evaluation report by November 20th. Users who chose English will receive the same English minutes: MeetingTopic: Quarterly Review of Project XX. Decisions: 1. Form an architectureevaluation team, led by Zhang San, to complete the report by Nov 20.
[0058] In some embodiments, the meeting record text consists of sentences arranged in chronological order, and each sentence contains the original text of the sentence, translations in multiple languages, a sentence ID, and a timestamp.
[0059] Specifically, to ensure the integrity and traceability of the data, the complete content of a meeting (the original text and translations in various languages) is saved in a structured manner to form an authoritative multilingual meeting archive. Through the sentence ID and timestamp, any speech segment in the meeting can be quickly located, facilitating subsequent review, verification, and content extraction.
[0060] For example, after the application server receives the source language text segment and multiple translation text segments returned by the AI model server in real time, it will classify them under the corresponding complete sentence according to the sentence ID. When a sentence is completely recognized, the system will generate a record containing the following fields and store it in the meeting record database: Sentence ID: SENT_20241105_001; Timestamp: 2023-11-05 10:05:30; Original text of the sentence (Chinese): 我们下一步要评估这个架构的瓶颈。 Our next step is to evaluate the bottleneck of this architecture.
[0061] English translation: Our next step is to evaluate the bottleneck of this architecture; Japanese translation: 私たちの次のステップは、このアーキテクチャのボトルネックを評価することです。 Our next step is to evaluate the bottleneck of this architecture.
[0062] The record text of the entire meeting is a data table or JSON file composed of such records sorted by timestamp, providing a unique and accurate source of facts for generating meeting minutes.
[0063] In some embodiments, the generation request sent by the application server to the AI model server contains a constraint instruction, which is used to limit the AI model server to only generate meeting minutes for the meeting record text.
[0064] Specifically, at the end of the meeting, the application server sends a constraint instruction (Prompt) to the AI model server, for example: You are a meeting minutes assistant. Please generate a concise meeting summary strictly and solely based on the following meeting transcript text, sorted by sentence ID and timestamp. Your task is to summarize the key points and decisions; adding any information not present in the transcript text or making any creative additions is prohibited. Without this instruction, the model might infer from its training data that the meeting may have discussed budget issues; however, with this instruction, the model will only summarize what is explicitly stated in the transcript, such as "Decision: The technical team will assess architectural bottlenecks and submit a report next week."
[0065] In some implementations, it also includes: (1) The meeting minutes generated by the AI model server are output in a hierarchical Markdown format; (2) The application server renders the meeting minutes in Markdown format into a meeting minutes mind map and distributes it to the user terminal.
[0066] Specifically, hierarchical Markdown is a lightweight markup language that uses different levels of headings (such as # for level 1, ## for level 2), lists, and other syntax to structure content and reflect the hierarchical relationship of information. Rendering here refers to the process of converting Markdown text code into visual graphic elements. A meeting minutes mind map is a way to visually represent the central theme, key points, and subtasks of meeting minutes in a radial, hierarchical graphical format.
[0067] For example, a Markdown-formatted summary generated by an AI model server might look like this: Figure 3 As shown, the application server calls a specialized database (such as markmap) to... Figure 3 The Markdown text shown is automatically converted into a visual mind map, as follows: Figure 4 As shown. Therefore, the user terminal will not receive a long text paragraph, but a clear graphic. The central node is "2024 Q1 Technology Planning Review Meeting", which extends to branches such as "Core Issues and Conclusions" and "Action Plan". Each branch has more detailed task and person in charge information.
[0068] In some implementations, the AI model server performs speech recognition on audio stream segments to obtain source language text segments as follows: (1) The AI model server receives audio stream segments; (2) The encoder encodes the current audio stream segment based on the encoding state after processing the previous audio stream segment, and updates the encoding state; (3) The decoder generates source language text segments in real time based on the encoding result of the current audio stream segment and the context of the preceding source language text segments.
[0069] Specifically, the encoder, part of the AI model server, is responsible for converting input audio stream segments into a series of mathematical representations (vectors) containing speech information. The encoded state is a hidden state generated by the encoder after processing the current audio segment, condensing all previous speech context information. It is passed to the next segment to maintain contextual coherence. The decoder, another part of the AI model, is responsible for autoregressively generating corresponding text segments based on the information output by the encoder.
[0070] For example, when the audio segment "architectural" is input, the encoder encodes it along with the encoded state left over from processing the previous segment "evaluate this". After encoding, the encoder generates a new encoded state containing all the speech information from "evaluate this" to "architectural" and passes it to the next segment "bottleneck". The decoder then uses the encoded result of "architectural" and the already generated preceding text "evaluate this" to infer and output the most likely current text "architectural".
[0071] In some implementations, it also includes: When the application server sends an audio stream segment to the AI model server, it also sends a list of target languages required for this translation task. The list of target languages is determined by the application server based on the languages selected by all user terminals in the current conference room.
[0072] Specifically, the target language list is a dynamically maintained list by the application server, containing the set of languages selected by all users in the current meeting room. The application server monitors meeting room members and their language selections in real time during the meeting. When a user speaks, the server immediately calculates the required language list (e.g., if there are Chinese, English, and Japanese users in the meeting room, the list would be [Chinese, English, Japanese]), and sends the audio stream segment along with this list to the AI model server.
[0073] For example, when the AI model server receives an audio clip containing the word "bottleneck" and a list [Chinese, English, Japanese], it outputs three results in parallel within a single computation: the original Chinese word " Nine-Bottleneck," the English word "bottleneck," and the Japanese word "ボトルネック." Without this list, the system might need to call the translation service three times, which is inefficient.
[0074] Please see Figure 5 As shown, in one embodiment, a multilingual accessible conference room system is provided, the system comprising: Multiple user terminals are used to connect to the application server via access links and send their respective selected target language information to the application server. The application server is used to create meeting room instances and generate corresponding access links; It receives audio data streams from user terminals in real time and segments them into continuous audio stream segments; Send audio stream segments to the AI model server; It receives text fragments translated by the AI model server, selects the corresponding language from the translated text fragments, and pushes them to the corresponding user terminals for display in real time; AI model server, which is used to translate source language text fragments into translated text fragments in multiple target languages in parallel through a single model inference, based on a list of target languages provided by the application server; Send the translated text fragment to the application server.
[0075] Specifically, such as Figure 5 As shown, multiple user terminal modules are positioned and function as the system's access and interactive front end. Multiple participants join the same virtual meeting room via a unified meeting room URL and select their desired target language on the client. The speaking user's terminal device captures microphone audio in real time and sends the compressed and segmented real-time audio stream to the application server via a WebSocket long connection. Other user terminals receive the translated text pushed by the application server in real time through the same WebSocket connection, synchronize according to sentence ID and timestamp, and dynamically display the translation on the screen.
[0076] Preferably, the system also includes a speech segment storage database, specifically designed to store segmented raw audio streams generated during user speech. During storage, each audio segment is associated with a unique sentence ID and timestamp, ensuring that each raw audio segment accurately corresponds to the relevant recognized text and translation result. When there is doubt about the recognition or translation results from the AI model server, the original audio segment can be quickly located and replayed based on the sentence ID for manual verification, ensuring the accuracy and seriousness of the meeting. Simultaneously, the massive, high-quality audio-text pairing data stored serves as a resource for incremental training of the AI model (speech recognition model), making the model increasingly accurate. When needed, the system can sequentially concatenate the stored audio segments based on the timestamp and sentence ID to generate a complete meeting recording file, meeting the needs for archiving or playback.
[0077] The application server acts as the system's scheduling and communication hub. It maintains a stable connection with all user terminals via a WebSocket gateway, acquiring audio stream segments in real time. Upon receiving the audio stream, the application server, acting as a coordinator, invokes the AI model server. It sends a request to the AI model server to perform streaming speech recognition and translation. After receiving the processing results (including the translation, sentence ID, and timestamp) from the AI model server, the application server executes two tasks in parallel: storage and push. Specifically, it sends the data to the speech segment storage and the meeting record database respectively. Simultaneously, it immediately pushes the translation results via WebSocket to user terminals that have selected the corresponding target language. This provides unified management of the meeting's lifecycle (creation, progress, and termination) and participant status.
[0078] The AI model server receives audio stream segments and translation requests from the application server. The AI model server comprises multiple model instances, which integrate speech recognition and translation processing. First, a streaming speech recognition model converts the audio into source language text segments. Then, a unified multilingual translation model generates translations for all target languages in parallel within a single inference iteration. This corresponds precisely to the description in the diagram of feeding the audio stream into a self-developed model for integrated processing. After processing, the structured results (including the text, unique sentence IDs, and timestamps) are returned to the application server.
[0079] The speech segment storage database serves as the raw audio archive for the conference. It receives storage instructions and data from the application server. Using sentence IDs as the core index, it stores the corresponding raw audio stream segments. This design enables precise association and rapid retrieval of audio and text, facilitating subsequent review, model training, or generation of conference recordings.
[0080] Preferably, it also includes a meeting record database, a structured text knowledge base for the meeting. It receives storage instructions and data from the application server. It stores complete, structured meeting text records, each record containing a sentence ID, the original text and all translations, timestamps, speaker information, etc., constituting a complete text log of the meeting. After the meeting, the meeting minutes model based on the RAG framework retrieves all meeting records from this database as the sole source of fact, summarizes and concludes them, generates the final meeting minutes, and stores them back in the database.
[0081] For example, when a user speaks, the audio stream flows from the user's terminal to the application server. The application server calls the AI model service for processing. The AI model service returns the results to the application server. Simultaneously, the application server sends storage requests to both the speech slice storage database and the meeting minutes database, and pushes the translation results to the target user's terminal in parallel. Ultimately, the structured text in the meeting minutes database provides a solid foundation for generating high-quality meeting minutes afterward.
[0082] This application allows participants to join meetings with a single link or QR code, eliminating the need for complex registration and configuration. This significantly reduces the barriers to meeting organization and time costs, achieving "simple and efficient" meeting organization. The system utilizes real-time speech recognition and multilingual parallel translation to provide each participant with a translation in their native or familiar language, fundamentally solving the core pain point of "participants speaking different languages not understanding each other," thus achieving truly "barrier-free" communication. The automatically generated hierarchical meeting minutes based on the RAG framework (such as Markdown and mind maps) ensure the accuracy and structure of the content, allowing participants to quickly review meeting decisions and to-do items, improving information utilization efficiency.
[0083] Employing streaming processing technology, the audio stream is segmented and processed. Combined with a high-efficiency streaming encoder architecture, this avoids the high latency of waiting for an entire sentence to finish speaking, achieving a millisecond-level response from user speech to translated text display, ensuring the natural fluency of the meeting. A unified AI model server design allows for parallel output of multilingual translations with a single model inference, significantly reducing computational resource consumption and processing latency compared to traditional solutions that serially call multiple translation engines. The combination of asynchronous message queues and a microservice architecture decouples the system modules, enabling smooth handling of high-concurrency scenarios and providing excellent scalability. Through a Retrieval Enhanced Generation (RAG) framework, the generation of meeting minutes is strictly limited to the actual recording of the meeting, fundamentally preventing information fabrication and ensuring the "factual fidelity" of the minutes, which is crucial for formal meetings.
[0084] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0085] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0086] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A method for implementing a multilingual, barrier-free conference room, characterized in that, The method is executed collaboratively by an application server, an AI model server, and multiple user terminals, and includes the following steps: The application server creates a meeting room instance and generates a corresponding access link. The user terminal connects to the application server through the access link and sends its selected target language information to the application server. The application server receives audio data streams from user terminals in real time and segments them into continuous audio stream segments. The application server sends the audio stream segment to the AI model server; The AI model server performs speech recognition on the received audio stream segments and generates corresponding source language text segments. The AI model server, based on the target language list provided by the application server, translates the source language text fragment into translated text fragments in multiple target languages in parallel through a single model inference. The AI model server sends the translated text fragment to the application server; The application server selects the corresponding language from the translated text fragments based on the target language selected by each user terminal, and pushes it to the corresponding user terminal for display in real time.
2. The method for implementing a multilingual, barrier-free conference room according to claim 1, characterized in that, The application server and the AI model server exchange data through an asynchronous message queue to decouple the request and processing processes.
3. The method for implementing a multilingual, barrier-free conference room according to claim 1, characterized in that, Both the source language text fragment and the translated text fragment are associated with corresponding sentence IDs and timestamp information.
4. The method for implementing a multilingual, barrier-free conference room according to claim 1, characterized in that, Also includes: The application server combines the source language text fragments and their corresponding multiple translated text fragments to form meeting record texts in different languages, and stores them in the meeting record database; Upon detecting the end of the meeting, the application server retrieves the meeting record text from the meeting record database and instructs the AI model server to generate meeting minutes in different languages based on the meeting record text; The application server distributes the generated meeting minutes to the corresponding user terminals.
5. The method for implementing a multilingual accessible conference room according to claim 4, characterized in that, The meeting minutes text consists of sentences arranged in chronological order, with each sentence containing the original text, translations in multiple languages, a sentence ID, and a timestamp.
6. The method for implementing a multilingual, barrier-free conference room according to claim 4, characterized in that, The generation request sent by the application server to the AI model server includes a constraint instruction, which restricts the AI model server to generate meeting minutes only from the meeting record text.
7. The method for implementing a multilingual accessible conference room according to claim 4, characterized in that, Also includes: The meeting minutes generated by the AI model server are output in a hierarchical Markdown format; The application server renders the Markdown format meeting minutes into a meeting minutes mind map and distributes it to user terminals.
8. The method for implementing a multilingual accessible conference room according to claim 1, characterized in that, The AI model server performs speech recognition on the received audio stream segments and generates corresponding source language text segments: The AI model server receives audio stream segments. The encoder encodes the current audio stream segment based on the encoding state of the previous audio stream segment and updates the encoding state. The decoder generates source language text segments in real time based on the encoding results of the current audio stream segment and the context of the preceding source language text segments.
9. The method for implementing a multilingual accessible conference room according to claim 1, characterized in that, Also includes: When the application server sends the audio stream segment to the AI model server, it also sends a list of target languages required for this translation task. The target language list is determined by the application server based on the language selected by all user terminals in the current conference room.
10. A multilingual accessible conference room system, used to implement the multilingual accessible conference room implementation method as described in any one of claims 1-9, characterized in that, include: Multiple user terminals are used to connect to the application server via an access link and send their respective selected target language information to the application server. The application server is used to create meeting room instances and generate corresponding access links; It receives audio data streams from user terminals in real time and segments them into continuous audio stream segments; The audio stream segment is sent to the AI model server; The system receives translated text fragments from an AI model server, selects the corresponding language from the translated text fragments, and pushes them to the corresponding user terminals for display in real time. The AI model server is used to translate the source language text fragment into multiple target language text fragments in parallel through a single model inference, based on the target language list provided by the application server. The translated text fragment is sent to the application server.