Cross-end translation method, apparatus and electronic device

Abstract

The application provides a cross-terminal translation method and device and electronic equipment, and relates to the technical field of computers. According to a translation request initiated by a plurality of clients, the application determines an application scenario of a translation task. According to the application scenario, an RTC (real-time communication) room is created, and a translation instance is allocated to the RTC room. An audio stream of at least one client is received through a communication channel of the RTC room. The audio stream is translated according to a translation model configured in the translation instance, and the translation result is sent to a plurality of clients accessing the RTC room through the communication channel of the RTC room. Based on the RTC channel technology, the application realizes seamless cross-terminal operation, effectively improves the stability and real-time performance of data transmission in the cross-terminal translation process, and thus improves the interactive experience of users. Meanwhile, different translation models are used for translation according to different application scenarios, so that the adaptability of the translation model to the application scenario is ensured, and the accuracy of translation is improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more particularly to a cross-platform translation method, apparatus, and electronic device. Background Technology

[0002] Currently, people have an increasingly urgent need for efficient and natural real-time cross-language communication tools. XR (Extended Reality) smart glasses, due to their unique interactive advantage of seamlessly integrating digital information with the real world, are regarded as the core terminal for the next generation of human-computer interaction, showing great application potential in building immersive cross-language communication environments.

[0003] However, when applying XR smart glasses to real-world translation scenarios, especially complex and ever-changing real-time interactive environments, existing technologies have significant shortcomings and struggle to meet user needs. Existing technologies often employ a one-size-fits-all, generic translation model, failing to provide refined processing. Furthermore, most existing solutions are closed systems within single devices or homogeneous devices, unable to achieve real-time state synchronization and command coordination between different terminals. Therefore, in complex multi-terminal, multi-scenario applications, existing technologies cannot simultaneously guarantee translation accuracy, real-time performance, and the synchronization and smoothness of interaction between terminals. Summary of the Invention

[0004] This invention provides a cross-platform translation method, apparatus, and electronic device to address the shortcomings of existing technologies in terms of low translation accuracy and poor real-time performance under complex multi-platform and multi-scenario applications, thereby improving translation accuracy and real-time performance under complex multi-platform and multi-scenario applications.

[0005] This invention provides a cross-platform translation method, comprising the following steps:

[0006] Based on translation requests initiated by multiple clients, determine the application scenarios for the translation tasks;

[0007] Based on the application scenario, create a real-time communication (RTC) room and assign translation instances to the RTC room;

[0008] The audio stream from at least one of the clients is received through the communication channel of the RTC room;

[0009] The audio stream is translated according to the translation model configured in the translation example, and the translation result is sent to multiple clients accessing the RTC room through the communication channel of the RTC room.

[0010] According to a cross-platform translation method provided by the present invention, the translation model is configured in the following manner:

[0011] Based on the application scenario, configure a translation model that matches the application scenario for the translation instance.

[0012] According to a cross-platform translation method provided by the present invention, configuring a translation model matching the application scenario for the translation instance includes:

[0013] Parse the metadata of the application scenario, which includes at least one of the following: business domain, text genre, terminology database identifier, and target user group;

[0014] Based on the metadata, one or more specialized translation models are selected from a pre-built model repository;

[0015] The selected translation models are loaded into the runtime environment of the translation instance.

[0016] According to a cross-platform translation method provided by the present invention, translating the audio stream based on a translation model configured in the translation instance includes:

[0017] The audio stream is preprocessed, including noise reduction, endpoint detection, and normalization, to obtain clean speech data.

[0018] The clean speech data is streamed using an Automatic Speech Recognition (ASR) model, and the recognized text is optimized using a large language model to obtain the source language text.

[0019] The source language text is input into the translation model for translation, and the translation result is obtained.

[0020] According to a cross-platform translation method provided by the present invention, the application scenario includes at least one of a dialogue scenario, a listening and translation scenario, a speech scenario, and a group chat scenario; the translation instance is also configured with dedicated processing logic matching the application scenario, wherein:

[0021] The dedicated processing logic configured for the dialogue scenario includes activating a recording control mechanism based on the voice wake-up word and an interruption response mechanism.

[0022] The dedicated processing logic configured for the listening and translation scenario includes enabling enhanced speech signal processing algorithms;

[0023] The dedicated processing logic configured for the speech scenario includes enabling integrated processing of visual aid information to optimize translation output;

[0024] The dedicated processing logic configured for the group chat scenario includes the separation and sorting of multiple audio streams based on sound source localization.

[0025] According to a cross-platform translation method provided by the present invention, the step of creating a real-time communication (RTC) room based on the application scenario includes:

[0026] A room identifier is generated based on the application scenario and the translation language selected by each client.

[0027] The RTC room is created based on the room identifier.

[0028] According to a cross-platform translation method provided by the present invention, generating a room identifier based on the application scenario and the translation language selected by each client includes:

[0029] Map the application scenario to a scenario code;

[0030] The translation languages ​​selected by each client are sorted and concatenated to form a language sequence encoding;

[0031] The scene code and the language sequence code are combined according to a predetermined format to generate the room identifier.

[0032] According to a cross-platform translation method provided by the present invention, the step of sending the translation result to multiple clients accessing the RTC room through the communication channel of the RTC room includes:

[0033] Based on the type of each client, the translation result is converted into a data format that matches the type of the client;

[0034] The translation results are sent to multiple clients connected to the RTC room via the communication channel of the RTC room.

[0035] The present invention also provides a cross-platform translation device, comprising the following modules:

[0036] The application scenario determination module is used to determine the application scenario of the translation task based on translation requests initiated by multiple clients.

[0037] The RTC room creation module is used to create real-time communication (RTC) rooms according to the application scenario and to assign translation instances to the RTC rooms.

[0038] An audio stream receiving module is used to receive an audio stream from at least one of the clients through the communication channel of the RTC room;

[0039] The translation module is used to translate the audio stream according to the translation model configured in the translation instance, and send the translation result to multiple clients accessing the RTC room through the communication channel of the RTC room.

[0040] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the cross-end translation method as described above.

[0041] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cross-end translation method as described above.

[0042] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the cross-end translation method as described above.

[0043] The cross-platform translation method, apparatus, and electronic device provided by this invention determine the application scenario of the translation task based on translation requests initiated by multiple clients; create a Real-Time Communication (RTC) room according to the application scenario and allocate translation instances to the RTC room; receive audio streams from at least one client through the communication channel of the RTC room; translate the audio streams according to the translation model configured in the translation instance, and send the translation results to multiple clients connected to the RTC room through the communication channel of the RTC room. This invention, based on RTC channel technology, achieves seamless cross-platform operation, effectively improving the stability and real-time performance of audio and text data transmission during cross-platform translation, thereby enhancing the user's interactive experience; simultaneously, it uses different translation models for different application scenarios, ensuring the adaptability of the translation model to the application scenario and improving translation accuracy. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0045] Figure 1 This is an application environment diagram of the cross-end translation method provided by the present invention.

[0046] Figure 2 This is a flowchart illustrating the cross-platform translation method provided by the present invention.

[0047] Figure 3 This is a schematic diagram of the translation process in an international group chat scenario provided by the present invention.

[0048] Figure 4 This is a schematic diagram of the cross-end translation device provided by the present invention.

[0049] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0051] Currently, translation technology based on XR smart glasses still faces many problems that need to be solved in practical applications.

[0052] First, there are serious shortcomings in adaptability to complex scenarios and translation experience. Current technology lacks the ability to intelligently recognize and differentiate between different scenarios such as dialogues, speeches, and group chats. In dialogue scenarios, existing solutions mostly adopt batch processing translation models that aim for final accuracy. The process of generating intermediate results is lengthy, resulting in frequent "frame skipping" or a single intermediate result being broken into multiple disjointed sentences when the translation is presented in the limited display area of ​​XR glasses. This severely disrupts the coherence and naturalness of communication, causing dizziness and discomfort for the wearer. In listening and translation and group chat scenarios, existing audio processing algorithms struggle to effectively separate the target sound source from background noise or distinguish multiple overlapping speakers, leading to a sharp drop in speech recognition accuracy and resulting in translation errors. This makes it impossible to provide reliable service in multi-person discussions or noisy environments.

[0053] Secondly, there is a lack of efficient collaborative translation capabilities supporting multiple devices and cross-terminals. In real-world scenarios such as international conferences and team collaborations, it is often necessary for multiple XR glasses users and other parties participating through mobile applications and H5 web pages to simultaneously receive consistent translation content. Existing solutions are mostly limited to simple interactions between single devices or homogeneous devices, lacking a unified, low-latency, and high-concurrency communication architecture to support real-time data synchronization and command coordination between multiple XR terminals and heterogeneous terminals (such as Apps and H5 pages). This significantly limits its application value in large-scale cross-language collaboration.

[0054] Furthermore, the underlying communication architecture struggles to support a high-quality real-time interactive experience. Most existing translation workflows are based on simple, traditional communication protocols such as WebSocket. These protocols have inherent limitations in handling real-time audio and video streaming, large-scale concurrency control, and stability in weak network environments. Their high latency and susceptibility to packet loss cannot meet the millisecond-level synchronization and ultra-smooth interactive experience required for real-time translation, thus becoming a bottleneck for improving service quality.

[0055] Based on the aforementioned problems, those skilled in the art have discovered that introducing RTC (Real-Time Communication) technology and employing cloud-based translation offers significant advantages. RTC technology effectively improves the stability and real-time performance of audio, text, and other data transmissions, greatly enhancing the interactive experience. However, related technologies have not yet adopted this approach to optimize the translation workflow for XR smart glasses.

[0056] This invention aims to solve a series of technical problems faced by XR smart glasses in cross-platform (XR smart glasses, APP, mobile web page H5) and multi-national environments, for translation in multiple scenarios such as dialogue, listening and interpretation, speech, and group chat, so as to improve the visual experience on the glasses.

[0057] Specifically, the first step is to address the issues of cross-platform data interaction and functional collaboration, ensuring that XR smart glasses can efficiently and accurately transmit and share translation-related data with apps and H5 pages using RTC technology, achieving seamless cross-platform operation. By optimizing data format conversion, communication protocols, and middleware technology, data interaction between different platforms can be made smoother, avoiding data loss, delays, or errors.

[0058] Secondly, to meet the diverse needs of various scenarios such as dialogue, interpretation, speeches, and group chats, translation technology requires targeted optimization. Specifically, in dialogue scenarios, the focus is on optimizing real-time performance and naturalness, ensuring that translation results are presented promptly and naturally without disrupting the flow of communication; in interpretation scenarios, the accuracy of speech recognition and translation is improved to reduce mistranslations; in speeches, richer and more accurate translations are achieved by incorporating contextual information and voiceprint recognition, and then displayed appropriately on XR smart glasses; in group chat scenarios, technologies such as multi-source sound recognition and intelligent sorting ensure that each speaker's content is accurately translated and displayed in an orderly manner.

[0059] Finally, we are committed to improving the visual experience on XR smart glasses. By studying user visual habits and the display characteristics of XR smart glasses, we optimize the visual presentation of translated information, including adjusting text size and color contrast, optimizing information layout, so that the translated content can be better integrated with the real scene, improving the efficiency and comfort of users in obtaining information, and reducing visual fatigue.

[0060] The following is combined Figures 1-5 The present invention describes a cross-end translation method, apparatus, and electronic device.

[0061] The cross-end translation method provided by this invention can be applied to, for example... Figure 1In the application environment shown, terminal 20 communicates with server 10 via a network. A data storage system can store the data that server 10 needs to process; this system can be integrated onto server 10 or located in the cloud or on another network server. Server 10 can determine the application scenario of the translation task based on translation requests initiated by multiple clients; based on the application scenario, it creates a Real-Time Communication (RTC) room and allocates translation instances to the RTC room; it receives audio streams from at least one client through the communication channel of the RTC room; it translates the audio streams according to the translation model configured in the translation instance, and sends the translation results to multiple clients connected to the RTC room through the communication channel of the RTC room. By using RTC channel technology, seamless cross-platform operation is achieved, effectively improving the stability and real-time performance of audio and text data transmission during cross-platform translation, thereby enhancing the user's interactive experience; simultaneously, different translation models are used for translation according to different application scenarios, ensuring the adaptability of the translation model to the application scenario and improving the accuracy of the translation. Terminal 20 can send the collected audio stream to server 10, which translates the audio stream and returns the translation result to terminal 20. Terminal 20 then displays the translation result based on the user's current language source. Terminal 20 can be XR smart glasses, an app, or an H5 page.

[0062] It should be noted that the execution entity of the cross-platform translation method provided by this invention is a server, such as a cloud server or a backend server.

[0063] Figure 2 This is a flowchart illustrating the cross-platform translation method provided by the present invention, as follows: Figure 2 As shown, the method includes the following:

[0064] Step 101: Determine the application scenario of the translation task based on the translation requests initiated by multiple clients.

[0065] The server establishes RTC connections with multiple clients, including at least XR smart glasses, and mobile terminal apps and H5 web clients associated with the XR smart glasses. The server receives translation requests from clients, which contain application scenario identifiers selected by the user. Based on these identifiers, the server determines the application scenario for the current translation task. Specifically, the server directly reads the scenario parameter values, explicitly selected by the user on the front-end interface, from the translation request packet and parses them into a specific scenario type according to a predefined mapping relationship.

[0066] For example, within the same main application on the client side (such as an XR translation assistant), there are multiple different function entry points or mode selection interfaces. After a user opens the translation application on their XR smart glasses, they will see a main interface with multiple icons or buttons: conversation mode for one-on-one communication; group chat mode for multi-person chat; speech mode for listening to and translating or watching speeches; AR subtitle mode for watching foreign language videos, etc. Assuming the user clicks the group chat mode icon, the client (XR smart glasses / App) will send a request to the backend server to create a translation session, carrying a scene parameter, such as "scene_type": "group_chat". After receiving the translation request, the backend server directly reads the scene_type field, which determines that the application scenario of the current translation task is a group chat scenario.

[0067] Optionally, the application scenario of the translation task can also be determined in the following ways: when two terminals are connected in the RTC room and the audio streams of the two terminals show alternating transmission characteristics (single audio duration ≤ 10 seconds, interaction interval ≤ 2 seconds), it is determined to be a listening and translation scenario; when there is one device marked as a speaker terminal in the RTC room and the audio stream sent by the terminal shows continuous transmission characteristics (single audio duration ≥ 30 seconds), it is determined to be a speech scenario; when there are ≥ 3 terminals connected in the RTC room and the audio streams of the terminals show random transmission characteristics (multiple terminal audio streams briefly overlap), it is determined to be a group chat scenario.

[0068] Step 102: Based on the application scenario, create a real-time communication (RTC) room and assign translation instances to the RTC room.

[0069] It should be understood that the RTC room is not a physical space, but a multi-terminal collaborative logical unit created by the backend server based on RTC technology. It is used to uniformly manage cross-terminal connections, data routing and communication parameters in specific application scenarios. Its core function is to enable multiple terminals (XR smart glasses, App, H5) in the same scenario to efficiently transmit translation-related data (audio stream, translation results).

[0070] It should be understood that a translation instance is a translation processing resource unit deployed on the backend server that can be independently scheduled and started and stopped on demand. It includes the AI ​​model, data processing module, and resource quota required to complete the translation of a specific scenario. Its core function is to receive the audio stream transmitted from the RTC room, perform scenario-based translation (such as low-latency processing for listening comprehension and contextual processing for group chats), and return the translation result.

[0071] The server internally stores a "scene-configuration" mapping table. For example, the RTC room for a dialogue scenario is configured as follows: set to two-person mode, prioritize optimizing end-to-end latency, and video stream can be turned off to save bandwidth; the RTC room for a group chat scenario is configured as follows: set to multi-person mode, enable speaker detection and audio stream prioritization to better handle multiple people speaking simultaneously; the RTC room for a speech scenario is configured as follows: set to broadcast mode, make the audio stream from the XR smart glasses the primary source, and other clients are only subscribers (receiving but not sending). The server calls the RTC service provider's API, passing in the above configuration parameters, to create an RTC room with the corresponding technical characteristics.

[0072] While creating the RTC room, the server notifies the resource manager (such as Kubernetes) to launch a new instance as a translation instance, such as dynamically launching a container on a cloud platform; or to allocate an idle instance from an instance pool, such as selecting one from a pre-created instance pool. The server sends an initialization configuration command to the translation instance, which contains key information such as the scene type. After receiving the command, the translation instance loads the corresponding AI model combination from the model repository (e.g., loading an ASR (Automatic Speech Recognition) model that supports speaker separation for the group_chat scene). The server treats the translation instance as a special virtual client and connects it to the created RTC room. Based on this, the translation instance can subscribe to the audio streams of all real clients in the RTC room and publish translation results to the room through the communication channel.

[0073] Step 103: Receive audio streams from at least one of the clients through the communication channel of the RTC room.

[0074] It should be understood that the communication channels in an RTC room contain at least two types of channels:

[0075] Media Channel / Audio Channel: A channel specifically designed for transmitting audio and video streams. It is characterized by being highly optimized and having the highest priority to ensure real-time performance.

[0076] Data channel: A channel specifically used for transmitting non-audio / video data such as text and control commands.

[0077] The RTC room receives audio streams from at least one client via its media / audio channel. For example, the microphone of an XR smart glasses or mobile app captures raw sound (PCM format). The client compresses and encodes the raw audio data (e.g., converting it to OPUS format) to significantly reduce data size for network transmission. The encoded audio data is packaged into network packets and sent to the RTC server via the established RTC media channel. The RTC server acts as a hub, receiving audio packets from clients and routing / forwarding them to specific subscribers within the same RTC room based on the room identifier within the audio packets. Translation instances are virtual subscribers; they subscribe to all client audio streams from the RTC server. This means the RTC server copies all audio packets within the RTC room and sends them to the translation instance.

[0078] Step 104: Translate the audio stream according to the translation model configured in the translation instance, and send the translation result to multiple clients accessing the RTC room through the communication channel of the RTC room.

[0079] The translation instance on the server continuously receives audio streams from the client via the RTC media channel and calls its internally configured translation model for translation processing. This translation processing mainly includes: converting the audio stream into source language text in real time, translating the source language text into target language text in real time, and generating the final translation result based on the target language text. Optionally, the translation result can carry metadata such as timestamps and speaker IDs.

[0080] The translation example encapsulates the translation result into a data packet (such as JSON format) and sends this packet to the RTC server via a pre-established data channel in the RTC room. It should be understood that a data channel, rather than a media channel, is used here because the translation result is text information, and the data channel provides high reliability and orderliness guarantees for text transmission, ensuring that the text is not lost or corrupted.

[0081] After receiving the data packet, the RTC server broadcasts it to all clients connected to the room. Upon receiving the data packet, each client (such as XR smart glasses, apps, and H5 pages) parses the translated text and renders it in the most appropriate way according to its own UI characteristics and user settings. For example, XR smart glasses anchor the translation as a floating subtitle next to the speaker, mobile apps display it as contextual bubbles in the chat interface, and H5 pages present it as a scrolling subtitle bar.

[0082] The cross-platform translation method provided in this invention determines the application scenario of the translation task based on translation requests initiated by multiple clients; creates a Real-Time Communication (RTC) room according to the application scenario and assigns translation instances to the RTC room; receives audio streams from at least one client through the communication channel of the RTC room; translates the audio streams according to the translation model configured in the translation instance, and sends the translation results to multiple clients connected to the RTC room through the communication channel of the RTC room. This invention, based on RTC channel technology, achieves seamless cross-platform operation, effectively improving the stability and real-time performance of audio and text data transmission during cross-platform translation, thereby enhancing the user's interactive experience; simultaneously, it uses different translation models for different application scenarios, ensuring the adaptability of the translation model to the application scenario and improving translation accuracy.

[0083] Based on the above embodiments, the translation model is configured in the following manner:

[0084] Based on the application scenario, configure a translation model that matches the application scenario for the translation instance.

[0085] It should be understood that a translation model is a combination of multiple AI models optimized for specific scenarios. This combination encompasses all AI models involved in the entire functional chain from audio to target language text. This AI model combination includes at least an ASR (Automatic Speech Recognition) model and a NMT (Neural Machine Translation) model. For different application scenarios, the ASR and NMT models are configured with specific optimized features. For example, in dialogue scenarios, the lightweight ASR model has extremely fast inference speed and focuses on near-field, clear speech; the conversational NMT model excels at handling short sentences, ellipses, and question-and-answer structures, producing natural and fluent translations with low latency, avoiding communication stutters. In speech / listening-to-translation scenarios, the noise-resistant ASR model's training data contains a large amount of noise and integrates a speech enhancement module; the formal text NMT model excels at handling long and complex sentences and complex logic, outputting translations with rigorous structure and formal vocabulary, thereby improving recognition rates in noisy environments and ensuring the accuracy and coherence of long text translations. In group chat scenarios, ASR models that support speaker separation can distinguish the voices of different speakers and label the voices coming from different directions; NMT models that are context-aware can understand the context of multi-person dialogues, solve the referential problem, and thus ensure the coherence of the context of multi-person dialogues.

[0086] Once the server creates an RTC room based on the application scenario, it immediately sends a configuration command to the translation instance assigned to that RTC room. This configuration command includes an application scenario identifier, such as scene_type: "group_chat". Upon receiving the configuration command, the translation instance, based on the value of the scene_type field, pulls the corresponding AI model file from a central model repository. This AI model is pre-trained and has been thoroughly trained and optimized for scenario-specific data. After successful retrieval, the model is loaded into the memory of a computing device (such as a GPU), completing initialization.

[0087] This invention achieves a shift from general-purpose to scenario-specific translation models by dynamically configuring dedicated translation models for different application scenarios. This results in synergistic optimization in translation accuracy, system real-time performance, and user experience immersion, solving the core problems of poor translation quality, high latency, and awkward interaction caused by a single general-purpose model in complex multi-scenario applications.

[0088] Based on the above embodiments, configuring a translation model that matches the application scenario for the translation instance includes:

[0089] Parse the metadata of the application scenario, which includes at least one of the following: business domain, text genre, terminology database identifier, and target user group;

[0090] Based on the metadata, one or more specialized translation models are selected from a pre-built model repository;

[0091] The selected translation models are loaded into the runtime environment of the translation instance.

[0092] It should be understood that metadata is used to describe the nature, background, and requirements of the translation task itself. It defines the context in which the translation work takes place and is prior information that is determined before or at the start of the translation task. Its core role is to provide a basis for decision-making in intelligently selecting the most suitable translation tool (model).

[0093] Metadata includes at least one of the following: business domain, text genre, terminology database identifier, and target user group. Specifically, business domain refers to the professional knowledge area or industry to which the text to be translated belongs; text genre refers to the stylistic form and functional purpose of the text to be translated; terminology database identifier refers to one or more pre-built, standardized sets of terms that must be followed in this context; and target user group refers to the final readership of the translated text.

[0094] A user configuration interface is provided, containing controls for inputting or selecting the business domain, text genre, terminology database, and target user group. This interface receives metadata explicitly specified by the user. Alternatively, the source text to be translated can be obtained and analyzed using a pre-trained classification model to infer its business domain and / or text genre. Furthermore, the upstream application or digital environment invoked by the translation instance can be identified, and corresponding metadata can be extracted from the attributes of the upstream application or digital environment according to predefined mapping rules.

[0095] Each translation model in the model repository is assigned one or more pre-defined attribute tags, which describe the business domain, text genre, and applicable user group that the model excels in. The metadata of the application scenario is matched with the attribute tags to filter out all translation models whose tags completely match the metadata. Alternatively, each translation model in the model repository can be assigned one or more pre-defined attribute tags, which describe the business domain, text genre, and applicable user group that the model excels in; the metadata of the application scenario is matched with the attribute tags to filter out all translation models whose tags completely match the metadata. Alternatively, multiple performance metrics of candidate translation models in historically similar metadata scenarios can be obtained, including translation accuracy, terminology consistency, fluency, and inference speed; according to a predefined weighting rule, multiple performance metrics are weighted to obtain a comprehensive performance score; candidate models are ranked based on the comprehensive performance score, and one or more models with the highest ranking are selected.

[0096] Pull a container image containing the translation model from the model repository. In the runtime environment of the translation instance, instantiate the container image to start a model inference service and connect the translation instance to the application programming interface of the model inference service. Alternatively, download the computation graph structure file and model weight file of the translation model from the model repository. In the runtime environment's memory, reconstruct the model architecture based on the computation graph structure file, load the model weight file to initialize the model parameters, and set the initialized model to a ready state to receive inference requests from the translation instance.

[0097] Optionally, a pre-built translation model performance profile can also be obtained, which describes the comprehensive performance index of each model under different business domains and text styles; the characteristics of the application scenario are matched with the performance profile, the translation model with the best comprehensive performance index is selected, the selected model is instantiated, and the computing resources required for its operation are configured.

[0098] Optionally, a general basic translation model can be configured for the translation instance to identify the specific domain corresponding to the application scenario, load the incremental parameter matrix corresponding to the specific domain, and superimpose the incremental parameter matrix with the general basic translation model to form a dedicated translation model optimized for that domain.

[0099] Optionally, resource constraint information of the target environment where the translation instance is located can also be obtained, including at least one of the following: available memory, processor computing power, and network bandwidth; based on the resource constraint information, a model version that achieves a balance between performance and resource consumption can be selected from the same model set for configuration.

[0100] This invention, through parsing and defining the scene metadata of the translation task, intelligently selects and loads the most suitable specialized translation model from the model repository, thereby achieving precise and dynamic matching between translation resources and translation task requirements. This ensures a high degree of consistency between the translation output and the target audience's habits in terms of professional terminology, writing style, and target audience before the task starts, improving the accuracy, professionalism, and applicability of the translation results and overcoming the shortcomings of general translation models in terms of quality in professional scenarios.

[0101] Based on the above embodiments, translating the audio stream according to the translation model configured in the translation instance includes:

[0102] The audio stream is preprocessed, including noise reduction, endpoint detection, and normalization, to obtain clean speech data.

[0103] The clean speech data is streamed using an Automatic Speech Recognition (ASR) model, and the recognized text is optimized using a large language model to obtain the source language text.

[0104] The source language text is input into the translation model for translation, and the translation result is obtained.

[0105] It should be understood that raw audio streams (such as speech captured by XR smart glasses) often contain interference such as environmental noise, silent segments, and volume fluctuations. Directly inputting these into an ASR model will lead to a significant drop in recognition accuracy. The core objective of preprocessing is to filter out interference and refine the speech. Preprocessing operations mainly include:

[0106] Noise Reduction: The original audio stream may contain ambient noise (such as echoes in a lecture, street noise in a translation scenario, and keyboard clicks in a group chat). This noise can mask speech features, leading to misidentification by the ASR model. Therefore, noise reduction processing is necessary for the original audio stream. For example, filters (such as adaptive filters) or deep learning algorithms (such as spectral subtraction) can be used to remove ambient noise (such as background conversations and device noise) to highlight the target speech signal. Taking deep learning algorithms as an example, the algorithm first analyzes segments of the audio stream without speech (such as the silence period before speaking), extracts the frequency features of the noise (such as the low-frequency persistence features of street noise and the high-frequency pulse features of keyboard clicks), and for segments containing speech, the AI ​​model distinguishes between speech features and noise features, filtering out signals that match the noise features and retaining the pure speech.

[0107] Endpoint detection: The raw audio stream contains a large number of silent segments (such as pauses between speech or blank spaces while waiting for a response). Inputting these segments into the ASR model would waste computational resources (identifying "no content") and might even cause the ASR model to misjudge them as low-volume speech. Therefore, endpoint detection is needed for the raw audio stream. For example, the energy (volume level) of the audio frames is calculated. If the energy is lower than a preset threshold (such as the average energy during a silent period), it is initially determined to be silent. For frames with energy close to the threshold, the frequency distribution is further analyzed (speech frequencies are concentrated in the 300-3400Hz range, while silence frequencies are more dispersed) to confirm whether it is speech. The continuous speech frames are then concatenated into a speech segment, silent frames are removed, and the output is a segment containing only speech.

[0108] Normalization: The volume in the original audio stream may fluctuate (e.g., the volume increases as the speaker approaches the microphone and decreases as they move away; in a translation scenario, the user's speech may vary in volume). The ASR model is sensitive to volume, and unstable volume can lead to fluctuations in recognition accuracy (e.g., low-volume speech is recognized as ambiguous text). Therefore, it is necessary to normalize the original audio stream. For example, parameters such as the audio volume and sampling rate (16000Hz) should be standardized to avoid processing deviations caused by differences in devices. Specifically, the average volume of the entire speech segment should be calculated. If the average volume is lower than the target value, the gain should be increased (amplified); if it is higher than the target value, the gain should be decreased (decreased), ensuring that the volume is stable after adjustment.

[0109] After preprocessing the audio stream to obtain clean speech data, streaming recognition and LLM (Large Language Model) optimization are performed on the clean speech data. It should be understood that cross-platform translation (especially in listening comprehension and speech scenarios) requires real-time processing; the translation result needs to be available as soon as the user finishes speaking, without waiting for the entire speech segment to end (otherwise, the latency would be too high, affecting interaction). Therefore, streaming recognition is necessary for the clean speech data. However, streaming recognition results may contain issues such as homophones, incomplete grammar, and contextual ambiguity. These problems can lead to incorrect outputs from subsequent translation models. Therefore, an LLM model is needed to optimize the streaming recognition results.

[0110] Specifically, in the process of streaming recognition, the intermediate recognition text corresponding to a speech segment is output by the ASR model, and the LLM model is called to optimize the intermediate recognition text. This optimization may include word sense disambiguation and / or correction of homonym errors based on context. The optimized intermediate recognition text is then concatenated in sequence to form the source language text.

[0111] After ASR streaming recognition and LLM optimization, the source language text is input into the translation model for translation, yielding the translation results. For example, based on the target language selection of each client (e.g., English for XR smart glasses, French for apps, and Spanish for H5), the translation model generates multilingual translation results in parallel (or sequentially according to client requirements to avoid resource waste), while simultaneously processing the translation results to adapt to the display needs of different devices.

[0112] This invention achieves end-to-end optimization from raw audio to high-quality translation by constructing a collaborative processing system that includes signal cleansing, streaming ASR, LLM optimization, and accurate translation. The preprocessing module effectively improves the audio signal quality, laying a solid foundation for recognition; streaming ASR ensures extremely low latency in translation; and the introduction of LLM to optimize the recognized text fundamentally solves the common problems of error accumulation and contextual breaks in real-time speech recognition, significantly improving the accuracy of the source language text; finally, a dedicated translation model outputs fluent translation, thus simultaneously achieving a leapfrog improvement in translation speed, accuracy, and fluency.

[0113] Based on the above embodiments, the application scenarios include at least one of dialogue scenarios, listening and translation scenarios, speech scenarios, and group chat scenarios; the translation instance is also configured with dedicated processing logic matching the application scenarios, wherein:

[0114] The dedicated processing logic configured for the dialogue scenario includes activating a recording control mechanism based on the voice wake-up word and an interruption response mechanism.

[0115] The dedicated processing logic configured for the listening and translation scenario includes enabling enhanced speech signal processing algorithms;

[0116] The dedicated processing logic configured for the speech scenario includes enabling integrated processing of visual aid information to optimize translation output;

[0117] The dedicated processing logic configured for the group chat scenario includes the separation and sorting of multiple audio streams based on sound source localization.

[0118] When a translation instance is created, the corresponding dedicated logic module will be automatically loaded according to the application scenario identifier of the current translation task. This module will work in conjunction with core modules such as the translation model and audio preprocessing to form a complete scenario-based processing chain.

[0119] The dedicated processing logic for each application scenario is designed around the core pain points of that application scenario:

[0120] (1) The dialogue scenario is a one-to-one real-time interaction, which requires solving two main problems: determining the start / stop time for recording (it cannot record continuously, wasting resources and potentially recording irrelevant content); and how to pause one's own recording when the other party is speaking (to avoid both parties speaking at the same time, causing audio mixing and translation errors). Based on this, the translation instance needs to load the dialogue interaction control module, which includes two sub-mechanisms:

[0121] The voice wake-up word recording control mechanism aims to trigger recording start / stop via the wake-up word, replacing manual operation (XR smart glasses lack a touchscreen, making manual operation inconvenient). The logic is as follows: predefine wake-up words and train a lightweight wake-up word recognition model; when the translation instance is in standby mode, only wake-up word recognition is initiated (without full recording and translation), listening to the user's voice; when a wake-up word for starting translation is detected, recording and real-time translation are automatically started; when a wake-up word for stopping or exiting translation is detected, recording is immediately stopped, releasing resources.

[0122] The interruption response mechanism automatically pauses the current user's recording when it detects the other party starting to speak, ensuring that the audio does not overlap. The logic is as follows: The translation instance receives audio streams from both clients via the RTC room; it analyzes the energy values ​​(volume levels) of the two audio streams in real time: when user A's audio energy is consistently greater than or equal to a threshold (e.g., -40dBFS, indicating speaking), while user B's audio energy suddenly jumps from below the threshold to above or equal to the threshold (indicating starting to speak), it is determined that user B has interrupted user A; a pause recording command is immediately sent to user A's client, while user B's recording and translation continue; when user B stops speaking (energy < threshold), a resume recording command is sent to user A.

[0123] Through these two mechanisms, no manual operation is required, improving the fluency of the conversation (users do not need to frequently raise their hands to click the physical buttons on the XR smart glasses); at the same time, it can avoid audio overlap between the two parties, improve the ASR model recognition accuracy, and make the translation results more accurate.

[0124] (2) Listening and translation scenarios are mostly one-on-one or one-to-many listening and translation in non-ideal environments (such as XR smart glasses users listening to foreign speeches in noisy venues or airport announcements). The core problem is poor speech quality, such as environmental noise (e.g., venue echoes, airport announcement noise) and weak speech signals (e.g., the other party speaking softly), resulting in low ASR model recognition accuracy and distorted translation results. Based on this, the translation instance needs to load an enhanced speech processing module, adding multiple advanced algorithms on top of basic preprocessing (noise reduction, normalization):

[0125] Adaptive noise reduction algorithm: Unlike basic noise reduction (which only filters fixed noise), it can learn dynamic noise features in real time (such as voices that fluctuate in volume in a meeting room or flight announcements at an airport). Its implementation method is as follows: It adopts a real-time noise reduction model based on deep learning, through two parallel branches, such as a noise estimation branch, which is used to analyze the noise features without speech segments in real time and dynamically update the noise model; and a speech separation branch, which is used to separate the mixed signal of speech + dynamic noise into clean speech and noise, filtering noise while preserving speech details (such as the ending sound of soft speech).

[0126] Echo cancellation enhancement algorithm: In the listening and translation scenario, the speaker of the XR smart glasses may play the translation result (such as English), while the microphone will also pick up the sound from this speaker, forming an echo (the translation result is re-recorded, causing the ASR model to recognize the mixed content of the translation result and the original speech). The implementation method is as follows: multi-channel echo cancellation is adopted, using two microphones, such as the main microphone to pick up the original speech + echo; the reference microphone only picks up the translation result played by the speaker (echo reference signal); the algorithm uses the reference signal to subtract the echo from the main microphone signal, completely eliminating echo interference.

[0127] Speech enhancement algorithm: When the original speech signal is weak (e.g., the other party is more than 5 meters away from the XR smart glasses) and the signal-to-noise ratio (SNR) is less than 10dB, ASR is difficult to recognize. The implementation method is to use a combination algorithm of spectral subtraction and Wiener filtering to amplify the speech signal while suppressing residual noise and improving the signal-to-noise ratio (to achieve the optimal signal-to-noise ratio range for ASR recognition).

[0128] The algorithm described above can improve the recognition accuracy of the ASR model in noisy environments, and the accuracy of the translation results can be improved simultaneously. At the same time, it eliminates the need for users to move close to the sound source or manually adjust the volume, thus adapting to the non-ideal environmental requirements of listening and translation scenarios.

[0129] (3) In speech scenarios, speakers often use visual aids (such as PPT slides, whiteboard drawings, and physical displays). Relying solely on speech translation can lead to the loss of key information, resulting in incomplete understanding by the audience. Therefore, the translation example needs to load a visual-speech fusion processing module to convert visual aids into supplementary text content and integrate it into the translation result. This is achieved by: constructing or updating the topic context model of the speech based on the visual aids; and using the topic context model to disambiguate polysemous words in the source language text during machine translation. Alternatively, the identified visual aids (such as text and chart types) can be fused with the intermediate results generated by speech recognition (such as phoneme sequences and candidate texts) in a multimodal manner. The fused multimodal features can then be used for joint reasoning to generate the final optimized recognized text and translation result.

[0130] (4) In group chat scenarios, where ≥3 people interact simultaneously (e.g., XR smart glasses user + 2 App users + 1 H5 user), the core problem is the chaotic multi-channel audio streams, where multiple people may speak at the same time, resulting in a jumbled audio stream. The ASR model cannot distinguish users, and the translation results cannot be associated with the speaker. Based on this, the translation instance needs to load a multi-channel audio processing module to solve the audio chaos problem. The implementation method is as follows: using the microphone array of the client device to collect multi-channel audio signals, the azimuth angle of each speaker's sound source is determined through the direction of arrival estimation algorithm, and a unique speaker identifier is assigned to the audio streams from different directions, thereby achieving audio stream separation. At the same time, voiceprint features are extracted from the mixed audio streams. Based on the extracted voiceprint features, the audio segments are clustered, and audio segments belonging to the same speaker are grouped into one category. A speaker identifier is assigned to each cluster category and continuously tracked. Optionally, the system can combine sound source localization information with voiceprint features to separate multiple audio streams, and assign speaker identifiers and precise timestamps to each separated audio segment; based on the order of the timestamps, the spoken texts of all speakers are sorted chronologically to generate the final dialogue record.

[0131] This invention, by configuring dedicated behavioral logic for different application scenarios, upgrades translation instances from passive audio processors to proactive scene awareness and interaction engines, achieving deep adaptation of translation strategies to real-world communication scenarios. It simulates human interaction rhythms in dialogues, enhances anti-interference capabilities in listening comprehension, introduces visual context for accurate understanding in speeches, and clarifies the flow of conversations among multiple people in group chats. This systematically solves the core contradiction that a single processing mode cannot handle diverse communication styles, ultimately leading to effective improvements in the naturalness, accuracy, robustness, and orderliness of translation.

[0132] Based on the above embodiments, the step of creating a real-time communication (RTC) room according to the application scenario includes:

[0133] A room identifier is generated based on the application scenario and the translation language selected by each client.

[0134] The RTC room is created based on the room identifier.

[0135] It should be understood that a room identifier is a unique string (such as a Universally Unique Identifier (UUID) or an encoding generated according to specific rules) used to uniquely identify a specific translation session on the server. For example, a room identifier could be a room number, a set of numeric codes automatically generated by the system (such as 358764).

[0136] The server can generate room identifiers according to preset rules. For example, assuming the application scenario is "group_chat" (group chat), the translation languages ​​selected by each client are: user A selects "en->zh" (English to Chinese), and user B selects "zh->en" (Chinese to English); the room identifier = scenario + "_" + language pair hash value + "_" + timestamp, such as group_chat_abc123ef_1737984000.

[0137] Optionally, the server will first check if an active room with the same application scenario and translation language combination already exists; if it exists, it can directly return the room's identifier (suitable for scenarios where multiple people join the same meeting); if it does not exist, a new one will be generated.

[0138] The server calls the createRoom(roomId) method of the RTC SDK, passing the room identifier generated in the previous step as a parameter. After receiving the instruction, the RTC server will register the room identifier internally to ensure its uniqueness, and initialize all management modules related to the room (member management, flow management, access control, etc.). Finally, it will return a signal indicating successful creation to the server.

[0139] This invention establishes a precise and unique mapping relationship between translation tasks and communication resources by directly encoding business logic (application scenarios and translation languages) into the generation rules of room identifiers. This achieves isolation of translation sessions with different scenarios and language requirements at the RTC infrastructure level. This not only ensures the orderliness of session management and the stability of data streams without interference, but also enables the system to quickly locate and allocate dedicated computing resources (such as scenario-based translation instances) based on identifiers, laying the foundation for achieving high-quality, low-latency cross-platform real-time translation.

[0140] Based on the above embodiments, generating a room identifier according to the application scenario and the translation language selected by each client includes:

[0141] Map the application scenario to a scenario code;

[0142] The translation languages ​​selected by each client are sorted and concatenated to form a language sequence encoding;

[0143] The scene code and the language sequence code are combined according to a predetermined format to generate the room identifier.

[0144] It should be understood that scene encoding is a predefined or dynamically generated, unique, machine-readable string identifier whose function is to accurately characterize the specific business domain served by the real-time communication session and its specific requirements for communication quality, functionality, or resources. Language sequence encoding is an ordered string formed by sorting and concatenating the translation target languages ​​selected by all clients in the room according to predefined standardized rules. This encoding clarifies all language paths that need to be supported in the real-time translation session.

[0145] Obtain the natural language description text of the application scenario, use a text classification model to perform semantic understanding on the description text, identify the predefined scenario category to which it belongs, and map the identified scenario category to its corresponding scenario code. Alternatively, a pre-defined enumerated mapping table of application scenarios and scenario codes can be provided, and the application scenario can be used as the query key to search in the enumerated mapping table and output the corresponding standardized scenario code.

[0146] The translation languages ​​selected by each client are sorted and concatenated to form a language sequence code. For example, the selected translation languages ​​are uniformly converted into two-letter or three-letter language codes conforming to the ISO 639 standard. All converted language codes are then arranged in ascending or descending alphabetical order to form an ordered list. Predefined connectors are used to link the language codes in the ordered list sequentially to generate the language sequence code. Alternatively, different priority weights are assigned to different languages ​​based on the application scenario or historical data. All language codes are then sorted according to their priority weights, with languages ​​with higher weights placed at the beginning of the sequence. The sorted language codes are then concatenated to generate the language sequence code. Another approach is to identify the language family or language group to which each language belongs. First, the languages ​​are clustered according to their family. Within the same family, the language codes are then sorted alphabetically. The final sorted language codes are then concatenated to generate the language sequence code.

[0147] Use a separator to connect the scene code and the language sequence code. The separator is used to clearly distinguish between the two logical parts: scene and language. Alternatively, convert the scene code into a fixed-length string, padding it with predetermined characters if the length is insufficient. Then, directly concatenate the fixed-length scene code with the language sequence code to generate the room identifier.

[0148] Optionally, the description of the application scenario and the translation language configuration information of all clients can be concatenated into a raw string in a specific order. A cryptographic hash function is then applied to this raw string to generate a unique hash value, which is used as the core part of the room identifier. Alternatively, depending on the application scenario, an identifier for a dedicated translation instance cluster providing services can be determined. Based on the translation language selected by each client, the required multilingual translation path can be determined. The identifier of the dedicated translation instance cluster and the multilingual translation path are encoded to generate a room identifier, enabling communication data packets to be routed to the specified translation instance cluster based on this identifier.

[0149] This invention constructs a session identification system that combines semantics and global uniqueness by mapping application scenarios and translated languages ​​to machine-readable scenario codes and language sequence codes, respectively, and generating room identifiers according to predetermined rules. This ensures the accuracy of session management, the efficiency of resource allocation, and the quality of service for cross-language communication in complex and ever-changing real-time interactive environments.

[0150] Based on the above embodiments, sending the translation results to multiple clients accessing the RTC room via the communication channel of the RTC room includes:

[0151] Based on the type of each client, the translation result is converted into a data format that matches the type of the client;

[0152] The translation results are sent to multiple clients connected to the RTC room via the communication channel of the RTC room.

[0153] It should be understood that different terminal devices have fundamental differences in hardware capabilities, software environment, and main uses. Client types can include: XR smart glasses: relatively limited computing resources, precious screen space (limited field of view), special interaction methods (voice, gesture), and sensitive to power consumption; mobile apps: strong computing resources, larger screens, touch interaction methods, and can display more complex information; H5 web clients: run in a browser environment, lightweight, no installation required, but performance and processing power are limited by the browser.

[0154] The core of format conversion is to customize data encapsulation for different terminals. Its goal is to maximize the advantages of each terminal and mitigate its disadvantages while ensuring consistency of core information (translation). For example, XR smart glasses require spatial positioning and display control parameters to overlay the translation results onto the correct field of view. The core format is JSON with rendering instructions. Optionally, it can be converted to a binary format, for example, containing only the most essential fields and removing metadata that the glasses may not need, such as speaker ID and confidence level. It should be understood that the binary format is smaller, has faster parsing speed, saves the glasses' CPU and power, and adapts to its limited hardware capabilities. Apps require structured data and local interaction parameters to adapt to native controls (such as lists and pop-ups) and support caching. The core format is detailed JSON with control identifiers. It should be understood that apps have larger screens and can display complete history, speaker information, confidence level (used to display translation quality), etc., providing a more comprehensive user experience. H5 requires HTML tags, CSS styles, and JS interactive instructions to adapt to different screen sizes and support web page operations. The core format is standard, lightweight JSON with DOM structure (or directly returns HTML fragments) to ensure that it can be easily parsed and displayed in various browsers.

[0155] The server sends different data packets to their respective clients via the data channel in the RTC room. Upon receiving the data packets, each client parses and displays them. For example, XR smart glasses directly extract the translation and render it at a designated location in the field of view, with fonts and colors optimized for the XR experience; mobile apps parse the JSON, inserting the current conversation as a chat bubble and displaying the speaker and time; and H5 web pages parse the data and display it at the bottom of the page as a scrolling text bar.

[0156] This invention achieves precise optimization of cross-platform data expression by dynamically generating data formats that match the hardware capabilities and interactive characteristics of different types of clients. On the one hand, it provides simplified and efficient data encapsulation for resource-constrained devices such as XR smart glasses, significantly reducing their computational load and power consumption, and ensuring the smoothness of translation display and battery life. On the other hand, it provides complete and structurally rich data packages for interactive terminals such as mobile apps and H5 pages, supporting the implementation of their complex functions. While ensuring the consistency of core information, it maximizes the performance advantages of various terminals, fundamentally solving the problem that a single data format cannot accommodate multi-platform experiences, and achieving the optimization of the overall user experience under cross-platform collaboration.

[0157] Based on the above embodiments, the cross-end translation method further includes:

[0158] During the session, the application scenario switching instruction sent by the client is received;

[0159] In response to the switching command, the translation instance is reconfigured with a translation model that matches the application scenario after the switch.

[0160] It should be understood that if switching application scenarios within a session is not supported, users would need to exit the current session, reselect a new scenario, and rebuild the RTC room. This is not only cumbersome but also results in the loss of the current session's historical records (such as previous dialogue translations), leading to a poor user experience. The in-session translation switching model is designed to address this dynamic need, ensuring uninterrupted service and a continuous experience.

[0161] The essence of the application scenario switching process is to dynamically update the core capabilities of the translation instance without interrupting the current session. Specifically, during a session, the user triggers a scenario switch through operations (such as switching to group chat mode via voice command on XR smart glasses, or clicking the speech mode button in an app). The client then sends a scenario switch command to the backend. Upon receiving the switch command, the backend loads the translation model for the target application scenario (i.e., the switched application scenario).

[0162] For example, in an online international technical seminar, the initial part is a Q&A session (group chat scenario), followed by a thematic sharing session (speech scenario). The session is established, and the translation instances are configured for the group chat scenario, using an ASR model supporting source separation and a conversational NMT model. The host clicks the "Enter Speech Mode" button on the app, and the app sends a command to the server. Upon receiving the command, the server immediately sends a reconfiguration signal to the translation instances serving the session. The translation instances begin a seamless switch, for example, completing the translation of the currently processed sentence and then pausing to receive new audio streams; or quickly loading a model optimized for the speech scenario from the model repository, such as a combination of models focusing on long sentence recognition, echo cancellation, and formal writing style. Simultaneously, their internal processing logic is switched from multi-channel audio stream separation to single-speaker optimization. The entire process is completed in a very short time (e.g., seconds). The translation instances complete the reconfiguration, notify the server that they are ready, and the system continues translating, but the translation style and accuracy focus are now adapted to the speech scenario.

[0163] This invention, by introducing dynamic scene switching capabilities, enables real-time translation services to seamlessly adapt to complex agendas in real-world communications, addressing the core pain point that a single static configuration cannot meet the needs of coherent and ever-changing communication. Users can smoothly switch scene modes without interrupting the session, allowing the system to intelligently follow the discussion's pace and automatically optimize translation strategies. This improves the fluency and naturalness of cross-language collaboration, avoiding interruptions and delays caused by restarting the session.

[0164] Based on the above embodiments, the cross-end translation method further includes:

[0165] Receive a session control command from any of the clients, the session control command including at least one of a start translation command, a stop translation command, and a switch translation language command;

[0166] According to the session control instructions, the parameters of the real-time translation processing are adjusted, and the adjusted parameter results are synchronously sent to all other clients in the same session.

[0167] It should be understood that session control commands are operations initiated by a user through any client during the current translation session that directly affect the translation's running status or core configuration. Session control commands may include at least one of the following: start translation command, stop translation command, and switch translation language command. The start translation command is used to initiate the audio acquisition, transmission, and translation process of the session; the stop translation command is used to pause audio acquisition, transmission, and translation while retaining the session; and the switch translation language command is used to change the configuration of the source or target language.

[0168] Parameters refer to dynamically adjustable configuration variables that control the overall behavior, state, and output of a real-time translation session. These parameters may include runtime status parameters (the master switch that controls whether translation is enabled or disabled), translation target parameters (used to explicitly specify the source and target languages ​​for translation), scene mode parameters (used to specify the current processing scene, thereby triggering the loading of different models and logic), audio processing parameters (used to dynamically adjust the intensity of noise reduction and echo cancellation to adapt to different environments), and output control parameters (controlling whether the returned translation includes metadata such as timestamps and speaker IDs).

[0169] Based on the session control instructions, the parameters for real-time translation processing are adjusted. For example, based on the type of session control instruction, the target parameters to be adjusted and their target values ​​are determined, and the values ​​of the target parameters are updated to the target values. Specifically, if the session control instruction is to switch translation languages, the client requesting the instruction is verified to have the necessary permissions; after successful verification, the translation target parameters are updated to the language pair specified in the instruction. Alternatively, if the session control instruction is to switch translation languages, the translation target parameters are first adjusted, and then, based on the adjusted translation target parameters, an instruction is sent to the translation instance to load the corresponding target language translation model. Optionally, before adjusting the parameters, the current values ​​of the parameters are recorded; after adjusting the parameters, if a reversal instruction is received within a preset time, the parameters are restored to the recorded current values.

[0170] The server synchronously sends the adjusted parameter results to all other clients in the same session. For example, after successfully adjusting the parameters, a status synchronization message containing the parameter results is immediately generated and sent via broadcast or multicast through the data channel of the RTC room to all clients except the one that initiated the command. Specifically, a globally incrementing sequence number is appended to each status synchronization message. Each client determines the order in which it receives messages based on the sequence number and only processes the message with the latest sequence number. Optionally, if the parameter adjustment only involves some parameters, only the changed parameters and their new values ​​are sent; if a client joins the session for the first time or detects a state inconsistency with the server, a full synchronization is triggered, sending the current values ​​of all parameters. Optionally, the server maintains a global state version number, which increments after each parameter adjustment; this version number is carried in the synchronization message. When a client establishes or reconnects, if it finds that its own state version number is lower than the server's, it actively requests the latest state from the server.

[0171] This invention establishes a centralized, state-synchronized instruction processing mechanism, achieving unified and efficient cross-terminal collaborative control. Instructions initiated by any client (such as start, stop, or language switch) trigger the server to globally adjust core operating parameters and immediately synchronize the adjusted state to all participating clients. This grants the system flexible control allocation, ensures consistency of states across multiple terminals, and effectively avoids collaborative obstacles such as translation interruptions and display confusion caused by asynchronous instruction responses or inconsistent states. Therefore, it provides a core guarantee for stable, reliable, and seamless translation collaboration in complex multi-terminal scenarios.

[0172] To further explain the cross-end translation method proposed in this invention, please refer to the following embodiments.

[0173] This invention, based on RTC technology, enables XR smart glasses, apps, and H5 pages to interact using a unified data format. Audio data is collected at a fixed sampling rate on the XR smart glasses, and the data stream is transmitted to a cloud server. The cloud server processes the data using a real-time translation module, including noise cancellation, feature extraction, real-time transcription, and real-time machine translation. Finally, the cloud server converts the translation results according to the requirements of different devices and broadcasts the messages to the corresponding clients. Each client displays the translation result based on the current user's country. Furthermore, specialized processing logic is designed for different scenarios such as dialogue, listening comprehension, speeches, and group chats. Specifically, in dialogue scenarios, real-time speech recognition and fast translation algorithms are used, combined with voice wake-up and interruption mechanisms (the system waits for the wake word; once detected, it enters recording mode and continues translating; if an interruption signal is detected, the current recording and translation process stops), ensuring the timeliness and naturalness of the translation. In listening comprehension scenarios, speech enhancement technology is added to improve speech recognition accuracy and optimize translation accuracy. In speech scenarios, computer vision technology is used to recognize the speaker's body language and contextual information (acquiring speaker body language and contextual information through cameras and sensors, as well as background information such as pre-recorded speech drafts by users; this information is analyzed to adjust the translation model and post-processing, assisting in achieving more accurate and appropriate translations), and the translation is displayed intuitively on XR smart glasses. In group chat scenarios, sound source localization and multi-threaded processing technology enable simultaneous recognition and translation of multiple speakers, and the translations are sorted and displayed according to the order of speaking or importance.

[0174] In an international business group chat, people from multiple countries speak simultaneously. XR smart glasses utilize sound source localization technology to identify the location and voice of different speakers, and RTC channel processing technology to simultaneously recognize and translate multiple speech streams. A multi-model selection and adaptation module selects the appropriate translation model based on the speaker's language (e.g., Chinese, English, Korean, etc.). The translation results are displayed on the XR smart glasses in the order of speaking, and different user icons distinguish different speakers, allowing users to quickly browse and understand each person's content, achieving efficient cross-language group chat communication.

[0175] refer to Figure 3 The translation process in an international group chat scenario is as follows:

[0176] 1. Users can choose from various client types, such as H5, App, XR smart glasses, etc., and log in.

[0177] 2. The backend side designs a unique room number based on the user's usage scenario and translation language, and creates an RTC room.

[0178] 3. The RTC room informs the translation service of the specific startup parameters and starts the translation instance.

[0179] 4. Deploy the AI ​​model service on the algorithm side and receive audio stream information: collect audio streams with a specified sampling rate from different ends and send them to the RTC room, while informing the translation instance to perform speech translation.

[0180] 4.1 Voice Acquisition: The original voice signal (sound wave) is captured by devices such as microphones, converted into an electrical signal, and further quantized into digital audio data (such as PCM format) to ensure that the signal is transmitted to the system intact.

[0181] 4.2. Preprocessing Optimization: Noise Reduction: Remove environmental noise (such as background conversations and device noise) using filters (e.g., adaptive filters) or deep learning algorithms (e.g., spectral subtraction) to highlight the target speech signal. Endpoint Detection: Identify the start and end points of the speech, eliminate invalid silence segments, and reduce the amount of data for subsequent processing. Normalization: Standardize audio parameters such as volume and sampling rate (16000Hz) to avoid processing deviations caused by device differences.

[0182] 4.3. ASR Speech-to-Text: Employing a real-time transcription mode, the system outputs corresponding text within hundreds of milliseconds simultaneously with speech input. Furthermore, it utilizes feature extraction, acoustic model processing, and an LLM model for optimization and ambiguity correction.

[0183] 5. Text translation: The source language text of ASR is converted into the target language text, and the translation is optimized after text preprocessing, translation model inference (NMT model).

[0184] 6. Return the translation result to the client based on the room number. The client will then display the translation result in the appropriate language based on the user's current language source.

[0185] This invention, based on RTC channel technology, enables seamless cross-platform operation, effectively improving the stability and real-time performance of audio and text data transmission during cross-platform translation, thereby enhancing the user's interactive experience. Simultaneously, it utilizes different translation models for different application scenarios, ensuring the adaptability of the translation model to the application scenario and improving translation accuracy.

[0186] The cross-end translation apparatus provided by the present invention is described below. The cross-end translation apparatus described below can be referred to in correspondence with the cross-end translation method described above.

[0187] refer to Figure 4 The cross-terminal translation device provided by the present invention includes an application scenario determination module 401, an RTC room creation module 402, an audio stream receiving module 403, and a translation module 404.

[0188] The application scenario determination module 401 is used to determine the application scenario of the translation task based on translation requests initiated by multiple clients.

[0189] The RTC room creation module 402 is used to create a real-time communication (RTC) room according to the application scenario and to assign translation instances to the RTC room.

[0190] The audio stream receiving module 403 is used to receive an audio stream from at least one of the clients through the communication channel of the RTC room;

[0191] The translation module 404 is used to translate the audio stream according to the translation model configured in the translation instance, and send the translation result to multiple clients accessing the RTC room through the communication channel of the RTC room.

[0192] The cross-platform translation device provided in this invention determines the application scenario of the translation task based on translation requests initiated by multiple clients; creates a Real-Time Communication (RTC) room according to the application scenario and assigns translation instances to the RTC room; receives audio streams from at least one client through the communication channel of the RTC room; translates the audio streams according to the translation model configured in the translation instance, and sends the translation results to multiple clients connected to the RTC room through the communication channel of the RTC room. This invention, based on RTC channel technology, achieves seamless cross-platform operation, effectively improving the stability and real-time performance of audio and text data transmission during cross-platform translation, thereby enhancing the user's interactive experience; simultaneously, it uses different translation models for different application scenarios, ensuring the adaptability of the translation model to the application scenario and improving translation accuracy.

[0193] In one embodiment, the translation module 404 is further configured to:

[0194] Based on the application scenario, configure a translation model that matches the application scenario for the translation instance.

[0195] In one embodiment, the translation module 404 is further configured to:

[0196] Parse the metadata of the application scenario, which includes at least one of the following: business domain, text genre, terminology database identifier, and target user group;

[0197] Based on the metadata, one or more specialized translation models are selected from a pre-built model repository;

[0198] The selected translation models are loaded into the runtime environment of the translation instance.

[0199] In one embodiment, the translation module 404 is further configured to:

[0200] The audio stream is preprocessed, including noise reduction, endpoint detection, and normalization, to obtain clean speech data.

[0201] The clean speech data is streamed using an Automatic Speech Recognition (ASR) model, and the recognized text is optimized using a large language model to obtain the source language text.

[0202] The source language text is input into the translation model for translation, and the translation result is obtained.

[0203] In one embodiment, the application scenario includes at least one of a dialogue scenario, a listening and translation scenario, a speech scenario, and a group chat scenario; the translation instance is also configured with dedicated processing logic matching the application scenario, wherein:

[0204] The dedicated processing logic configured for the dialogue scenario includes activating a recording control mechanism based on the voice wake-up word and an interruption response mechanism.

[0205] The dedicated processing logic configured for the listening and translation scenario includes enabling enhanced speech signal processing algorithms;

[0206] The dedicated processing logic configured for the speech scenario includes enabling integrated processing of visual aid information to optimize translation output;

[0207] The dedicated processing logic configured for the group chat scenario includes the separation and sorting of multiple audio streams based on sound source localization.

[0208] In one embodiment, the RTC room creation module 402 is also used for:

[0209] A room identifier is generated based on the application scenario and the translation language selected by each client.

[0210] The RTC room is created based on the room identifier.

[0211] In one embodiment, the RTC room creation module 402 is also used for:

[0212] Map the application scenario to a scenario code;

[0213] The translation languages ​​selected by each client are sorted and concatenated to form a language sequence encoding;

[0214] The scene code and the language sequence code are combined according to a predetermined format to generate the room identifier.

[0215] In one embodiment, the translation module 404 is further configured to:

[0216] Based on the type of each client, the translation result is converted into a data format that matches the type of the client;

[0217] The translation results are sent to multiple clients connected to the RTC room via the communication channel of the RTC room.

[0218] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can invoke logical instructions in the memory 530 to execute a cross-platform translation method. This method includes: determining the application scenario of the translation task based on translation requests initiated by multiple clients; creating a real-time communication (RTC) room based on the application scenario and allocating translation instances to the RTC room; receiving audio streams from at least one client through the communication channel of the RTC room; translating the audio streams according to the translation model configured in the translation instance; and sending the translation results to multiple clients accessing the RTC room through the communication channel of the RTC room.

[0219] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0220] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the cross-platform translation method provided by the above methods. The method includes: determining the application scenario of the translation task based on translation requests initiated by multiple clients; creating a real-time communication (RTC) room based on the application scenario and allocating translation instances to the RTC room; receiving audio streams from at least one of the clients through the communication channel of the RTC room; translating the audio streams according to the translation model configured in the translation instance; and sending the translation results to the multiple clients accessing the RTC room through the communication channel of the RTC room.

[0221] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the cross-platform translation method provided by the above methods. The method includes: determining the application scenario of the translation task based on translation requests initiated by multiple clients; creating a real-time communication (RTC) room based on the application scenario and allocating translation instances to the RTC room; receiving an audio stream from at least one of the clients through the communication channel of the RTC room; translating the audio stream according to a translation model configured in the translation instance; and sending the translation result to the multiple clients accessing the RTC room through the communication channel of the RTC room.

[0222] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0223] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0224] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-end translation method, characterized in that, include: Based on translation requests initiated by multiple clients, determine the application scenarios for the translation tasks; Based on the application scenario, a real-time communication (RTC) room is created, and translation instances are assigned to the RTC room. The RTC room is a multi-terminal collaborative logical unit created by the backend server based on RTC technology, configured with communication parameters matching the application scenario. The translation instance is a translation processing resource unit that can be independently scheduled and started / stopped on demand. It includes the AI ​​model and data processing module required to complete the translation of a specific scenario. The translation instance acts as a virtual client to access the RTC room, subscribe to the audio streams of all clients in the RTC room, and publish translation results. The RTC room is used to provide a data transmission channel for audio streams and translation results for multiple clients that have accessed the room. The translation instance is used as a virtual client to access the RTC room, and at least one audio stream from the client is received through the communication channel of the RTC room; The audio stream is translated according to the translation model configured in the translation example, and the translation result is sent to multiple clients accessing the RTC room through the communication channel of the RTC room. The translation model is configured as follows: Parse the metadata of the application scenario, which includes at least one of the following: business domain, text genre, terminology database identifier, and target user group; Based on the metadata, one or more specialized translation models are selected from a pre-built model repository; The selected translation models are loaded into the runtime environment of the translation instance.

2. The cross-end translation method according to claim 1, characterized in that, The step of translating the audio stream according to the translation model configured in the translation instance includes: The audio stream is preprocessed, including noise reduction, endpoint detection, and normalization, to obtain clean speech data. The clean speech data is streamed using an Automatic Speech Recognition (ASR) model, and the recognized text is optimized using a large language model to obtain the source language text. The source language text is input into the translation model for translation, and the translation result is obtained.

3. The cross-end translation method according to claim 1, characterized in that, The application scenarios include at least one of dialogue scenarios, listening and translation scenarios, speech scenarios, and group chat scenarios; the translation instance is also configured with dedicated processing logic matching the application scenarios, wherein: The dedicated processing logic configured for the dialogue scenario includes activating a recording control mechanism based on the voice wake-up word and an interruption response mechanism. The dedicated processing logic configured for the listening and translation scenario includes enabling enhanced speech signal processing algorithms; The dedicated processing logic configured for the speech scenario includes enabling integrated processing of visual aid information to optimize translation output; The dedicated processing logic configured for the group chat scenario includes the separation and sorting of multiple audio streams based on sound source localization.

4. The cross-end translation method according to claim 1, characterized in that, The step of creating a real-time communication (RTC) room according to the application scenario includes: A room identifier is generated based on the application scenario and the translation language selected by each client. The RTC room is created based on the room identifier.

5. The cross-end translation method according to claim 4, characterized in that, The step of generating a room identifier based on the application scenario and the translation language selected by each client includes: Map the application scenario to a scenario code; The translation languages ​​selected by each client are sorted and concatenated to form a language sequence encoding; The scene code and the language sequence code are combined according to a predetermined format to generate the room identifier.

6. The cross-end translation method according to claim 1, characterized in that, The step of sending the translation results to multiple clients connected to the RTC room via the communication channel of the RTC room includes: Based on the type of each client, the translation result is converted into a data format that matches the type of the client; The translation results are sent to multiple clients connected to the RTC room via the communication channel of the RTC room.

7. A cross-end translation device, characterized in that, include: The application scenario determination module is used to determine the application scenario of the translation task based on translation requests initiated by multiple clients. The RTC room creation module is used to create real-time communication RTC rooms according to the application scenario and allocate translation instances to the RTC rooms. The RTC room is a multi-terminal collaborative logical unit created by the backend server based on RTC technology, configured with communication parameters matching the application scenario. The translation instance is a translation processing resource unit that can be independently scheduled and started and stopped on demand. It includes the AI ​​model and data processing module required to complete the translation of a specific scenario. The translation instance acts as a virtual client to access the RTC room to subscribe to the audio streams of all clients in the RTC room and publish translation results. The RTC room is used to provide audio stream and translation result data transmission channels for multiple clients that have accessed the room. An audio stream receiving module is used to connect the translation instance as a virtual client to the RTC room and receive audio streams from at least one of the clients through the communication channel of the RTC room. The translation module is used to translate the audio stream according to the translation model configured in the translation instance, and send the translation result to multiple clients accessing the RTC room through the communication channel of the RTC room; The translation model is configured based on parsing the metadata of the application scenario, which includes at least one of the following: business domain, text genre, terminology database identifier, and target user group; Based on the metadata, select one or more specialized translation models from the pre-built model repository; load the selected translation models into the runtime environment of the translation instance.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the cross-end translation method as described in any one of claims 1 to 6.

Images