An AI cloud-based cross-language call and external network video and audio translation method and system

Abstract

The application discloses a kind of based on AI cloud cross-language conversation and external network video and audio translation method and system, comprising the following steps: in the exclusive foreign language access communication identification of intelligent communication terminal preset;When the external input source request of exclusive foreign language access communication identification is monitored to intelligent communication terminal, the transmission path of original audio signal is redirected to AI intelligent body cloud service module;AI intelligent body cloud service module receives original audio signal, determines source language type, and then executes text translation and mother language synthesis, generates target voice stream;Original audio signal is executed in cloud to delete instruction, and target voice stream is encapsulated into downstream data packet;Intelligent communication terminal receives downstream data packet, and outputs pure mother language voice through audio synthesizer.The AI intelligent body service architecture deployed in cloud and the system kernel of intelligent communication terminal are deeply coupled in the application, interception, redirection, translation and original sound are removed to audio stream.

Description

Technical Field

[0001] This invention relates to the field of communication and multimedia information processing technology, and in particular to an AI-based cloud-based method and system for cross-language calls and external network audio-visual translation. Background Technology

[0002] With the acceleration of globalization, cross-language communication and consumption of overseas multimedia content have become important needs for smart terminal users. In the current fields of communication technology and artificial intelligence speech processing, achieving efficient and convenient language translation is of great significance for improving cross-border communication experiences and enriching cultural life.

[0003] Existing cross-language translation technologies still have significant limitations in practical applications. Current solutions mostly rely on dedicated translation headsets, external hardware devices, or standalone third-party translation applications. These technologies typically require cumbersome manual operations from users, such as manually opening the application, manually selecting the target language, and frequently switching between different modes. This results in a disjointed translation process and prevents seamless, automated operation at the system level. Especially when users are abroad or accessing overseas online resources, although they can access a wealth of audio-visual content such as overseas online TV, movies, streaming videos, and social entertainment programs, language barriers and the inability of current technology to provide real-time native language translation and synchronized playback support prevent users from properly understanding and using this content.

[0004] Furthermore, existing translation products generally lack effective audio management mechanisms, failing to physically shield or discard the original foreign language audio and tracks at the processing end. This often results in overlap and mixing of the original foreign language audio and the translated audio, making it difficult to achieve forced isolation of single-language output. Existing communication terminals also lack a core native translation system that is uniformly scheduled by an AI agent and allows for multi-terminal collaborative operation. This makes cross-border communication and overseas audio-visual entertainment processes extremely cumbersome, severely lacking in practicality and failing to meet users' urgent need for a high-quality, system-level native translation experience. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for cross-language calls and external network audio-visual translation based on AI cloud. By deeply coupling the AI ​​intelligent agent service architecture deployed in the cloud with the system kernel of the intelligent communication terminal, the interception, redirection, translation and original sound removal of the audio stream can be achieved.

[0006] To achieve the objectives of this invention, this invention provides a method for cross-language communication and external network audio-visual translation based on AI cloud computing, comprising the following steps:

[0007] S1. A dedicated foreign language access communication identifier, independent of the local physical number or main account identifier, is preset and activated in the underlying communication module of the intelligent communication terminal. This dedicated foreign language access communication identifier includes a virtual communication sub-number, a specific Internet Protocol address, a preset communication port number, or an encrypted Uniform Resource Locator (URL). This dedicated foreign language access communication identifier serves as the logical trigger source for translation services; when the system detects an incoming call signal or data stream request based on this identifier, it automatically activates the global translation mode.

[0008] S2. When the intelligent communication terminal receives a voice call request or an external network audio / video data stream accessed through the dedicated foreign language access communication identifier, the hardware abstraction layer of the terminal operating system and the kernel audio driver module work together to redirect the transmission path of the original audio signal from the default local audio decoding path to the uplink of the AI ​​intelligent agent cloud service. Furthermore, the intelligent communication terminal, through the system's underlying routing table configuration, encapsulates the collected pulse code modulation data stream into real-time transmission protocol data packets and transmits them transparently to the AI ​​intelligent agent cloud service. This process does not require enabling any third-party translation plugins at the application layer.

[0009] S3. After receiving the audio data stream from the terminal, the cloud-based AI agent cloud service module calls the integrated speech recognition engine to extract features and convert the original foreign language speech into text, generating a corresponding first language text stream. Specifically, the AI ​​agent cloud service module uses a preset national language encoding matching algorithm to automatically identify the source language type based on the metadata or spectral characteristics of the audio stream and matches the corresponding translation model. Further, the AI ​​agent cloud service module inputs the first language text stream into the machine translation engine, outputting a second language text stream that matches the user's preset language, and the speech synthesis engine generates a target speech stream with native language features based on the second language text stream.

[0010] S4. While the AI ​​intelligent agent cloud service module generates the target speech stream, the original audio masking component in the cloud service module performs data cleaning operations. The original audio masking component ensures that the original foreign language speech data is not sent to the terminal by marking the payload of the original audio data packet and executing deletion instructions. Furthermore, the cloud service module only encapsulates the synthesized target speech stream into downlink data packets, thereby achieving complete physical isolation of the original audio at the data source.

[0011] S5. The downlink receiving module of the intelligent communication terminal acquires the clean target speech stream from the cloud and directly delivers it to the audio synthesizer at the system's bottom layer. The audio synthesizer converts the received target speech stream into an analog signal, which is then played through the terminal's speaker or receiver. Since the original foreign language speech has been discarded in the cloud, the terminal hardware layer only receives and processes a single native language speech signal, thereby achieving an interference-free cross-language communication experience.

[0012] This invention also provides an AI-based cloud-based cross-language calling and external network audio-visual translation system, including an intelligent communication terminal, a dedicated foreign language access identifier module, an AI intelligent agent cloud service module, a national language encoding matching module, an original sound masking and discarding module, and a native language output module.

[0013] As a preferred embodiment of the present invention, the intelligent communication terminal serves as the physical platform for the entire system, and integrates a system kernel-level audio routing controller. When a specific communication task is detected, the audio routing controller takes over control of the audio input / output interfaces. The intelligent communication terminal establishes a full-duplex communication link with the AI ​​agent cloud service module via a high-speed network communication interface.

[0014] As a preferred embodiment of the present invention, the dedicated foreign language access identifier module is deployed in the system registry of the smart communication terminal to store and manage multiple differentiated foreign language access identifiers. The dedicated foreign language access identifier module establishes a logical association with the terminal's dialer and streaming media player. When a user dials or receives a number associated with a specific identifier, or when the streaming media player accesses an address marked as an external network audio / video resource, the module sends an interrupt command to the system kernel, forcibly switching the audio stream direction.

[0015] As a preferred embodiment of the present invention, the AI ​​intelligent agent cloud service module consists of a high-performance computing server cluster, including a central scheduling unit, a speech processing cluster, and a context-related database. The central scheduling unit is responsible for coordinating the allocation of computing resources among the ASR, MT, and TTS sub-modules. Furthermore, the speech processing cluster adopts a deep neural network architecture to perform millisecond-level parallel processing on the input audio stream, ensuring that the translation latency is lower than a preset perception threshold.

[0016] As a preferred embodiment of the present invention, the national language encoding matching module is electrically connected to the AI ​​intelligent agent cloud service module. The national language encoding matching module automatically determines the source language by analyzing the spectrogram features of the audio signal or parsing the country code field in the communication protocol. Furthermore, this module dynamically loads the corresponding language model library based on the recognition results, achieving automatic adaptation to multiple mainstream languages ​​such as English, French, Japanese, and German, without requiring manual user intervention.

[0017] As a preferred embodiment of the present invention, the original audio blocking and discarding module is integrated into the output end of the AI ​​intelligent agent cloud service module. The original audio blocking and discarding module intercepts the originally acquired speech frames using a logic gate circuit simulator or software-defined networking technology. After the TTS module generates a new native language speech frame, this module sets the address pointer of the original speech frame in the buffer to null and releases the corresponding memory space, thereby ensuring that the bitstream sent to the terminal only contains the translated audio sample data.

[0018] As a preferred embodiment of the present invention, the native language output module is deployed in the audio subsystem of the intelligent communication terminal. The native language output module includes an audio decoder, a digital-to-analog converter, and a power amplifier. This module receives mono or multi-channel native language speech streams from the cloud and performs real-time compensation based on the terminal's current sound field environment, ultimately restoring the speech to clear native language through the terminal's electroacoustic transducer.

[0019] As a preferred technical solution of the present invention, when processing external network audio-visual scenarios, the system captures audio track data from the streaming media protocol and redirects it to the AI ​​intelligent agent cloud service module. The original audio masking and discarding module, when processing audio-visual files, performs a separation operation between background sound and human voice, discarding only the original foreign language human voice while retaining the background music and environmental sound effects in the video. It then remixes the synthesized native language human voice with the retained background sound and sends it to the terminal for playback, achieving a synchronized audio-visual viewing experience in the native language.

[0020] As a preferred embodiment of the present invention, in the step of configuring the dedicated foreign language access communication identifier, the system assigns a specific logical number to the user at the mobile switching center or core network side. When this logical number is dialed, network-side signaling triggers the underlying listener of the intelligent communication terminal. After the listener captures the incoming call interruption in the kernel space, it does not immediately invoke the standard audio subsystem, but instead redirects the audio stream to a preset cloud AI intelligent agent access gateway by modifying the socket options. This process is completed at the driver layer, avoiding application layer permission restrictions and resource scheduling delays.

[0021] As a preferred embodiment of the present invention, in the real-time translation processing step of the cloud-based AI agent, the AI ​​agent manages each translation task by establishing a session state machine. The ASR module employs streaming speech recognition technology, outputting text fragments in real time while the user speaks, without waiting for the entire sentence to finish. The MT module utilizes an attention-based neural machine translation model, combining contextual information to perform semantic correction and word order reordering on the recognized text. The TTS module, based on the identified speaker characteristics (such as gender, age, and emotional tendency), selects a matching voice library for synthesis, making the output native-speaker speech more realistic.

[0022] As a preferred embodiment of the present invention, the cloud-based AI agent employs a dual-buffer mechanism. The first buffer temporarily stores the received raw audio frames for ASR recognition; the second buffer constructs the downlink data packets to be sent to the terminal. Once ASR completes feature extraction, the corresponding data frames in the first buffer are marked as "invalid" and immediately cleaned up by the garbage collection mechanism. When encapsulating downlink data packets, the system logic strictly prohibits reading any payload from the first buffer, allowing only the audio samples generated by TTS to enter the second buffer. This logical circuit-breaking design ensures that the original foreign language signal cannot physically cross the cloud boundary to be transmitted back to the user terminal.

[0023] Compared with existing technologies, this invention provides a method and system for AI-based cloud-based cross-language communication and external network audio-visual translation, which has the following beneficial effects:

[0024] This invention deeply embeds translation functionality into the operating system's communication link by pre-setting a dedicated foreign language access communication identifier at the system's underlying layer and establishing an automatic routing mechanism. When users answer cross-border calls or watch overseas streaming videos, they do not need to manually launch any applications; the system automatically triggers translation logic based on the identifier, greatly simplifying the operation process and improving the naturalness of human-computer interaction. A physical-level shielding mechanism in the cloud solves the problem of original audio interference. Existing translation software often plays both the original and translated audio locally, leading to a confusing listening experience. This invention directly discards the original foreign language audio data during the cloud-based AI agent processing stage, ensuring that the signal transmitted back to the terminal contains only pure native language speech. This forced isolation mechanism at the data source fundamentally eliminates overlapping sound interference, making the translated call quality and audiovisual experience reach the standard of the native language. This invention has extremely high scenario versatility and cost advantages, not only suitable for two-way voice calls but also compatible with various audiovisual scenarios such as overseas online TV, movies, and live broadcasts. Since the core computing tasks are handled by the cloud-based AI agent, the intelligent communication terminal only needs basic audio acquisition and playback functions, eliminating the need for expensive dedicated translation hardware or high-performance local computing chips. This significantly reduces users' hardware purchase costs and system maintenance complexity. Leveraging the powerful computing capabilities of the AI ​​agent's cloud service, combined with the automatic recognition capabilities of the national language encoding matching module, accurate identification and rapid translation of multiple languages ​​can be achieved. Through optimized scheduling at the system's underlying level, data transmission and processing link losses are minimized, ensuring real-time synchronization of native language output. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Please see Figure 1 This invention provides a method and system for cross-language calling and external network audio-visual translation based on AI cloud computing, comprising: a cloud service module for AI intelligent agents deeply collaboratively constructed with multiple intelligent communication terminals. The intelligent communication terminals include, but are not limited to, smartphones, smartwatches, in-vehicle systems, smart TVs, and tablets. In the overall system architecture, external input sources enter the system through a dedicated foreign language access communication identifier. The external input sources include voice requests initiated by external calling terminals, as well as audio-visual data streams such as external network TV, external network movies, singing programs, and streaming media sources. The intelligent communication terminal internally presets and activates the dedicated foreign language access communication identifier, which is independent of the terminal's own conventional communication identifier, achieving underlying isolation of business logic.

[0028] During the system initialization phase, step S1 is executed to receive call requests or external network audio / video playback requests. When the external input source initiates access to the smart communication terminal via network protocols or telecommunications signaling, the terminal-side system's underlying routing module monitors for incoming call interruptions or data requests in real time. The dedicated foreign language access communication identifier acts as a trigger switch, and its physical manifestation includes a specific virtual phone number, a specific IP address mapping, a preset communication port, or an access domain name. For cross-language call scenarios, the smart communication terminal applies for and binds this dedicated identifier to the core network during the hardware registration phase. When an external call hits this identifier, the mobile switching center or core network signaling gateway routes the call request to the smart communication terminal's specific logical channel.

[0029] Then, step S2 is executed to identify the dedicated foreign language access communication identifier / audio-visual data stream. The system's underlying routing module resides in the operating system's kernel space and determines whether the current access request belongs to the translation service scope by performing deep parsing of the socket descriptor or signaling packet header. If the access request matches the dedicated foreign language access communication identifier, the system kernel immediately blocks the standard audio path to prevent the original audio stream from directly entering the local audio decoder. In audio-visual playback scenarios, when a user accesses overseas video websites or streaming media applications through the smart communication terminal, the system's underlying routing module intercepts data requests in the media framework, identifies the URL or data packet characteristics belonging to the external network audio-visual source, and thus triggers subsequent redirection logic.

[0030] In step S3, the system's underlying routing is directed to the AI ​​agent cloud service. The underlying routing module utilizes virtual sound card technology or audio stream hook functions to encapsulate the acquired raw audio stream or extracted foreign language audio track into real-time transmission protocol data packets. The intelligent communication terminal then transmits these data packets through an encrypted secure tunnel to the AI ​​agent cloud service module deployed on a global cloud platform. During this process, the intelligent communication terminal's underlying system forcibly executes redirection instructions to ensure that the original foreign language signal is not cached or played locally in any way. This system-level native routing mechanism enables data uploading to the cloud without requiring third-party applications, ensuring the mandatory and stable operation of the translation service from the ground up.

[0031] During the cloud processing phase, step S4 is executed: speech / audio track recognition, translation, and speech synthesis. After receiving the uplink audio stream, the AI ​​agent cloud service module first invokes the national language encoding matching module. This module automatically identifies the encoding type of the original language by analyzing the spectral envelope features, phoneme distribution patterns, and associated signaling metadata of the speech stream. For example, when the source language is identified as English, the module automatically loads the corresponding English acoustic and language models. Subsequently, the speech recognition unit within the AI ​​agent cloud service module converts the audio signal into a text stream. Next, the machine translation unit performs text-to-text conversion using a neural network translation model based on a preset target language (usually the user's native language). Finally, the speech synthesis unit generates a natural-sounding native language speech stream based on the translated native language text.

[0032] While generating native language speech, the core step S5 is executed, where the original foreign language speech / audio track is masked and discarded. This original audio masking and discarding module, as the endpoint of the cloud processing chain, executes a strict data filtering protocol. This module receives raw audio data from the recognition unit and native language speech data from the synthesis unit, and its internal logic forcibly cuts off the downlink transmission path of the raw audio data. At the memory management level, after the native language speech stream is encapsulated, the original audio masking and discarding module immediately performs an erase operation on the memory buffer storing the raw foreign language speech and releases the relevant handles. This means that the bitstream transmitted back from the cloud to the intelligent communication terminal physically contains only the translated native language sample data. For audio-visual scenarios, this module executes audio-video separation instructions, discarding only the foreign language voice track while retaining the background sound effects and ambient sounds in the video, and then mixing and reconstructing them with the synthesized native language voice.

[0033] Finally, in step S6, the terminal outputs only the native language voice / audio track. The intelligent communication terminal receives a pure native language voice stream from the cloud via a downlink communication link. The system's underlying routing module takes over this audio data and sends it to the audio playback interface of the hardware abstraction layer. Since the original foreign language signal has been completely discarded in step S5, the terminal hardware's speaker or receiver only reproduces the translated native language sound. During calls or movie viewing, the user's auditory system receives only a single, clear native language information, completely eliminating auditory interference caused by the superposition of the original and translated sounds.

[0034] In cross-border business call scenarios, the intelligent communication terminal is a smartphone. The user activates a dedicated foreign language access communication identifier in the phone settings, which corresponds to an international virtual number. When a partner located abroad dials this virtual number, the call request, as the external input source, triggers the phone's kernel's monitoring mechanism. The system's underlying routing module automatically takes over the call link without activating any translation app. The phone's microphone captures the user's native language speech and sends it to the cloud via step S3. Simultaneously, the other party's voice transmitted back from the cloud has been processed by the AI ​​intelligent agent's cloud service module. Due to the original audio filtering and discarding module, the other party's original English or French speech is intercepted and destroyed in the cloud. The user hears fluent Chinese translation directly through the phone's earpiece. Because the entire process is executed natively at the system's underlying level, call latency is controlled to the millisecond level, and the call interface is identical to a regular phone call, greatly enhancing the user experience.

[0035] In the scenario of watching overseas streaming video, the smart communication terminal is a smart TV. When a user opens a pre-installed overseas video aggregation application and plays a foreign language movie, the external input source appears as encrypted streaming media slice data. The system's underlying routing module identifies that the streaming media source belongs to an external network resource at the network protocol stack level, and then initiates an audio track stripping process. The video stream continues to undergo hardware decoding locally to ensure 4K or 8K high-definition picture quality, while the original foreign language audio track is redirected to the AI ​​intelligent agent cloud service module. In the cloud, the national language encoding matching module identifies that the movie dialogue is German and drives the translation engine to convert it into Chinese. The original audio masking and discarding module removes the original German audio track from the downlink packets after synthesizing Chinese dubbing. Finally, the smart TV's audio system receives the Chinese audio track aligned with the video timestamp. The user sees the original video and hears the real-time synthesized Chinese dubbing, without the need for external subtitles.

[0036] When the intelligent communication terminal executes the above process, its internal native language output module uses direct memory access technology to directly transfer the received audio samples to the output register of the audio chip. This technical solution bypasses the audio mixer at the operating system application layer, ensuring the highest priority for translated speech. Simultaneously, the system's underlying routing module dynamically adjusts the uplink audio sampling rate and compression ratio based on network fluctuations to adapt to the current bandwidth environment, ensuring stable data redirection even under weak network conditions.

[0037] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for AI-based cloud-based cross-language communication and external network audio-visual translation, characterized in that, Includes the following steps: S1. Preset and activate a dedicated foreign language access communication identifier independent of the local physical number in the system's underlying communication module of the intelligent communication terminal; S2. When the intelligent communication terminal detects an external input source request based on the exclusive foreign language access communication identifier, the system's underlying routing module redirects the transmission path of the original audio signal to the AI ​​intelligent agent cloud service module. S3. The AI ​​intelligent agent cloud service module receives the original audio signal, and the national language encoding matching module determines the source language type, and then performs text conversion, machine translation and native language speech synthesis to generate the target speech stream; S4. The original audio signal is physically marked and deleted by the original audio shielding and discarding module in the cloud, and only the synthesized target speech stream is encapsulated into downlink data packets. S5. The intelligent communication terminal receives the downlink data packet and outputs pure native language speech through the audio synthesizer at the system's bottom layer.

2. The method for AI-based cloud-based cross-language communication and external network audio-visual translation according to claim 1, characterized in that: The dedicated foreign language access communication identifier includes a virtual communication sub-number, an Internet Protocol address, a preset communication port number, and an encrypted Uniform Resource Locator.

3. The method for AI-based cloud-based cross-language communication and external network audio-visual translation according to claim 1, characterized in that: The kernel space of the intelligent communication terminal is equipped with a low-level listener. After the low-level listener detects an incoming call interruption, it redirects the audio stream to the access gateway of the AI ​​intelligent agent cloud service module by modifying the socket options.

4. The method for AI-based cloud-based cross-language communication and external network audio-visual translation according to claim 1, characterized in that: The specific steps of the intelligent communication terminal processing the original audio signal in step S2 include: the intelligent communication terminal encapsulates the collected pulse code modulation data stream into real-time transmission protocol data packets and transmits them transparently through the routing table configuration at the system's bottom layer.

5. The method for AI-based cloud-based cross-language communication and external network audio-visual translation according to claim 4, characterized in that: The intelligent communication terminal uses the audio policy manager to modify the audio routing matrix, adjusts the sampling rate of the raw pulse code modulation data from the microphone to a fixed frequency that matches the cloud algorithm, and disables the mixing feedback in the local echo cancellation module.

6. The method for AI-based cloud-based cross-language communication and external network audio-visual translation according to claim 1, characterized in that: Step S3 also includes: the AI ​​intelligent agent cloud service module establishes a conversation state machine to manage translation tasks, uses streaming speech recognition technology to output text fragments in real time, and selects a matching timbre library to synthesize native language speech based on the identified speaker characteristics.

7. The method for AI-based cloud-based cross-language communication and external network audio-visual translation according to claim 1, characterized in that: The AI ​​intelligent agent cloud service module adopts a dual buffer mechanism. The first buffer is used to temporarily store the received raw audio frames, and the second buffer is used to construct downlink data packets. After feature extraction is completed, the corresponding data frames in the first buffer are marked as invalid and cleaned up by the garbage collection mechanism.

8. The method for AI-based cloud-based cross-language communication and external network audio-visual translation according to claim 1, characterized in that: When the external input source is an external network audio and video data stream, the system's underlying routing module performs an audio-video separation operation. The original audio masking and discarding module discards the original foreign language voice and retains the background sound effects. The synthesized native language voice and the retained background sound are then remixed.

9. A cloud-based AI-powered cross-language calling and external network audio-visual translation system, characterized in that, include: The intelligent communication terminal integrates a system kernel-level system routing module. A dedicated foreign language access identifier module is used to store and manage dedicated foreign language access communication identifiers deployed in the registry of the intelligent communication terminal system; The AI ​​intelligent agent cloud service module establishes a full-duplex communication link with the intelligent communication terminal through a network interface; The national language encoding matching module is electrically connected to the AI ​​intelligent agent cloud service module and is used to automatically determine the source language; The original audio blocking and discarding module is integrated into the output end of the AI ​​intelligent agent cloud service module, and is used to intercept the original audio frames and release the corresponding memory space.

Images