A multi-modal offline translation terminal and method based on edge computing
By integrating heterogeneous computing system-on-a-chip through a multimodal offline translation terminal based on edge computing, it supports multimodal input and completes translation locally, solving the problems of existing devices being unable to be compatible with multiple signal sources and lacking security, and realizing efficient and secure real-time bilingual subtitle output.
Patent Information
- Application Number
- CN202610856032.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-12
- Publication Date
- 2026-07-24
AI Technical Summary
Existing translation devices are incompatible with multiple audio and video signal sources, and insufficient computing power leads to low translation accuracy and high latency, as well as the risk of data leakage, failing to meet the professional and privacy requirements of business and confidential occasions.
It adopts a multimodal offline translation terminal based on edge computing, integrates a heterogeneous computing system-on-a-chip, supports multimodal input interfaces, has local large-scale model computing power, realizes real-time bilingual subtitle output through heterogeneous computing resource scheduling, and completes the speech recognition, text extraction and translation process locally, avoiding data uploading.
It achieves multimodal translation covering all scenarios, with ultimate privacy and security, low latency and high image quality translation experience, suitable for scenarios such as cross-border conferences, foreign language teaching and live events, ensuring information security and real-time subtitle synchronization.
Smart Images

Figure CN122452586A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of artificial intelligence and multimedia processing technology, and in particular to a multimodal offline translation terminal and method based on edge computing. Background Technology
[0002] Current translation devices mainly fall into two categories: handheld translators, which can only capture speech through a microphone and cannot process text information from video feeds or conference projections, and their interaction methods are not professional enough for formal business occasions; and cloud-based software translation solutions, which heavily rely on the network environment, pose a risk of data leakage, and cannot meet the privacy requirements of medical, legal, and confidential meetings. Furthermore, existing translation devices with screens typically have limited computing power, making it difficult to run large language models smoothly locally, resulting in low translation accuracy and high latency.
[0003] Therefore, there is an urgent need for a hardware terminal that can be compatible with multiple audio and video signal sources, has local large-scale model computing power, and can output bilingual subtitles in real time. Summary of the Invention
[0004] Based on this, the embodiments of this application provide a multimodal offline translation terminal and method based on edge computing, which is compatible with multiple audio and video signal sources, has local large-scale model computing power, and can output bilingual subtitles in real time.
[0005] Firstly, a multimodal offline translation terminal based on edge computing is provided. This terminal includes a main control unit, a multimodal input interface module, a local storage module, a high-definition output module, and a live streaming module. Specifically:
[0006] The main control unit adopts a heterogeneous computing system-on-a-chip that integrates a neural network processing unit and a video processing unit, and is used to schedule heterogeneous computing resources for each functional module.
[0007] The multimodal input interface module, including but not limited to an HDMI input interface, a UVC camera interface, a network stream interface, and a local storage medium interface, is used to collect audio and video signals from multiple sources and transmit them to the main control unit.
[0008] The local storage module is used to store quantized offline large language models, speech recognition models and optical character recognition models with parameter sizes of 1B-7B.
[0009] The high-definition output module includes an HDMI output interface for outputting processed audio and video signals;
[0010] The live streaming module is used to compress the audio and video processed by the high-definition output module using H264 / H265 and then stream it over the network.
[0011] The main control unit specifically includes: performing hardware decoding or decapsulation processing on the incoming audio and video streams through the video processing unit to extract the audio stream and video frames; calling the speech recognition model in the local storage module to infer the audio stream to obtain source language speech text, and calling the optical character recognition model to infer the video frames to obtain on-screen text information; inputting the source language speech text or the on-screen text information into the quantized offline large language model for local translation inference to generate target language text; and when the presence of original subtitles in the video frame is detected, calling the image restoration algorithm to perform pixel-level background filling on the original subtitle area to erase the original subtitles, and rendering the target language text as a subtitle layer superimposed on the restored video screen, and outputting it in real time through the high-definition output module.
[0012] Optionally, the terminal further includes a full-duplex audio processing module, specifically comprising:
[0013] When the speech synthesis model generates audio and outputs it through the high-definition output module, the audio signal is fed back to the audio acquisition channel in real time as a reference signal. The digital signal processor or the neural network processing unit runs an adaptive echo cancellation algorithm to eliminate the acoustic interference of the locally synthesized speech on the microphone input audio.
[0014] Optionally, the network stream interface of the multimodal input interface module supports RTMP, RTSP, HTTP, HTTPS, UDP, m3u8, SRT or NDI protocols, and the local storage medium interface supports direct decoding of local media files in MP4 or MKV format.
[0015] Optionally, the quantized offline large language model stored in the local storage module adopts INT8 or INT4 quantization technology. The speech recognition model and optical character recognition model are both deployed locally and run on the neural network processing unit. All speech recognition, text extraction and translation processes are completed locally.
[0016] Optionally, when the optical character recognition model detects the presence of original foreign language subtitles in the video frame, the main control unit performs text erasure and redrawing of the video frame, specifically including:
[0017] The image restoration algorithm is invoked, and the GPU computing power is used to extend the texture according to the pixel color around the mask. The original subtitle area is filled with pixel-level background to erase the original subtitle. The 2D graphics acceleration engine is invoked to re-render the translated target language text back to the screen according to the font, size and color of the original subtitle.
[0018] Optionally, the main control unit further includes a heterogeneous computing resource scheduling module, which specifically includes:
[0019] The activity level of the input signal source is monitored in real time. When a drastic change in the picture is detected in the HDMI input source, the inference priority of the optical character recognition model is automatically increased. When human voice is detected in the audio stream, the inference priority of the speech recognition model is automatically increased.
[0020] The weight data of the quantized offline large language model is kept in on-chip memory by using a neural network acceleration processor, avoiding latency jitter caused by frequent DDR access.
[0021] Secondly, a multimodal offline translation method based on edge computing is provided, applied to a multimodal offline translation terminal as described in any of the first aspects above, the method comprising:
[0022] Signal Access and Preprocessing: The main control unit acquires the raw audio and video streams through the multimodal input interface module. If the input source is HDMI or UVC, it performs hardware decoding through the video processing unit. If it is a network stream, it performs decapsulation processing. If it is a local media file, it is directly decoded.
[0023] Multimodal AI reasoning: The audio stream is extracted and converted into source language speech text through the speech recognition model on the neural network processing unit; video frames are extracted and recognized as text information on the screen through the optical character recognition model on the neural network processing unit.
[0024] Local large model translation: The source language speech text or the on-screen text information is fed as input to a locally deployed quantized offline large language model with 1B-7B parameters. The inference is accelerated by using INT8 or INT4 quantization technology to generate the translated text of the target language.
[0025] Subtitle rendering and output: The graphics processing unit or 2D acceleration engine is called to render the translated target language text into a subtitle layer, which is then superimposed on the original video image through hardware and displayed in real time via the HDMI output interface.
[0026] Optionally, the method further includes a speech synthesis link, comprising:
[0027] The translated target language text is input into a locally deployed speech synthesis model. The neural network processing unit drives the text front-end model to generate a Mel spectrogram, and then calls the vocoder model to perform waveform synthesis to generate high-fidelity original PCM audio waveform data.
[0028] Optionally, the method further includes document generation, including:
[0029] The background system synchronously saves the identified source language text and the translated target language text by timestamp, automatically generating structured bilingual meeting documents that users can export via USB or TF card.
[0030] Optionally, the method further includes live streaming and image transmission, including:
[0031] The processed audio and video are compressed using H264 / H265 to generate a network stream, and then transmitted over the network using RTMP, SRT, WebRTC, NDI, or UVC network streaming protocols to achieve real-time sharing and long-distance image transmission of the translated video content.
[0032] The beneficial effects of the technical solutions provided in this application include at least the following:
[0033] (1) Full-scene coverage: The terminal internally uses parallel reasoning through the voice link (ASR model) and the visual link (OCR model) to achieve what is seen is what is translated. At the same time, through video screen text erasure and redrawing technology, the image restoration algorithm is called to fill the original subtitle area with pixel-level background to erase the original subtitles, and the 2D graphics acceleration engine is called to re-render the translated text back to the screen according to the original font parameters, supporting multiple layout modes, and achieving the physical disappearance of the original subtitles and the perfect integration of the new subtitles. In addition, the terminal integrates live streaming and structured document generation functions, which can cover multiple application scenarios such as cross-border conferences, foreign language teaching, film and television localization and live events. It can be seen that this application breaks through the single-modal limitation of traditional translators that only process voice, and is compatible with heterogeneous signal sources such as conference projection, camera screen, network live broadcast and local video through multi-modal input interfaces such as HDMI, UVC, network stream and local storage media.
[0034] (2) Ultimate Privacy and Security: All speech recognition, text extraction, and translation processes are completed locally on the NPU, without the need for an internet connection. The local storage module deploys quantized offline large language models (INT8 / INT4 quantization), speech recognition models, and optical character recognition models with parameters ranging from 1B to 7B. The main control unit calls these models to perform closed-loop inference locally, without uploading any data to the cloud or third-party servers. This architecture physically eliminates the risk of leakage of meeting content, trade secrets, or sensitive conversations, making it particularly suitable for scenarios with extremely high information security requirements, such as medical consultations, legal negotiations, confidential business dealings, and educational copyright protection.
[0035] (3) Low latency and high image quality: By utilizing high-performance heterogeneous computing SoC (such as RK3576) and its video processing unit and 2D graphics acceleration engine, the hardware encoding and decoding and hardware layer subtitle overlay are used to achieve millisecond-level subtitle synchronization and 4K / 8K high-definition image output, ensuring the meeting viewing experience. Attached Figure Description
[0036] To more clearly illustrate the embodiments of this application or the technical solutions in the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.
[0037] Figure 1 A schematic diagram of the hardware architecture of a multimodal offline translation system based on edge computing is provided for an embodiment of this application;
[0038] Figure 2 A flowchart illustrating the steps of a multimodal offline translation method based on edge computing, provided in an embodiment of this application. Detailed Implementation
[0039] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0040] In the description of this application, the terms "comprising," "having," and any variations thereof are intended to cover non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those steps or units that are expressly listed, but may also include other steps or units that are not expressly listed but are inherent to these processes, methods, products, or apparatuses, or steps or units added based on further optimizations conceived in this application.
[0041] This application provides a multimodal offline translation terminal based on edge computing. The specific application scenario refers to an embedded terminal device capable of accessing multiple signal sources such as HDMI, UVC, network streams, and local video files, and performing real-time speech recognition, text extraction, and large-scale model translation locally. The terminal includes a main control unit, a multimodal input interface module, a local storage module, a high-definition output module, and a live streaming module. Specifically:
[0042] The main control unit employs a heterogeneous computing system-on-a-chip (SoC) integrating a neural network processing unit and a video processing unit, used for heterogeneous computing resource scheduling of various functional modules. In the embodiments of this application, the main control unit specifically adopts a high-performance heterogeneous computing SoC (including RK3576) with ≥6 TOPS NPU computing power, and integrates a high-performance neural network processing unit (NPU, computing power ≥6 TOPS) and a video processing unit (VPU).
[0043] A multimodal input interface module, including but not limited to an HDMI input interface, a UVC camera interface, a network stream interface, and a local storage media interface, is used to acquire audio and video signals from various sources and transmit them to the main control unit. In this embodiment, it includes an HDMI input interface, a UVC (USB Video Class) camera interface, and a 1000M Ethernet / WiFi 6 network stream interface, specifically used to acquire audio and video signals from external conferences, real-time footage from a local camera, or live web streaming.
[0044] The local storage module is used to store quantized offline large language models, speech recognition models (such as HY-MT1.5), and optical character recognition (OCR) models with parameters ranging from 1B to 7B.
[0045] The high-definition output module includes an HDMI output interface for outputting processed audio and video signals. Specifically, it supports pixel-level restoration of original video frames and real-time generation of bilingual subtitles. Audio is then overlaid at a hardware layer to enable live streaming, recording, and output to display devices. The recording module can record the processed video signal in ultra-high definition 4K 60Hz.
[0046] The live streaming module is used to compress the audio and video processed by the high-definition output module using H264 / H265 and then stream it over the network (RTMP, SRT, WebRTC, NDI, UVC, etc.) to achieve network sharing and long-distance transmission.
[0047] The main control unit specifically includes: hardware decoding or decapsulation of the incoming audio and video streams through the video processing unit to extract the audio stream and video frames; inference of the audio stream by calling the speech recognition model in the local storage module to obtain the source language speech text, and inference of the video frame by calling the optical character recognition model to obtain the on-screen text information; inputting the source language speech text or on-screen text information into the quantized offline large language model for local translation inference to generate the target language text; and when the original subtitles are detected in the video frame, calling the image restoration algorithm to fill the original subtitle area with pixel-level background to erase the original subtitles, and rendering the target language text as a subtitle layer superimposed on the restored video screen, and outputting it in real time through the high-definition output module.
[0048] In this embodiment, the working logic and technical solution of this terminal are as follows:
[0049] Signal Input and Preprocessing: The main control unit acquires the raw audio and video streams through the multimodal input interface module. If the input source is HDMI or UVC, hardware decoding is performed by the VPU; if it is a network stream (RTMP, RTSP, HTTP, HTTPS, UDP, .m3u8, SRT, NDI), decapsulation processing is performed. If it is a local media file (such as MP4, MKV format), it is directly decoded.
[0050] Multimodal AI inference:
[0051] Voice link: Extract the audio stream, infer it on the NPU via the ASR model, and convert it into source language text in real time.
[0052] Visual Link: Extract video frames, infer them on the NPU via an OCR model, and identify subtitle information in the original image.
[0053] Local large-scale model translation: The speech text output by ASR or the on-screen text output by OCR is fed as input to a locally deployed large language model. The NPU uses INT8 / INT4 quantization technology to accelerate inference and generate translated text in the target language.
[0054] Text-to-Speech (TTS) Linkage:
[0055] The translated target language text is input into a locally deployed text-to-speech (TTS) model. The NPU first drives the text front-end model to generate a Mel spectrogram, then calls the vocoder model to perform waveform synthesis, and finally generates high-fidelity original PCM audio waveform data.
[0056] Audio processing and echo cancellation:
[0057] The terminal has a built-in full-duplex audio processing module. When the TTS model generates audio and outputs it through the HDMI / audio interface, the system feeds back this audio signal as a reference signal to the audio acquisition channel in real time. The adaptive echo cancellation (AEC) algorithm is then run using the RK3576's DSP (Digital Signal Processor) or NPU computing power.
[0058] Subtitle rendering and output: The main control unit calls the graphics processing unit (GPU) or 2D acceleration engine (RGA) to render the translated target language text as a subtitle layer, which is then overlaid on the original video image through hardware and finally displayed in real time through the HDMI output interface.
[0059] Document generation: The system backend synchronously saves the identified source text and the translated target text according to the timestamp, and automatically generates a structured bilingual meeting document.
[0060] Live streaming and image transmission: The processed audio and video are compressed using H264 / H265 to generate a network stream, and then transmitted over the network using network streaming protocols such as RTMP, SRT, WebRTC, NDI, and UVC.
[0061] In optional embodiments of this application, the main control unit computing power scheduling scheme specifically includes:
[0062] Heterogeneous computing resource scheduling module
[0063] To smoothly run ASR (speech recognition), OCR (on-screen text recognition), and LLM (large model translation) simultaneously on the RK3576's 6TOPS NPU computing power, this terminal introduces a dynamic priority preemption mechanism:
[0064] Scene Awareness Unit: The system monitors the activity level of the input signal source in real time. When a drastic change in the image is detected from the HDMI input source, the inference priority of the OCR model is automatically increased; when human voice is detected in the audio stream (VAD detection), the priority of the ASR model is automatically increased.
[0065] Memory bandwidth optimization: Utilizing the NNAP (Neural Network Accelerator Processor) feature of the RK3576, the weight data of the HY-MT1.5 large model is kept in on-chip SRAM to avoid latency jitter caused by frequent access to DDR, ensuring that the system can still maintain an inference frame rate of over 30fps when overlaying subtitles on 4K video streams.
[0066] As can be seen, the computing power scheduling scheme of this application has extremely smooth performance: it solves the stuttering problem caused by multiple models running concurrently under limited computing power, and realizes a smooth experience of "watching videos, translating and generating subtitles at the same time".
[0067] In optional embodiments of this application, the screen erasure and redraw process specifically includes:
[0068] Original subtitle erasure technology: When the system detects the presence of original foreign subtitles in the video (OCR detection), it not only extracts the text for translation, but also calls the image inpainting algorithm. Utilizing the GPU computing power of the RK3576, it performs pixel-level background filling (erasure) of the foreign subtitle area in the original image to remove interference information.
[0069] Bilingual subtitle generation: The translated target language text is re-rendered back onto the screen according to the font, size, and color of the original subtitles. It supports multiple layout modes such as "English at the top, Chinese at the bottom" and "original on the left, translation on the right," and the font edges are anti-aliased, with clarity comparable to the original video encoding.
[0070] The user plays an English MP4 movie file via USB drive. After the terminal loads the file, it automatically starts the OCR module to scan each frame. When white English subtitles are detected at the bottom of the screen, the system uses an image restoration model to smooth out the white text area, and then renders the translated Chinese subtitles in bright yellow font in their original position. What the user sees on the large screen is no longer a layer of "English subtitles + Chinese subtitles," but a clean "pure Chinese subtitle version" of the movie, achieving a localized "embedded subtitle" effect.
[0071] After translation, users' video content can be streamed and transmitted via video, meaning the translated video content can be streamed to a live streaming platform or transmitted over a long distance via a wide area network.
[0072] like Figure 1 A hardware architecture diagram of a multimodal offline translation terminal based on edge computing is presented. The core control unit is the RK3576 heterogeneous computing SoC (with a built-in ≥6TOPS NPU). It receives HDMI input signals via an RK628F, UVC camera signals via a USB interface, and network stream signals via an FG6252BASR (WiFi 6) and an RTL8211 (1000M wired network), enabling multimodal audio and video access. The terminal is configured with DDR (8G) and eMMC (64G) as local storage modules to deploy quantized offline large language models (LLM) with 1B-7B parameters, as well as ASR and OCR models. All speech recognition, text extraction, and translation inference are completed in the local NPU. The processed audio and video signals are output to a display device via HDMI Out (displaying bilingual Chinese and English subtitles). It also supports UVC Out and network sharing via live streaming and image transmission modules. In addition, the system includes a 12V DC power supply (managed by an RK806S), a TF card expansion interface, and remote device upgrade functionality via WiFi 6.
[0073] Based on the above solution, specific embodiments of this application are given as follows:
[0074] Example 1: Cross-border video communication
[0075] In multinational video conferencing scenarios, users connect the HDMI output of their conference computer to the HDMI input port of this terminal, and then connect the HDMI output of this terminal to the conference room's large screen. The terminal automatically detects the signal, activates its local ASR model to recognize English speech, and translates it into Chinese in real time using its local large model. The translated Chinese subtitles are displayed at the bottom of the large screen through a hardware overlay layer. Simultaneously, the terminal automatically generates a bilingual (Chinese-English) meeting minutes file, which users can export via USB or TF card.
[0076] Example 2: Semantic-level repair and redrawing of video sources
[0077] When a user connects a USB flash drive containing English movies via USB, the terminal loads the MP4 file stream. The system activates the OCR detection module, scanning the bottom area of the screen frame by frame. When a high-contrast foreign language subtitle area is detected, the image restoration process is triggered:
[0078] Background filling: The GPU parallel computing unit of RK3576 or higher-performance SOC is called to use a deep learning-based image inpainting algorithm to extend the texture according to the pixel color around the mask and "smooth out" the original subtitle area.
[0079] Subtitle redrawing: Invoke RGA (2D graphics acceleration engine) to render the translated Chinese subtitles in the repaired area using a specified font (such as Source Han Sans) and color (such as gold).
[0080] The final output achieved a physical-level disappearance of the original subtitles and a perfect fusion with the new subtitles.
[0081] Example 3: Immersive Foreign Language Teaching Scenario
[0082] Suitable for foreign language teaching scenarios, it can erase the text in the foreign language textbook screen in real time and replace it with the native language explanation. In the language teaching scenario, users (such as students) can access foreign language teaching video sources (such as original English documentaries and videos accompanying foreign language textbooks) through the HDMI interface, or load foreign language learning courseware through local storage media.
[0083] Semantic-level redrawing and annotation: Bilingual comparison mode: The translated target language (such as Chinese) text is rendered on one side of the screen as a semi-transparent overlay or sidebar for beginners to compare and understand.
[0084] Immersive Replacement Mode: The translated Chinese text is re-rendered back to its original position, following the font style of the original video subtitles. Students then see the original video with Chinese dubbing, achieving barrier-free "native language" comprehension and greatly reducing the listening comprehension threshold.
[0085] Improved teaching efficiency: By using the technology of "erasing the original subtitles and redrawing the translation", the pain point of "staring at the subtitles but not being able to hear the pronunciation" in traditional foreign language learning is solved, and immersive learning with synchronized audio-visual learning is achieved.
[0086] Offline privacy protection: All teaching video content is processed locally without uploading to the cloud, protecting the teaching copyright of educational institutions and the privacy data of students.
[0087] Example 4: Live streaming and image transmission are applicable to various types of live streaming activities. The original video is translated and processed, and can be streamed and transmitted through the device to realize the sharing and dissemination of the video.
[0088] like Figure 2 This application also provides a multimodal offline translation method based on edge computing, which may include:
[0089] Signal Access and Preprocessing: The main control unit acquires the raw audio and video streams through the multimodal input interface module. If the input source is HDMI or UVC, it performs hardware decoding through the video processing unit. If it is a network stream, it performs decapsulation processing. If it is a local media file, it is directly decoded.
[0090] Multimodal AI reasoning: The audio stream is extracted and converted into source language speech text through the speech recognition model on the neural network processing unit; video frames are extracted and recognized as text information on the screen through the optical character recognition model on the neural network processing unit.
[0091] Local large model translation: The source language speech text or on-screen text information is fed as input to a locally deployed quantized offline large language model with 1B-7B parameters. The inference is accelerated by using INT8 or INT4 quantization technology to generate translated text in the target language.
[0092] Subtitle rendering and output: The graphics processing unit or 2D acceleration engine is called to render the translated target language text into a subtitle layer, which is then superimposed on the original video image through hardware and displayed in real time via the HDMI output interface.
[0093] In this embodiment of the application, the method further includes a speech synthesis link, comprising:
[0094] The translated target language text is input into the locally deployed speech synthesis model. The neural network processing unit drives the text front-end model to generate a Mel spectrogram, and then calls the vocoder model to perform waveform synthesis to generate high-fidelity original PCM audio waveform data.
[0095] The method also includes document generation, including:
[0096] The background system synchronously saves the identified source language text and the translated target language text by timestamp, automatically generating structured bilingual meeting documents that users can export via USB or TF card.
[0097] The methods also include live streaming and image transmission, including:
[0098] The processed audio and video are compressed using H264 / H265 to generate a network stream, and then transmitted over the network using RTMP, SRT, WebRTC, NDI, or UVC network streaming protocols to achieve real-time sharing and long-distance image transmission of the translated video content.
[0099] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods.
[0100] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0101] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A multimodal offline translation terminal based on edge computing, characterized in that, The terminal includes a main control unit, a multimodal input interface module, a local storage module, a high-definition output module, and a live streaming module, specifically: The main control unit adopts a heterogeneous computing system-on-a-chip that integrates a neural network processing unit and a video processing unit, and is used to schedule heterogeneous computing resources for each functional module. The multimodal input interface module, including but not limited to an HDMI input interface, a UVC camera interface, a network stream interface, and a local storage medium interface, is used to collect audio and video signals from multiple sources and transmit them to the main control unit. The local storage module is used to store quantized offline large language models, speech recognition models and optical character recognition models with parameter sizes of 1B-7B. The high-definition output module includes an HDMI output interface for outputting processed audio and video signals; The live streaming module is used to compress the audio and video processed by the high-definition output module using H264 / H265 and then stream it over the network. The main control unit specifically includes: performing hardware decoding or decapsulation processing on the incoming audio and video streams through the video processing unit to extract the audio stream and video frames; calling the speech recognition model in the local storage module to infer the audio stream to obtain source language speech text, and calling the optical character recognition model to infer the video frames to obtain on-screen text information; inputting the source language speech text or the on-screen text information into the quantized offline large language model for local translation inference to generate target language text; and when the presence of original subtitles in the video frame is detected, calling the image restoration algorithm to perform pixel-level background filling on the original subtitle area to erase the original subtitles, and rendering the target language text as a subtitle layer superimposed on the restored video screen, and outputting it in real time through the high-definition output module.
2. The multimodal offline translation terminal based on edge computing according to claim 1, characterized in that, The terminal also includes a full-duplex audio processing module, specifically comprising: When the speech synthesis model generates audio and outputs it through the high-definition output module, the audio signal is fed back to the audio acquisition channel in real time as a reference signal. The digital signal processor or the neural network processing unit runs an adaptive echo cancellation algorithm to eliminate the acoustic interference of the locally synthesized speech on the microphone input audio.
3. The multimodal offline translation terminal based on edge computing according to claim 1, characterized in that, The network stream interface of the multimodal input interface module supports RTMP, RTSP, HTTP, HTTPS, UDP, m3u8, SRT or NDI protocols, and the local storage medium interface supports direct decoding of local media files in MP4 or MKV format.
4. The multimodal offline translation terminal based on edge computing according to claim 1, characterized in that, The quantized offline large language model stored in the local storage module adopts INT8 or INT4 quantization technology. The speech recognition model and optical character recognition model are both deployed locally and run on the neural network processing unit. All speech recognition, text extraction and translation processes are completed locally.
5. The multimodal offline translation terminal based on edge computing according to claim 1, characterized in that, When the optical character recognition model detects the presence of original foreign language subtitles in the video frame, the main control unit performs text erasure and redrawing of the video frame, specifically including: The image restoration algorithm is invoked, and the GPU computing power is used to extend the texture according to the pixel color around the mask. The original subtitle area is filled with pixel-level background to erase the original subtitle. The 2D graphics acceleration engine is invoked to re-render the translated target language text back to the screen according to the font, size and color of the original subtitle.
6. The multimodal offline translation terminal based on edge computing according to claim 1, characterized in that, The main control unit also includes a heterogeneous computing resource scheduling module, which specifically includes: The activity level of the input signal source is monitored in real time. When a drastic change in the picture is detected in the HDMI input source, the inference priority of the optical character recognition model is automatically increased. When human voice is detected in the audio stream, the inference priority of the speech recognition model is automatically increased. The weight data of the quantized offline large language model is kept in on-chip memory by using a neural network acceleration processor, avoiding latency jitter caused by frequent DDR access.
7. A multimodal offline translation method based on edge computing, applied in the multimodal offline translation terminal based on edge computing as described in any one of claims 1-6, characterized in that, The method includes: Signal Access and Preprocessing: The main control unit acquires the raw audio and video streams through the multimodal input interface module. If the input source is HDMI or UVC, it performs hardware decoding through the video processing unit. If it is a network stream, it performs decapsulation processing. If it is a local media file, it is directly decoded. Multimodal AI reasoning: The audio stream is extracted and converted into source language speech text through the speech recognition model on the neural network processing unit; video frames are extracted and recognized as on-screen text information through the optical character recognition model on the neural network processing unit. Local large model translation: The source language speech text or the on-screen text information is fed as input to a locally deployed quantized offline large language model with 1B-7B parameters. The inference is accelerated by using INT8 or INT4 quantization technology to generate the translated text of the target language. Subtitle rendering and output: The graphics processing unit or 2D acceleration engine is called to render the translated target language text into a subtitle layer, which is then superimposed on the original video image through hardware and displayed in real time via the HDMI output interface.
8. The multimodal offline translation method based on edge computing according to claim 7, characterized in that, The method also includes a speech synthesis link, comprising: The translated target language text is input into a locally deployed speech synthesis model. The neural network processing unit drives the text front-end model to generate a Mel spectrogram, and then calls the vocoder model to perform waveform synthesis to generate high-fidelity original PCM audio waveform data.
9. The multimodal offline translation method based on edge computing according to claim 7, characterized in that, The method also includes document generation, including: The background system synchronously saves the identified source language text and the translated target language text by timestamp, automatically generating structured bilingual meeting documents that users can export via USB or TF card.
10. The multimodal offline translation method based on edge computing according to claim 7, characterized in that, The method also includes live streaming and image transmission, including: The processed audio and video are compressed using H264 / H265 to generate a network stream, and then transmitted over the network using RTMP, SRT, WebRTC, NDI, or UVC network streaming protocols to achieve real-time sharing and long-distance image transmission of the translated video content.
