Real-time translation recognition system under cloud service framework

Abstract

The invention discloses a real-time translation recognition system under a cloud service framework, belongs to the technical field of real-time translation, and solves the problems that an existing translation system is insufficient in real-time performance, poor in scene adaptability, weak in privacy protection, slow in model evolution and the like. The dynamic model management engine obtains adaptive slices from a model slice factory according to scenes, equipment states and network quality and distributes the adaptive slices to edges, and the adaptive slices are distributed to a cloud-side collaborative reasoning system; the cloud-side collaborative reasoning system comprises a cloud-side collaborative reasoning system, a cloud-side collaborative reasoning system, a cloud-side collaborative reasoning system and a cloud-side collaborative reasoning system; the multi-modal perception engine fuses audio, images and dialogue history to generate a structured context vector and improve translation context fitting degree, the cloud edge cooperation engine takes an edge model as a core, processes different complexity tasks in combination with a cloud end, and constructs a data closed loop by incremental learning and a federation engine to realize model optimization and privacy protection; according to the system, the real-time performance, accuracy and safety are improved through cloud edge collaboration, dynamic adaptation and continuous learning, and the system is suitable for multi-scene real-time translation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of real-time translation, more particularly, to a real-time translation recognition system under a cloud service framework. BACKGROUND

[0002] With the acceleration of globalization and the increasing frequency of cross-cultural exchanges, real-time translation recognition systems have become a core tool for breaking down language barriers and are widely used in international conferences, cross-border business, travel, and many other scenarios. Traditional translation systems rely on a pure cloud architecture, where user-collected data such as voice and text are uploaded to the cloud server, and the cloud model performs translation processing before returning the results. This model can take advantage of the powerful computing power and complex models of the cloud, but it has significant limitations: first, the data transmission process is vulnerable to network bandwidth and latency, making it difficult to meet real-time requirements, especially in unstable mobile scenarios, where translation response lag is a major problem; second, the uploading of large amounts of raw data increases the risk of privacy leakage, especially when dealing with sensitive information such as personal conversations and business secrets, data security is difficult to guarantee.

[0003] To address the real-time issue, edge computing technology has been introduced into translation systems, with lightweight models deployed on user devices for preliminary processing. However, existing edge deployment solutions use fixed models that cannot be dynamically adjusted according to the scene - for example, professional fields such as medicine and law have significantly different translation needs than everyday conversations, and fixed models cannot meet the accuracy requirements of different scenarios; at the same time, the hardware performance (such as memory and computing power) of different user devices varies greatly, and fixed models may not be fully utilized on high-performance devices or may run slowly on low-performance devices.

[0004] In addition, existing systems do not make full use of multi-modal information. In actual translation scenarios, conversations are often accompanied by visual information (such as conference PPTs and product labels) and historical conversation context, and translation relying solely on audio or text can easily lead to ambiguity. At the same time, iterative optimization of the model faces the challenge of data closed loop: low-confidence results generated by the edge, user feedback and other valuable data are difficult to effectively return and used for model updating, making it difficult to continuously improve the performance of the model; and traditional centralized training requires the collection of large amounts of user data, further exacerbating the privacy protection dilemma.

[0005] With the continuous expansion of the size of pre-trained models, super-large-scale models can improve translation accuracy, but direct deployment on edge devices has the problem of high resource occupation. How to ensure model performance while achieving lightweight adaptation, how to coordinate the computing power of the cloud and the edge, and how to build a data closed loop that takes into account privacy protection and model evolution, are key issues that need to be addressed in the development of current real-time translation recognition systems. SUMMARY

[0006] The purpose of this invention is to provide a real-time translation and recognition system under a cloud service framework to solve the above-mentioned problems.

[0007] To achieve the above objectives, an embodiment of the present invention provides the following technical solution:

[0008] A real-time translation recognition system based on a cloud service framework includes:

[0009] The cloud service subsystem includes a large-scale pre-trained model library, a model slicing factory, and an incremental learning center;

[0010] The edge computing subsystem, deployed in user-side devices, is used to run lightweight task-specific models and collect multimodal data;

[0011] The dynamic model management engine connects the cloud service subsystem and the edge computing subsystem, and is used to generate and distribute adapted dynamic model slices from the model slice factory to the designated edge computing subsystem based on the received scene context information and device status information.

[0012] A multimodal context-aware engine is used to extract and fuse semantic information from audio streams, image streams, and dialogue history in real time to generate structured context vectors.

[0013] The cloud-edge collaborative inference engine takes the dynamic model slices built into the edge computing subsystem as its core and receives the structured context vector as conditional input to perform local preliminary translation; for complex tasks that exceed the local capabilities, it performs collaborative inference with the pre-trained model in the cloud service subsystem.

[0014] The incremental learning data closed-loop system is used to collect uncertainty data and user feedback data generated by the edge computing subsystem during the inference process, and send them back to the incremental learning center to optimize the ultra-large-scale pre-trained model library.

[0015] As a further improvement of the present invention, the decision-making basis of the dynamic model management engine includes:

[0016] Scene tags identified in real time by the multimodal context-aware engine, the scene tags including professional fields and dialogue formats;

[0017] The edge computing subsystem reports device performance parameters in real time, including available memory and CPU / GPU computing power;

[0018] Current real-time quality assessment metrics for the network include bandwidth, latency, and jitter.

[0019] As a further improvement of the present invention, the model slicing factory is configured as follows:

[0020] Receive a request from the dynamic model management engine, the request containing target scene labels and performance constraints;

[0021] From the aforementioned large-scale pre-trained model library, a dedicated sub-model that meets performance constraints and has high accuracy for the target scene is generated through structured pruning, knowledge distillation, or feature extraction.

[0022] The sub-model and its configuration file are packaged into dynamic model slices and distributed to the target edge computing subsystem.

[0023] As a further improvement of the present invention, the multimodal context-aware engine includes:

[0024] The scene recognition unit is used to identify the scene in which the dialogue occurs in real time based on visual elements in the current image stream and keywords in the audio stream.

[0025] The dialogue history management unit is used to maintain a fixed-length time window and store the text history of recent dialogues and their semantic embeddings.

[0026] The context vector synthesis unit is used to fuse and encode scene labels, current dialogue history semantic embeddings, and key text information identified from images to generate a unified context vector.

[0027] As a further improvement of the present invention, the cloud-edge collaborative inference engine, deployed within the edge computing subsystem, receives inputs including: a preprocessed audio feature sequence and a context vector generated by the context-aware engine; the context vector serves as a conditional input to the model, participating in the calculation of the attention mechanism at each step, thereby guiding the model to generate translation results highly relevant to the current context.

[0028] As a further improvement of the present invention, the incremental learning data closed-loop system includes:

[0029] The data acquisition unit is used to anonymously collect low-confidence inference results, results manually corrected by users, and raw data that triggers cloud-edge collaboration at the edge.

[0030] The data encryption and compression unit is used to perform privacy protection processing on the collected data before transmission back;

[0031] The incremental learning scheduler, located in the incremental learning center, is used to schedule the collected datasets to fine-tune specific models in the pre-trained model library, while ensuring that catastrophic forgetting does not occur after model updates through adversarial validation.

[0032] As a further improvement to the present invention, the system further includes:

[0033] The model performance evaluation and rolling update module is used for:

[0034] Monitor the performance metrics of dynamic model slices running on each edge computing subsystem;

[0035] When the average confidence or user satisfaction of a certain model slice is consistently higher than the threshold, it is triggered to be set as the recommendation benchmark model in that scenario.

[0036] Once the new incremental learning version is verified to be effective, the model slicing factory is notified to generate new slices based on the new version of the model, and these slices are gradually pushed to the edge computing subsystem for rolling updates.

[0037] As a further improvement of the present invention, the collaborative workflow of the cloud-edge collaborative inference engine includes:

[0038] The edge computing subsystem first uses local model slices for inference and calculates the output confidence in real time;

[0039] When the confidence level is higher than the first threshold, the local result is output directly.

[0040] When the confidence level is between the first and second thresholds, the local output and context vector are uploaded to the cloud for quick verification and refinement.

[0041] When the confidence level is below the second threshold or a completely unknown modal input is detected, the raw data is uploaded and a request is made to the cloud for full depth processing.

[0042] As a further improvement of the present invention, the system also includes a unified resource orchestrator for:

[0043] Monitor the status of all online edge computing subsystems and the load of the cloud computing resource pool;

[0044] When the performance of a certain edge computing subsystem device is insufficient, the instruction dynamic model management engine distributes a model slice with a smaller computational load to it.

[0045] When the cloud load is too high, the instruction dynamic model management engine appropriately increases the confidence threshold of the edge computing subsystem, leaving more computing tasks on the edge side to achieve global load balancing.

[0046] As a further improvement to the present invention, the system also includes an edge model personalized federated engine:

[0047] The engine runs locally on the edge computing subsystem and receives local data from the incremental learning data closed-loop system.

[0048] The engine generates local model update parameters by fine-tuning the currently running dynamic model slices using the local data on the local device.

[0049] The cloud service subsystem includes a federated aggregation center for encrypted collection of local model update parameters from multiple edge computing subsystems. These updates are aggregated through a federated averaging algorithm to generate a global, performance-enhanced, and privacy-protecting next-generation pre-trained model.

[0050] The model slicing factory generates new dynamic model slices based on the next-generation pre-trained model and distributes them to the edge computing subsystem through the dynamic model management engine, completing a federated learning loop.

[0051] Compared with the prior art, the advantages of this invention are:

[0052] (1) This invention achieves localized preliminary translation through an edge computing subsystem, which greatly reduces data transmission latency and can maintain a fast response, especially in unstable network scenarios. The dynamic model management engine distributes adapted model slices according to the scenario and device status, avoiding the contradiction of "large models cannot run and small models have low accuracy". The multimodal context awareness engine integrates audio, images and dialogue history to make the translation results more in line with the specific context and effectively reduce ambiguity. The cloud-edge collaborative reasoning mechanism uses a layered processing strategy to ensure fast local response while using cloud capabilities to solve complex translation tasks, taking into account both real-time performance and accuracy.

[0053] (2) The unified resource orchestrator of this invention achieves load balancing of cloud and edge computing resources through global monitoring and dynamic scheduling, avoiding resource idleness or overload; the lightweight model generated by the model slicing factory reduces the resource consumption of edge devices and adapts to hardware environments with different performance. In terms of privacy protection, the incremental learning data closed-loop system uses differential privacy and data encryption technology to process the returned data, and the edge model personalized federated engine replaces the original data upload with parameter aggregation, reducing the risk of sensitive information exposure; the inference mode with local processing as the main method further reduces the amount of data uploaded, protecting user privacy from the source.

[0054] (3) The closed-loop mechanism combining incremental learning and federated learning in this invention enables the system to continuously absorb new data and user feedback from the edge, constantly optimize the pre-trained model library, and avoid model performance stagnation. The model performance evaluation and rolling update module ensures the rapid promotion of high-quality models and maintains the system's high performance. At the same time, the edge model personalized federated engine supports lightweight fine-tuning based on user's local data, allowing the translation model to gradually adapt to the language habits of specific users (such as accents and common expressions), meet personalized translation needs, and improve user experience. Attached Figure Description

[0055] Figure 1 This is a diagram illustrating the overall system architecture and cloud-edge collaboration of the present invention.

[0056] Figure 2 This is a flowchart of the dynamic model slice generation and distribution process of the present invention;

[0057] Figure 3 This is a flowchart illustrating the workflow of the multimodal context-aware engine of the present invention.

[0058] Figure 4 This is a flowchart of the cloud-edge collaborative reasoning decision-making process of the present invention;

[0059] Figure 5 This is a flowchart of the incremental learning and federated learning closed-loop process of this invention. Detailed Implementation

[0060] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0061] Example:

[0062] A real-time translation recognition system based on a cloud service framework, specifically including the following:

[0063] I. Cloud Service Subsystem: Utilizing a distributed cluster architecture, this subsystem uses the Hadoop Distributed File System (HDFS) to shard model parameters, supporting efficient reading and writing of petabyte-scale model files. The integrated base models include: multilingual Transformer models (containing a 128-layer encoder and decoder, with 1.2 trillion parameters), cross-modal understanding models (a hybrid Transformer architecture integrating speech spectral features, image visual features, and text semantic features), and domain-specific models (optimized for 20 professional domains such as medicine, law, and finance, with each domain model adding 50,000+ domain terminology embedding layers on top of the general model). The model version management module employs a Git-like version control mechanism, recording the training data, hyperparameters, and performance metrics for each model version, supporting version backtracking and comparative analysis.

[0064] The core processing flow of the model slicing factory is divided into four stages: The first stage, the requirement analysis stage, uses Natural Language Processing (NLP) technology to analyze the request parameters of the dynamic model management engine, transforming the constraint of "medical scenario + memory < 512MB" into quantifiable metrics (such as model parameter count < 80 million, single-sentence inference time < 200ms); the second stage, the model selection stage, matching basic models (such as medical domain models) from the pre-trained model library and extracting network substructures strongly related to the scenario (such as medical terminology attention heads, drug name entity recognition layers); and the third stage, the lightweight processing stage, employing a hybrid compression strategy— —Structured pruning removes redundant convolutional kernels (pruning rate 30%-50%, preserving accuracy loss <2%). Knowledge distillation uses the original model as the teacher and trains student models through the softmax output with a temperature coefficient τ=3.0. Feature extraction retains the first 8 layers of the encoder as the basic feature extractor, and after freezing, it is connected to a lightweight decoder. In the slicing and packaging stage, the generated sub-models are stored in ONNX format. The configuration file contains metadata such as input and output tensor dimensions, inference dependency library versions, and scene label mapping tables. It is packaged into a .zip format slice package with a digital signature to ensure integrity.

[0065] The incremental learning center is deployed on a GPU cluster (containing 128 A100 graphics cards) and adopts a pipelined data processing architecture: the data access layer receives encrypted data packets uploaded from the edge via Kafka, decrypts them, and then stores them in a distributed database; the cleaning layer uses a rule engine (such as filtering data containing garbled characters or with a repetition rate >90%) and a machine learning model (anomaly detection algorithm to identify noisy samples) to purify the data and retain valid samples; the scene clustering layer uses BERT semantic vectors and the K-means algorithm to divide the data into 100+ subcategories based on "domain + language pair + scene complexity"; the fine-tuning training layer calls the corresponding base model for each category and uses mini-batch gradient descent (batch size = 32) and cosine annealing learning rate (initial 0.0001, minimum 0.00001) for 5-10 rounds of training, and evaluates the BLEU and TER values ​​on the validation set (accounting for 15%) after each round; the model fusion layer uses a model ensemble strategy (such as weighted average parameters) to fuse the fine-tuned model with the historical best model to avoid overfitting.

[0066] II. Edge Computing Subsystem: The hardware adaptation layer supports heterogeneous device deployment, including: mobile devices (smartphone chips such as Snapdragon 8 Gen3, with integrated NPU computing power up to 40 TOPS), embedded devices (Raspberry Pi 4B, etc., equipped with ARM Cortex-A72 CPU), and dedicated translation devices (with built-in neural network processing units, supporting INT8 quantization calculations). The device driver module optimizes computing library calls for different hardware (e.g., mobile devices call NNAPI, embedded devices call OpenCL) to maximize computing power utilization.

[0067] The local model execution module employs a three-level optimization strategy: Model quantization converts the FP32 model delivered from the cloud into INT8 or FP16 format, calculating quantization parameters using a calibration dataset (1000 parallel sentences) to ensure accuracy loss is less than 5%; Layer fusion merges consecutive convolutional layers and activation layers, multi-head attention layers and residual connections to reduce memory access frequency; Memory reuse utilizes memory pool technology, pre-allocating a fixed-size buffer (e.g., 512MB) and dynamically reusing tensor storage space during inference to reduce memory fragmentation. The inference engine is custom-developed based on ONNXRuntimeMobile, supporting dynamic shape input (adapting to speech segments of different lengths) and integrating hardware acceleration plugins (such as the Adreno acceleration plugin for mobile devices).

[0068] The multimodal data acquisition module includes: an audio acquisition unit (supporting 16kHz / 44.1kHz sampling rates, built-in Wiener filter for noise reduction, real-time extraction of Mel spectral features (40-dimensional) and Voice Activity Detection (VAD) markers); an image acquisition unit (adapted to front / rear cameras, resolution adjustable to 320×240-1920×1080, using JPEG compression (80% quality factor), identifying and cropping text regions using a lightweight object detection model (such as YOLOv5n); and a text acquisition unit (recording user input text, system output translation, user corrections, and associating timestamps with device IDs (anonymized through hashing)). All data is temporarily stored in a local SQLite database and managed according to a "one data block per hour" strategy; data not uploaded for more than 72 hours is automatically cleaned up.

[0069] III. Dynamic Model Management Engine: The decision-making system adopts a Deep Reinforcement Learning (DRL) framework. The state space includes: scene feature vector (32-dimensional, encoding domain, dialogue form, etc.), device performance vector (16-dimensional, including memory usage, CPU / GPU load, battery power, etc.), and network quality vector (8-dimensional, including bandwidth, latency, jitter, etc.). The action space is a candidate set of model slices (50+ predefined commonly used slices, classified according to scene and resource requirements). The reward function is designed as "α×precision score + β×(1 / latency) + γ×(1 / resource usage)", where α, β, and γ are weight coefficients (optimized to 0.5, 0.3, and 0.2 through offline training). The policy network is trained using the Proximal Policy Optimization (PPO) algorithm, taking the state vector as input and outputting the probability distribution of each action. The decision-making ability is continuously optimized through online learning (updating the network with new interaction data every 24 hours).

[0070] The communication module is built on a publish-subscribe architecture based on the MQTT protocol, with an MQTT broker deployed in the cloud (supporting 100,000+ concurrent connections). Edge devices, acting as clients, periodically (every 30 seconds) report status information (JSON format, compressed to <1KB). Model slice distribution employs a breakpoint resumption mechanism. Slice packets are divided into multiple 1MB data blocks. Edge devices verify the CRC32 value upon receipt, and request retransmission for missing blocks. The distribution priority scheduler sorts devices based on "device activity (usage frequency) + scenario urgency (e.g., real-time conferencing is higher than offline translation)," ensuring that high-priority devices receive slices first.

[0071] The monitoring and feedback module records the slice distribution success rate (target > 99.5%), edge device loading time (target < 3 seconds), and operational stability (crash rate < 0.1%) in real time, generating daily reports that are fed back to the model slice factory for optimizing slice size and compatibility. When the loading failure rate of a certain slice on 500+ devices exceeds 5%, an automatic rollback mechanism is triggered, pushing the previous version of the compatible slice to the relevant devices.

[0072] IV. Multimodal Context Awareness Engine: The scene recognition unit adopts a dual-channel fusion model: The visual branch inputs the cropped scene image, extracts features (1280 dimensions) using MobileNetV3, and obtains the image feature vector through global average pooling; the audio branch segments the speech stream into frames (20ms / frame), extracts acoustic features (39 dimensions) using MFCC, and inputs them into a BiLSTM (2 layers, 128 hidden layer dimensions) to obtain the audio feature vector; the fusion layer calculates the weights of the two types of features through an attention mechanism (visual feature weight 0.3-0.7, dynamically adjusted according to image clarity), and after weighted summation, it inputs into the fully connected layer, outputting the probability distribution of 10 scene labels (such as "medical consultation", "business negotiation", etc.), and takes the one with the highest probability as the final label.

[0073] The dialogue history management unit uses a circular buffer to store the most recent 10 rounds of dialogue. Each round of dialogue data includes: source language text, target language translation, timestamp, and semantic embedding vector (768-dimensional, generated by the MiniLM model). When a new dialogue is added, if the buffer is full, the oldest round of data is removed, and the historical semantic summary vector is updated (by averaging all embedding vectors). To reduce storage usage, the text content is compressed using the LZ77 algorithm, and the semantic embedding vector is quantized in INT8 format.

[0074] The context vector synthesis unit employs a hierarchical attention mechanism: the first layer calculates attention weights for the semantic embedding and scene label vector (32-dimensional) of each round of dialogue, highlighting historical information relevant to the current dialogue; the second layer fuses the feature vector of the image OCR text (extracted by a lightweight OCR model (such as CRNN, 512-dimensional)) with the output of the first layer, and controls the integration ratio of image information through a gating mechanism (sigmoid function) (e.g., the weight is 0 when there is no text image); finally, the fused features are mapped to 512 dimensions through linear transformation and output as a context vector, which is then concatenated with audio features and input into the translation model.

[0075] V. Cloud-Edge Collaborative Inference Engine: The edge inference model is based on a lightweight Transformer architecture: the encoder has 6 layers (4 attention heads per layer, 512-dimensional hidden layers), and the decoder has 6 layers (4 attention heads per layer). Relative position encoding is used instead of absolute position encoding to reduce the storage of position vectors. The input processing module maps the Mel-spectral features (40×100) to a 512-dimensional sequence through convolutional layers (3 layers, kernelsize=3), and fuses them with the context vector (linearly transformed to 512-dimensional) through cross-attention. The inference process uses beam search (beamsize=2) to accelerate decoding. For each generated token, its probability entropy value (H=-Σp_ilogp_i) is calculated. An entropy value <0.5 corresponds to high confidence (>0.8), 0.5-1.0 corresponds to medium confidence (0.5-0.8), and >1.0 corresponds to low confidence (<0.5).

[0076] The cloud-based collaborative processing module has a three-level response: the fast verification layer is deployed on the cloud mirror of the edge node (latency <10ms), which receives the medium confidence result, corrects it with a 2-layer Transformer decoder (focusing only on error-prone positions such as proper nouns), and returns the corrected translation; the deep processing layer is deployed in the central cloud, which calls the complete pre-trained model to perform end-to-end translation on the low confidence input, supporting complex scenarios such as mixed languages ​​and dialects; the result integration layer is located on the edge device, which compares the consistency between the local and cloud results. If the difference rate is <5%, it outputs directly; otherwise, it marks suspicious areas and prompts the user for confirmation.

[0077] The communication optimization module employs data compression and priority scheduling: medium-confidence data (local output + context) is transmitted after ZSTD compression (3-5 times compression ratio); raw data (audio + image) for low-confidence scenarios uses adaptive bitrate encoding (audio 16kbps, image resolution adjusted according to bandwidth). The transmission channel is configured with "low latency priority," using UDP protocol to transmit real-time data and TCP protocol to transmit acknowledgment information, ensuring that key frames can be retransmitted in case of data loss.

[0078] VI. Incremental Learning Data Closed-Loop System: The data acquisition rule engine defines trigger conditions: low-confidence results with an inference entropy value > 1.0 are automatically labeled; when the user clicks the "Correct" button on the interface, the difference between the original translation and the corrected content is recorded; input data that triggers deep cloud processing (such as mixed language speech) is automatically included in the acquisition scope. Each data point is appended with metadata (scene label, device type, network quality) for subsequent classification training.

[0079] The privacy protection module employs multi-layered processing: data anonymization removes identifying information such as device ID and timestamps, replacing them with random strings; differential privacy processing adds minor perturbations to the text translation results (such as replacing 1% of non-keywords) and adds Gaussian noise (mean 0, variance 0.01) to numerical features (such as confidence levels); encrypted transmission uses the TLS 1.3 protocol, data blocks are encrypted with AES-256, and the key is negotiated through asymmetric encryption (RSA-2048) to ensure that the transmission process is unbreakable.

[0080] The incremental learning scheduling system employs a priority queue: sorted by "scene coverage (prioritizing scarce scenes) + data quality (high-correction-rate data first) + time freshness (data from the last 7 days first)," selecting the top 10% of data daily for fine-tuning. The training process uses a regularization strategy (L2 regularization coefficient 0.0001, dropout rate 0.1) to prevent overfitting. After each training round, the model is validated on the test set; training terminates if the BLEU score improves by less than 0.5%. The adversarial validation module retains 20% of historical scene data to detect performance degradation of the new model on older scenes (maximum degradation allowed 1%), ensuring no catastrophic forgetting.

[0081] VII. Model Performance Evaluation and Rolling Update Module: The performance indicator monitoring system collects data in real time: translation accuracy (BLEU-4, TER, CHRF++), inference speed (average time per sentence), resource usage (peak memory, CPU utilization), and user feedback (satisfaction rating 1-5 stars, correction rate). Data is categorized and statistically analyzed by device model and scenario type, generating hourly-level monitoring dashboards.

[0082] Benchmark model identification mechanism: When a model slice has an average BLEU score that is 10% higher than other slices in the same scenario in 5000+ inferences, and the user satisfaction is >4.5 stars, and the standard is consistently met for 14 days, it is marked as the recommended benchmark model for that scenario and stored in the benchmark model library as a reference for the generation of subsequent slices.

[0083] The rolling update strategy employs a canary release approach: In the first phase (10% of devices), new slices are pushed to devices with different hardware types. If no anomalies are detected for 48 hours (crash rate <0.05%, performance degradation <5%), the second phase (50% of devices) begins. After 72 hours of stability in the second phase, a full rollout is implemented. The entire process supports one-click rollback; if a new slice triggers an anomaly threshold at any stage, the update is automatically paused and reverted to the old version. The update process is performed during device idle periods (such as overnight charging) to avoid impacting user experience.

[0084] 8. Unified Resource Orchestrator: The resource monitoring network consists of agents deployed on edge devices and in the cloud. Edge agents collect hardware status (CPU / memory / storage usage, temperature) and network status (upload / download bandwidth, latency) every 10 seconds; cloud agents collect cluster resources (GPU / CPU load, memory usage, task queue length) every 5 seconds. All data is aggregated to a central monitoring platform and stored in a time-series database (InfluxDB), retaining 30 days of historical data for trend analysis.

[0085] The load balancing algorithm employs dynamic threshold adjustment: when the edge device's memory utilization exceeds 85% or CPU load exceeds 90%, a "downgrade request" is sent to the dynamic model management engine, triggering a model slice switch with a 30% reduction in computational load; when the cloud GPU cluster load exceeds 85%, the first edge confidence threshold is automatically increased by 0.1 (e.g., from 0.8 to 0.9), and the second threshold is increased by 0.05 (e.g., from 0.5 to 0.55), increasing the proportion of local processing; when network latency exceeds 300ms, the edge device is forced to use offline available model slices to avoid dependence on the cloud.

[0086] The resource scheduling optimization module predicts resource demand for the next hour (such as during peak conference periods) based on a predictive model (LSTM network), pushes a lightweight model to edge devices in high-load areas 30 minutes in advance, and requests temporary capacity expansion from the cloud cluster to achieve proactive load balancing.

[0087] IX. Edge Model Personalized Federated Engine: The local fine-tuning module designs a lightweight training process for user data (such as specific accents and common expressions): freeze the low-level parameters of the model slice (the first 4 encoder layers) and only fine-tune the top-level parameters (the last 2 encoder layers + decoder); train for 3-5 rounds using mini-batch (batch size = 8) and low learning rate (0.00001), and evaluate on the local validation set (correct translations confirmed by the user) after each round. If the loss function decreases by <1%, fine-tuning stops; the generated parameter updates only include the parameter differences of the fine-tuned layers (about 5-10MB), reducing the amount of data uploaded.

[0088] The federated aggregation center is deployed in a secure cloud zone and uses the federated averaging (FedAvg) algorithm: it receives the encrypted parameter differences from each edge device (homomorphically encrypted with Paillier), decrypts them, and calculates the average value as a global update; to prevent malicious device attacks, a trimmed mean strategy is used to remove the top 10% and bottom 10% of outlier parameters before averaging; the aggregated parameter updates are applied to the pre-trained model library to generate a new version of the model.

[0089] Privacy enhancement mechanisms include: local data not leaving the device (only parameter differences are uploaded), encrypted parameter transmission during aggregation, differential privacy protection of aggregation results (adding noise to make the privacy budget ε < 1.0), and periodic replacement of encryption keys. At the end of each federated learning cycle (7 days), a privacy audit report is generated, recording the number of participating devices, the amount of data, and the implementation status of privacy protection measures.

[0090] Working principle:

[0091] The system workflow can be divided into two closed loops: "real-time translation" and "model evolution," as detailed below:

[0092] Real-time translation closed loop: Users initiate translation requests through edge devices (such as translation machines), generating multimodal input (speech, dialogue scene images, and historical text). The edge computing subsystem collects and preprocesses the data (audio to Mel spectrum, image compression), and simultaneously sends it to the multimodal context-aware engine. This engine identifies scene tags in real time (such as "asking for directions while traveling"), extracts semantic embeddings from nearly 10 rounds of dialogue, fuses image OCR text information, and generates a context vector.

[0093] The dynamic model management engine receives scene tags, edge device status (e.g., 2GB of available memory, 30% CPU utilization) and network quality (50ms latency, 2Mbps bandwidth), requests adapted dynamic model slices (e.g., for tourism scenes, 30MFLOPs computation) from the model slice factory through a decision algorithm, and distributes them to the edge devices.

[0094] The cloud-edge collaborative inference engine calls edge model slices, taking preprocessed audio features and context vectors as input, to perform local translation inference and calculate the output confidence score. If the confidence score is >0.8, the translation is output directly; if it is between 0.5 and 0.8, the local result and context are uploaded to the cloud, where a simplified model is used for verification and correction before the result is returned; if the confidence score is <0.5, the original data is uploaded to the cloud, the full model is called for processing, and the result is returned. The edge device integrates the final result and presents it to the user via voice or text.

[0095] Model evolution closed loop: During the translation process, the incremental learning data closed loop system automatically collects low-confidence results, user-corrected content, and cloud-processed data, which are then encrypted, compressed, and sent back to the cloud-based incremental learning center. The center categorizes data by scenario, fine-tunes the corresponding pre-trained models, and ensures no catastrophic forgetting through adversarial validation.

[0096] The edge model personalized federated engine uses local user data to perform lightweight fine-tuning on the currently running model slice, generating personalized parameter differences, which are then encrypted and uploaded to the cloud-based federated aggregation center. The center aggregates multi-user parameters and updates the global pre-trained model library.

[0097] The model slicing factory generates new dynamic model slices based on the updated global model. The dynamic model management engine, combined with model performance evaluation results (such as a 15% improvement in accuracy for the new version slices), gradually pushes the new slices to edge devices through a rolling update mechanism, completing one model iteration.

[0098] The unified resource orchestrator monitors the global resource status throughout the process and dynamically adjusts the model slice distribution strategy and cloud-edge collaboration threshold to ensure that the system continues to operate efficiently when device performance fluctuates or the network changes.

[0099] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and not restrictive.

[0100] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style of the specification is merely for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A real-time translation and recognition system under a cloud service framework, characterized in that: include: The cloud service subsystem includes a large-scale pre-trained model library, a model slicing factory, and an incremental learning center; The edge computing subsystem, deployed in user-side devices, is used to run lightweight task-specific models and collect multimodal data; The dynamic model management engine connects the cloud service subsystem and the edge computing subsystem, and is used to generate and distribute adapted dynamic model slices from the model slice factory to the designated edge computing subsystem based on the received scene context information and device status information. A multimodal context-aware engine is used to extract and fuse semantic information from audio streams, image streams, and dialogue history in real time to generate structured context vectors. The cloud-edge collaborative inference engine takes the dynamic model slices built into the edge computing subsystem as its core and receives the structured context vector as conditional input to perform local preliminary translation; for complex tasks that exceed the local capabilities, it performs collaborative inference with the pre-trained model in the cloud service subsystem. The incremental learning data closed-loop system is used to collect uncertainty data and user feedback data generated by the edge computing subsystem during the inference process, and send them back to the incremental learning center to optimize the ultra-large-scale pre-trained model library.

2. The real-time translation and recognition system under a cloud service framework according to claim 1, characterized in that: The decision-making criteria of the dynamic model management engine include: Scene tags identified in real time by the multimodal context-aware engine, the scene tags including professional fields and dialogue formats; The edge computing subsystem reports device performance parameters in real time, including available memory and CPU / GPU computing power; Current real-time quality assessment metrics for the network include bandwidth, latency, and jitter.

3. The real-time translation and recognition system under a cloud service framework according to claim 2, characterized in that: The model slicing factory is configured as follows: Receive a request from the dynamic model management engine, the request containing target scene labels and performance constraints; From the aforementioned large-scale pre-trained model library, a dedicated sub-model that meets performance constraints and has high accuracy for the target scene is generated through structured pruning, knowledge distillation, or feature extraction. The sub-model and its configuration file are packaged into dynamic model slices and distributed to the target edge computing subsystem.

4. The real-time translation and recognition system under a cloud service framework according to claim 1, characterized in that: The multimodal context-aware engine includes: The scene recognition unit is used to identify the scene in which the dialogue occurs in real time based on visual elements in the current image stream and keywords in the audio stream. The dialogue history management unit is used to maintain a fixed-length time window and store the text history of recent dialogues and their semantic embeddings. The context vector synthesis unit is used to fuse and encode scene labels, current dialogue history semantic embeddings, and key text information identified from images to generate a unified context vector.

5. The real-time translation and recognition system under a cloud service framework according to claim 4, characterized in that: The cloud-edge collaborative inference engine is an inference model deployed within the edge computing subsystem. The inputs received include: a preprocessed audio feature sequence and a context vector generated by the context-aware engine. This context vector serves as the conditional input of the model and participates in the calculation of the attention mechanism at each step, thereby guiding the model to generate translation results that are highly relevant to the current context.

6. The real-time translation and recognition system under a cloud service framework according to claim 1, characterized in that: The incremental learning data closed-loop system includes: The data acquisition unit is used to anonymously collect low-confidence inference results, results manually corrected by users, and raw data that triggers cloud-edge collaboration at the edge. The data encryption and compression unit is used to perform privacy protection processing on the collected data before transmission back; The incremental learning scheduler, located in the incremental learning center, is used to schedule the collected datasets to fine-tune specific models in the pre-trained model library, while ensuring that catastrophic forgetting does not occur after model updates through adversarial validation.

7. The real-time translation and recognition system under a cloud service framework according to claim 6, characterized in that: The system also includes: The model performance evaluation and rolling update module is used for: Monitor the performance metrics of dynamic model slices running on each edge computing subsystem; When the average confidence or user satisfaction of a certain model slice is consistently higher than the threshold, it is triggered to be set as the recommendation benchmark model in that scenario. Once the new incremental learning version is verified to be effective, the model slicing factory is notified to generate new slices based on the new version of the model, and these slices are gradually pushed to the edge computing subsystem for rolling updates.

8. The real-time translation and recognition system under a cloud service framework according to claim 1, characterized in that: The collaborative workflow of the cloud-edge collaborative inference engine includes: The edge computing subsystem first uses local model slices for inference and calculates the output confidence in real time; When the confidence level is higher than the first threshold, the local result is output directly. When the confidence level is between the first and second thresholds, the local output and context vector are uploaded to the cloud for quick verification and refinement. When the confidence level is below the second threshold or a completely unknown modal input is detected, the raw data is uploaded and a request is made to the cloud for full depth processing.

9. The real-time translation and recognition system under a cloud service framework according to claim 1, characterized in that: The system also includes a unified resource orchestrator for: Monitor the status of all online edge computing subsystems and the load of the cloud computing resource pool; When the performance of a certain edge computing subsystem device is insufficient, the instruction dynamic model management engine distributes a model slice with a smaller computational load to it. When the cloud load is too high, the instruction dynamic model management engine appropriately increases the confidence threshold of the edge computing subsystem, leaving more computing tasks on the edge side to achieve global load balancing.

10. A real-time translation and recognition system under a cloud service framework according to any one of claims 1-9, characterized in that: The system also includes an edge model personalized federated engine: The engine runs locally on the edge computing subsystem and receives local data from the incremental learning data closed-loop system. The engine generates local model update parameters by fine-tuning the currently running dynamic model slices using the local data on the local device. The cloud service subsystem includes a federated aggregation center for encrypted collection of local model update parameters from multiple edge computing subsystems. These updates are aggregated through a federated averaging algorithm to generate a global, performance-enhanced, and privacy-protecting next-generation pre-trained model. The model slicing factory generates new dynamic model slices based on the next-generation pre-trained model and distributes them to the edge computing subsystem through the dynamic model management engine, completing a federated learning loop.

Images