Speech translation model training method, speech translation method and device based on cross-modal attention, global memory and dynamic convolution

Abstract

The invention discloses a cross-modal attention, global memory and dynamic convolution-based speech translation model training method and device, and a speech translation method and device, and relates to the technical field of speech processing and machine translation. A speech translation model is designed to comprise a speech encoder, a text embedding layer, a cross-modal attention adapter, a large language model decoder, a global memory network, a dynamic convolution decoder and an output layer. The cross-modal attention adapter projects audio features and performs multi-head cross attention fusion with text embedding; the global memory network updates and enhances historical memory on the basis of a gating mechanism and a Transform Encoder; and the dynamic convolution decoder performs multi-scale convolution extraction on the decoded hidden representation and fuses with the memory, so that the translation quality is improved. According to the method, deep fusion of voice and text, continuous memory with contextual coherence and high-quality translation generation can be realized, the end-to-end voice translation performance is remarkably improved, and the actual requirements of real-time and high-quality end-to-end voice translation in a complex scene are met.

Description

Technical Field

[0001] This application relates to the fields of speech processing and machine translation technology, and in particular to a speech translation model training method, speech translation method and apparatus based on cross-modal attention, global memory and dynamic convolution. Background Technology

[0002] With the development of neural networks and large-scale pre-trained models, end-to-end speech translation (STT) has become a research hotspot. A typical approach is to first use an automatic speech recognition (ASR) module to convert speech into intermediate text, and then use a machine translation (MT) module to generate the target language text; or to directly train an end-to-end model. Although existing solutions have achieved some success in small-scale scenarios, they still cannot meet the practical needs of real-time, high-quality speech translation in complex scenarios (such as long speeches with multiple accents and multi-turn dialogues). Summary of the Invention

[0003] The purpose of this application is to provide a speech translation model training method, speech translation method and device based on cross-modal attention, global memory and dynamic convolution, which can meet the actual needs of real-time, high-quality end-to-end speech translation in complex scenarios.

[0004] To achieve the above objectives, this application provides the following solution:

[0005] Firstly, this application provides a method for training a speech translation model based on cross-modal attention, global memory, and dynamic convolution, including:

[0006] Obtain a training dataset, wherein the training dataset contains multiple parallel samples, each sample including source language audio data, corresponding source language text transcription data, and corresponding target language text translation data;

[0007] The samples in the training dataset are preprocessed to obtain source language audio feature sequences and text token sequences, wherein the text token sequences include source language text token sequences and target language text token sequences.

[0008] Construct a speech translation model, wherein the speech translation model includes:

[0009] A speech encoder is used to encode the source language audio feature sequence and output an audio feature representation;

[0010] A text embedding layer is used to map the text token sequence into a text embedding representation;

[0011] A cross-modal attention adapter, with its first input connected to the output of the speech encoder and its second input connected to the output of the text embedding layer, is used to fuse the audio feature representation and the text embedding representation based on a multi-head cross-attention mechanism, and output the fused multimodal feature representation;

[0012] The large language model decoder, whose input is connected to the output of the cross-modal attention adapter, is used to generate a sequence of hidden states of the target language step by step based on the multimodal feature representation;

[0013] A global memory network, whose input is connected to the output of the large language model decoder, is used to update the global memory matrix based on the hidden state sequence using a gating mechanism, and outputs the updated global memory matrix.

[0014] A dynamic convolutional decoder, whose first input is connected to the output of the large language model decoder and whose second input is connected to the output of the global memory network, is used to perform multi-scale convolution on the hidden state sequence to extract local features, and to fuse the local features with the updated global memory matrix based on an attention mechanism to output an optimized target language hidden representation.

[0015] The output layer, whose input is connected to the output of the dynamic convolutional decoder, is used to output the target language translated text based on the optimized target language hidden representation.

[0016] The speech translation model is trained based on the source language audio feature sequence and the text token sequence until the convergence condition is met.

[0017] Secondly, this application provides a speech translation method based on cross-modal attention, global memory, and dynamic convolution, including:

[0018] Acquire the audio data of the language to be processed;

[0019] Based on the audio data of the language to be processed and the initial text prompt, the target language translation text is obtained using a trained speech translation model. The speech translation model is trained using the speech translation model training method based on cross-modal attention, global memory and dynamic convolution as described in the first aspect. The initial text prompt is a preset text instruction or a translation task text prompt provided by the user.

[0020] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the speech translation method based on cross-modal attention, global memory, and dynamic convolution as described in the first aspect, or the speech translation method based on cross-modal attention, global memory, and dynamic convolution as described in the second aspect.

[0021] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech translation method based on cross-modal attention, global memory, and dynamic convolution as described in the first aspect, or the speech translation method based on cross-modal attention, global memory, and dynamic convolution as described in the second aspect.

[0022] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the speech translation method based on cross-modal attention, global memory, and dynamic convolution as described in the first aspect, or the speech translation method based on cross-modal attention, global memory, and dynamic convolution as described in the second aspect.

[0023] According to the specific embodiments provided in this application, this application has the following technical effects:

[0024] This application provides a speech translation model training method, speech translation method, and apparatus based on cross-modal attention, global memory, and dynamic convolution. The speech translation model is designed to include a speech encoder, a text embedding layer, a cross-modal attention adapter, a large language model decoder, a global memory network, a dynamic convolutional decoder, and an output layer. The cross-modal attention adapter projects audio features and fuses them with multi-head cross-attention of the text embedding, achieving deep semantic alignment between speech and text and addressing noise and accent robustness issues. The global memory network updates and enhances historical memory based on gating mechanisms and a Transformer Encoder. The enhancement mechanism dynamically maintains long-range contextual information, ensuring the coherence of translation for long speech segments and multi-turn dialogues. The dynamic convolutional decoder extracts the decoded hidden representation through multi-scale convolution and fuses it with memory, simultaneously capturing local details and global semantics, significantly improving the fluency and accuracy of the translation. Therefore, this invention can meet the application requirements of complex scenarios requiring real-time interactive speech translation and high accuracy. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of the speech translation model in Embodiment 1 of this application;

[0027] Figure 2 This is a schematic diagram of the cross-modal attention adapter in Embodiment 1 of this application;

[0028] Figure 3 This is a schematic diagram of the global memory network structure in Embodiment 1 of this application;

[0029] Figure 4 This is a schematic diagram of the structure of the dynamic convolutional decoder in Embodiment 1 of this application;

[0030] Figure 5 This is a schematic diagram of the training and inference process of the speech translation model in Embodiment 1 of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] Research has revealed the following shortcomings in existing end-to-end speech translation solutions:

[0033] 1. Insufficient efficiency of cross-modal fusion: The fusion of speech features and text generation often relies on concatenation or simple attention, which makes it difficult to fully align audio and target text representations.

[0034] 2. Lack of contextual memory – In long speech or dialogue scenarios, the model has difficulty retaining historical information, resulting in incoherent translation or loss of context.

[0035] 3. Limited decoding quality – When generating long sequences or complex sentences, standard self-attention or simple convolution mechanisms are insufficient in capturing local details, which can easily lead to fluency or accuracy issues.

[0036] 4. High model fine-tuning and deployment costs – Direct end-to-end fine-tuning of large-scale models is expensive and difficult to apply in resource-constrained scenarios.

[0037] The aforementioned defects ultimately lead to problems such as low accuracy, poor contextual coherence, and insufficient fluency in speech translation, making it difficult to be practical in complex scenarios (such as long speech with multiple accents and multi-turn dialogues) and on resource-constrained devices (such as edge devices), and thus failing to meet the actual needs of real-time, high-quality speech translation.

[0038] Therefore, there is an urgent need for an end-to-end speech translation method that can efficiently integrate audio and generation, possess continuous memory capabilities, improve decoding and generation quality, and take into account fine-tuning efficiency. This would meet the requirements of high accuracy, strong contextual coherence, and high translation fluency in complex scenarios such as multiple accents, long speech, and multi-turn dialogues, while also being adaptable to resource-constrained device deployments, thus achieving the goal of practical application of real-time, high-quality end-to-end speech translation.

[0039] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] Example 1

[0041] This embodiment provides a method for training a speech translation model based on cross-modal attention, global memory, and dynamic convolution, including:

[0042] S1: Obtain the training dataset, wherein the training dataset contains multiple parallel samples, each sample including source language audio data, corresponding source language text transcription data, and corresponding target language text translation data.

[0043] S2: Preprocess the samples in the training dataset to obtain source language audio feature sequences and text token sequences, wherein the text token sequences include source language text token sequences and target language text token sequences.

[0044] S2 specifically includes:

[0045] S21: Resample, frame, and extract features from the source language audio data to obtain the source language audio feature sequence;

[0046] S22: Perform word segmentation and sub-wording on the source language text transcription data and the target language text translation data to obtain the source language text token sequence and the target language text token sequence.

[0047] S3: Construct a speech translation model, where, for example Figure 1 As shown, the speech translation model includes:

[0048] A speech encoder is used to encode the source language audio feature sequence and output an audio feature representation;

[0049] A text embedding layer is used to map the text token sequence into a text embedding representation;

[0050] A cross-modal attention adapter, with its first input connected to the output of the speech encoder and its second input connected to the output of the text embedding layer, is used to fuse the audio feature representation and the text embedding representation based on a multi-head cross-attention mechanism, and output the fused multimodal feature representation;

[0051] The Large Language Model (LLM) decoder, whose input is connected to the output of the cross-modal attention adapter, is used to generate a sequence of hidden states of the target language step by step based on the multimodal feature representation.

[0052] A global memory network, whose input is connected to the output of the large language model decoder, is used to update the global memory matrix based on the hidden state sequence using a gating mechanism, and outputs the updated global memory matrix.

[0053] A dynamic convolutional decoder, whose first input is connected to the output of the large language model decoder and whose second input is connected to the output of the global memory network, is used to perform multi-scale convolution on the hidden state sequence to extract local features, and to fuse the local features with the updated global memory matrix based on an attention mechanism to output an optimized target language hidden representation.

[0054] The output layer, whose input is connected to the output of the dynamic convolutional decoder, is used to output the target language translated text based on the optimized target language hidden representation.

[0055] S4: Train the speech translation model based on the source language audio feature sequence and the text token sequence until the convergence condition is met.

[0056] S4 specifically includes:

[0057] S41: Based on the parameter update strategy, the speech translation model is fine-tuned and trained using the source language audio feature sequence and the text token sequence until the convergence condition is met. The parameter update strategy includes: injecting LoRA low-rank adaptation parameters into the key attention sub-modules in the large language model decoder, updating only the injected LoRA parameters and all parameters of the custom module, and keeping the remaining original parameters of the large language model decoder frozen. The custom module includes the cross-modal attention adapter, the global memory network, and the dynamic convolutional decoder.

[0058] like Figure 2 As shown, the cross-modal attention adapter specifically includes:

[0059] A linear projection layer, whose input is connected to the output of the speech encoder, is used to project the dimension of the audio feature representation onto the dimension of the text feature representation to obtain the projected audio feature representation.

[0060] The cross-attention module has its first input connected to the output of the linear projection layer and its second input connected to the output of the text embedding layer. This attention layer is used to perform multi-head cross-attention calculation with the text embedding representation as the query and the projected audio feature representation as the key and value, and outputs a cross-modal attention feature representation.

[0061] The residual connection and layer normalization (LayerNorm) layer has its first input connected to the output of the multi-head cross-attention layer and its second input connected to the output of the text embedding layer. It is used to add the cross-modal attention feature representation and the text embedding representation, and perform layer normalization on the addition result to output the fused multimodal feature representation.

[0062] like Figure 3 As shown, the global memory network specifically includes:

[0063] The sequence compression module, whose input is connected to the output of the large language model decoder, is used to compress and project the hidden state of the current time step into a single vector representation and output the current context feature vector. The sequence compression module compresses the hidden state through mean pooling or linear projection.

[0064] The gated update module (i.e., the GRU network layer) has its input connected to the output of the sequence compression module. It is used to control the fusion ratio of historical information and current information through a gating mechanism based on the current context feature vector and the global memory matrix of the previous time step, and output the updated initial memory matrix. The gating mechanism of the gated update module includes a reset gate and an update gate.

[0065] The context enhancement module, whose input is connected to the output of the gated update module, is used to perform deep encoding on the updated initial memory matrix through a self-attention mechanism and a feedforward neural network to enhance its semantic representation ability and capture the dependencies between memory slots, and output the updated global memory matrix. The context enhancement module is composed of multiple layers of Transformer encoder layers stacked together, each layer containing a multi-head self-attention sub-layer and a feedforward neural network sub-layer.

[0066] like Figure 4 As shown, the dynamic convolutional decoder specifically includes:

[0067] A multi-scale convolutional layer, whose input is connected to the output of the large language model decoder, is used to perform multi-scale convolutional features based on the hidden state sequence to obtain multi-scale features. The multi-scale convolutional layer includes multiple parallel sets of one-dimensional convolutional layers, each set of one-dimensional convolutional layers using convolutional kernels of different sizes to capture local context-dependent features of different granularities in the hidden state sequence.

[0068] The feature fusion layer, whose input is connected to the output of the multi-scale convolutional layer, is used to perform fusion operations on the multi-scale features to generate a unified multi-scale fused feature representation.

[0069] The memory attention fusion layer has its first input connected to the output of the feature fusion layer and its second input connected to the output of the global memory network. It is used to use the multi-scale fusion feature representation as a query and the global memory matrix as a key and value to perform multi-head attention calculation, realize the interaction and fusion of the current local context and global historical context information, and output attention-enhanced features.

[0070] A linear output layer, whose input is connected to the output of the memory-attention fusion layer, is used to linearly transform the attention-enhancing features and project them onto a preset hidden dimension, outputting an optimized target language hidden representation.

[0071] The following is combined Figure 5 The training and inference process of the speech translation model in this embodiment will be described in detail.

[0072] (1) Obtain the dataset and process it.

[0073] This includes data sources and acquisition, data preprocessing, vocabulary generation and encoding.

[0074] Data Sources and Acquisition: Training data was obtained from the publicly available speech-text translation parallel corpus (CoVoST 2 Chinese-English dataset). The training data includes source language speech (Chinese audio) and text-translation pairs. Each text-translation pair includes source language text (Chinese transcribed text) and target language text (corresponding English translated text), ensuring data coverage of diverse accents and scenarios. Speech data is in .wav format, and text data is in the corresponding sentence alignment file (TSV format).

[0075] Data preprocessing:

[0076] 1) Audio preprocessing

[0077] Uniform sampling rate: All audio files are resampled to 16kHz to ensure consistent input.

[0078] Denoising: Algorithms such as filtering and spectral subtraction are used to reduce background noise and improve the signal-to-noise ratio.

[0079] Segmentation: For long audio clips, segment them into segments of appropriate length (within 5 seconds) to facilitate model training.

[0080] Format conversion: Convert to a format suitable for the voice encoder input.

[0081] Use the original waveform directly: Employ a pre-trained model (Whisper) and use the original waveform directly as input.

[0082] 2) Text preprocessing: unify capitalization and remove special characters.

[0083] Target text (English) processing: Clean the text, remove abnormal symbols, duplicate spaces, and unify capitalization; filter abnormal data, remove sentences that are too long (more than 250 words) and too short (less than 3 words); standardize the processing, unify and standardize special formats such as numbers and dates.

[0084] Source text (Chinese transcription) processing: standardization, unifying simplified and traditional characters, removing redundant spaces and punctuation errors; alignment processing to ensure accurate Chinese transcription and English translation for each audio segment.

[0085] (2) Cross-modal attention adapter design: Project audio coding features into the text space and align and fuse them with text cues through multi-head cross attention.

[0086] The design steps of a cross-modal attention adapter include: input feature preparation, linear mapping of audio features, cross-modal multi-head attention computation, residual connections and layer normalization, and adapter module integration.

[0087] Input feature preparation: Obtain high-dimensional audio features of dimension D from the speech encoder. a Simultaneously prepare the input embedding for the text embedding layer, with dimension D. t Since the two dimensions are inconsistent, mapping is required to align them within the same vector space.

[0088] Data Alignment: Establish a one-to-one correspondence between audio and text translation pairs, and construct an alignment dataset suitable for end-to-end training. Audio is normalized by sampling rate, segmented (fixed duration 1-5 seconds), and converted into a feature matrix of shape (T_a, F), where T_a is the number of time frames and F is the frequency dimension. Text is segmented and converted into a token sequence of shape (T_t, ), where T_t is the sentence length (number of tokens). Establish the audio-text alignment relationship to form training sample pairs: (audio features, source language token sequence, target language token sequence).

[0089] Audio Feature Linear Mapping: Design a linear mapping layer to map audio features from dimension D. a Mapping to text feature dimension Dt (Consistent with the hidden dimension of LLM embedding), enabling the fusion of cross-modal semantic spaces. This mapping layer is implemented through a fully connected linear layer.

[0090]

[0091] Where A represents the audio input feature. For the mapped audio features, W a Let b be the weight matrix. a For bias.

[0092] Cross-modal multi-head attention computation: Based on the multi-head attention mechanism, text embeddings are used as queries, and mapped audio features are used as keys and values ​​to compute cross-modal attention representations. Specifically:

[0093]

[0094] In this system, the query is embedded in text, and the key-value pairs are mapped audio features, enabling the perception and fusion of audio semantic information from the text.

[0095] Residual connections and layer normalization: To alleviate the vanishing gradient problem and improve training stability, residual connections are designed to add the cross-modal attention output to the original text embedding, followed by layer normalization.

[0096] H=LayerNorm(AttentionOutput+TextEmbedding)

[0097] This step ensures that the model can effectively integrate audio and text information while maintaining training stability.

[0098] Adapter module integration: The cross-modal attention adapter module mentioned above is integrated into the speech translation system as an important bridge connecting the speech encoder and the text embedding layer, realizing deep fusion and complementarity of the two modal information, and improving the quality and generalization ability of speech translation.

[0099] (3) Global memory network design: Based on the combination of gating mechanism (GRU) and Transformer Encoder, the historical memory update of intermediate decoding representation and global context modeling are realized, supporting long speech streams and multi-turn continuous memory of dialogue.

[0100] The design steps of a global memory network include: initializing the global memory matrix, designing the gating mechanism, the memory update mechanism, enhancing the multi-layer Transformer encoding, memory initialization and persistence, and integration and application.

[0101] Initialize the global memory matrix: Design a memory matrix of fixed size. Where B is the batch size, N is the memory dimension, and D is the feature dimension. This memory matrix is ​​used for long-term storage and updating of model context information, and is initially initialized as a zero matrix.

[0102] Gating mechanism design: Construct a gating network, the input of which is the concatenated vector of the current time-stack model output features and global memory features:

[0103] g=σ(W g [h curr M mean ]+b g )

[0104] in, The mean of the features output by the current model. Let be the mean of the features in each dimension of the memory matrix, σ be the sigmoid activation function, and g ∈ [0,1]. D This represents the gating coefficient for memory updates.

[0105] Memory update mechanism: A gated recurrent unit (GRU Cell) is used to update the memory. The current output feature adjusted by the gate value and the original memory feature are used as input to achieve dynamic fusion and evolution of memory information.

[0106] M'=GRUCell(M,g⊙h curr )

[0107] Here, ⊙ represents element-wise multiplication, and the updated memory maintains a consistent shape, reflecting the incremental impact of the current context on the global memory.

[0108] Multi-layer Transformer Encoding Enhancement: A multi-layer Transformer encoder layer is designed to further encode and enhance the updated memory matrix, capturing complex temporal dependencies and semantic interactions.

[0109] M” = TransformerEncoder(M')

[0110] This structure enhances the expressive power of memory features and improves the long-term context capture effect.

[0111] Memory initialization and persistence: During the inference process, a memory initialization method is designed, and the memory state is continuously passed between batches to achieve contextual continuity across time steps and global memory utilization.

[0112] This global memory network receives the hidden representation output by the large language model decoder, performs gating updates on historical states, and enhances the representation using a Transformer Encoder to output a memory representation. It can save and update cross-segment or cross-turn context during speech translation, achieving long-term dependency preservation and information accumulation.

[0113] Integrated Application: This global memory network is embedded into the speech translation model to assist the generation process by utilizing historical context and global information, thereby improving the fluency and consistency of translation.

[0114] Global memory network data processing procedure:

[0115] The input is the hidden output of the current large language model decoder (i.e., the hidden state output by the current large language model decoder), with the shape of... It is the context representation generated by the decoder at the current time step, used for fusion with historical memory; the previous memory state, with the shape of... Save historical translation context information, including semantic features accumulated from previous time steps.

[0116] The hidden layer output of the large language model decoder is compressed and projected into a single-step representation to obtain... Then, GRU updates and generates new memory states based on the last memory state of h_in and Mem_prev. This process controls information flow through gating mechanisms (reset gate, update gate) to effectively integrate historical memory and current information. Finally, a Transformer encoder processes the memory state, enhancing semantic expression in memory and extracting contextual information through multi-layered self-attention and feedforward networks. This makes memory not just simple accumulation, but also capable of semantic understanding and reasoning.

[0117] (4) Dynamic Convolutional Decoder: Multi-scale convolutional feature extraction is performed on the LLM output, and the feature is fused with the memory representation through attention to generate the final hidden representation, thereby improving the local semantic capture capability and global consistency of the translation generation. In the generation process, in addition to relying on global attention, the ability to capture neighboring tokens and local patterns is enhanced to improve the generation quality.

[0118] The design steps of a dynamic convolutional decoder include: design of multi-scale dynamic convolutional modules, nonlinear activation and regularization, multi-scale feature fusion, memory attention fusion mechanism, feature concatenation and linear transformation, and output features for translation generation.

[0119] Multi-scale dynamic convolution module design: This embodiment designs multiple sets of one-dimensional convolutional layers, using different kernel sizes (3, 5, 7) to capture contextual dependencies of different granularities, thus achieving multi-scale convolution operations. The input feature dimension is... (Batch size B, sequence length S, feature dimension D), by performing multi-scale convolution on the input features along the time dimension, contextual dependency information of different ranges is captured:

[0120] {C k (X)} k∈{3,5,7} =Conv1d k (X)

[0121] The kernel size k varies, and padding is used in the convolution to maintain a consistent sequence length.

[0122] Nonlinear activation and regularization: Each convolution output is activated by the GELU function and regularized by Dropout to enhance the model's nonlinear expressiveness and prevent overfitting.

[0123]

[0124] Multi-scale feature fusion: The output features of different convolutional kernels are fused using mean pooling to generate a unified multi-scale convolutional feature representation.

[0125]

[0126] Memory-Attention Fusion Mechanism: A multi-head attention mechanism is designed, using the fused convolutional features as the query and the output of the global memory network as the key and value, to achieve effective interaction between contextual memory information and the current convolutional features.

[0127] A = MultiHeadAttention (Q = C) fused (K=M, V=M)

[0128] Feature concatenation and linear transformation: The results of convolutional features and attention fusion are concatenated along the feature dimension and input into a linear projection layer for dimension mapping and fusion to generate the final decoded features.

[0129] Z = W o [C fused ;A]+b o

[0130] Output features are used for translation generation: the decoded features are input into the subsequent linear output layer and mapped to the vocabulary dimension for use by downstream translation generation tasks.

[0131] (5) Model Integration

[0132] The speech encoder, text embedding layer, cross-modal attention adapter, large language model decoder, global memory network, dynamic convolutional decoder, and output layer are integrated to form a speech translation model: speech is processed by the speech encoder to obtain speech vectors, and text is processed by the text embedding layer to obtain text embeddings. Then, the speech vectors and text embeddings are input into the cross-modal attention adapter to obtain the input of the LLM. The LLM layer outputs the target language text vector. The target language text vector is input into the global memory network to store the current memory and obtain the global memory output. The global memory data and the target language text vector are jointly input into the dynamic convolutional decoder to obtain the semantically enhanced target language text vector. Then, this vector is input into the output layer to obtain the target text output.

[0133] (6) Efficient fine-tuning strategy: Combine LoRA and other parameter efficient fine-tuning methods to reduce the number of trainable parameters, accelerate training and reduce resource requirements.

[0134] This embodiment proposes a modular and hierarchical fine-grained parameter update strategy, combined with LoRA (Low-Rank Adaptation) technology, which significantly reduces training costs and memory consumption while ensuring model performance, and has good transferability and resource adaptability.

[0135] Fine-tuning target module selection: Low-rank parameter insertion is performed only on key attention submodules (q_proj, k_proj, v_proj) in the LLM. Custom modules (such as cross-modal adapters, global memory networks, and dynamic convolutional decoders) are fully updated.

[0136] Fine-tuning the structure design: LoRA is configured as follows: r = 8: low-rank matrix dimension, lora_alpha = 32: scaling factor, lora_dropout = 0.05: to prevent overfitting.

[0137] Optimizer strategy: Use the AdamW optimizer, combined with group learning rate. Use a lower learning rate of 1e-5 for LLM (LoRA part) and a relatively higher learning rate of 2e-4 for custom modules. Set the weight decay to 0.01 to prevent overfitting.

[0138] The learning rate scheduler uses the inverse_sqrt strategy: the initial learning rate is 2e-5, and the number of warmup steps is set to 4000.

[0139] The steps of an efficient fine-tuning strategy include: multi-stage fine-tuning.

[0140] The fine-tuning process is divided into multiple stages, with each stage involving the fine-tuning of only one or a portion of the modules:

[0141] Phase 1: Fine-tuning of the cross-modal adapter and dynamic convolutional decoder

[0142] Unfreeze the cross-modal attention adapter and dynamic convolutional decoder, and adjust their parameters to achieve better fusion of audio and text features and optimize decoding capabilities.

[0143] Phase 2: Fine-tuning of the global memory network

[0144] Freeze all parameters except the global memory network, and only unfreeze the global memory network to enhance long-term memory representation and context capture capabilities.

[0145] Phase 3: Fine-tuning of the LLM decoder and output layer

[0146] Freeze the above modules, unfreeze the large language model decoder and output layer to improve the language quality and accuracy of the generated translation.

[0147] (7) End-to-end training and inference: Design corresponding data preprocessing, training scheduling, memory initialization and persistence mechanisms to realize a deployable and scalable speech translation framework.

[0148] The end-to-end training steps include: learning rate stratification, optimizer selection, and learning rate scheduling strategy.

[0149] Layered learning rate settings: For the low-rank adapter module (LLM-LoRA) in the large language model decoder, a lower learning rate, such as 1e-5 to 2e-5, is used to ensure stable fine-tuning and prevent the destruction of pre-trained knowledge. For custom modules (such as cross-modal adapters, global memory networks, dynamic convolutional decoders, etc.), a relatively higher learning rate, such as 1e-4 to 5e-4, is used to accelerate module convergence. Independent learning rate control for different modules is achieved through parameter grouping.

[0150] Optimizer selection: The AdamW optimizer is used, combined with weight decay to suppress overfitting and improve generalization ability. The weight decay coefficient is set between 0.01 and 0.001. Weight decay is typically not used for bias terms and normalization layer parameters such as LayerNorm.

[0151] Learning rate scheduling strategy: The learning rate decays inversely with the square root of the training steps. A pre-set warm-up phase is used to linearly increase the learning rate to alleviate instability in the early stages of training. A decay phase begins in the middle of training to prevent excessively large step sizes from causing model oscillations.

[0152] Inference process: Input audio segments are passed in step by step; global memory is retained between segments to prevent context loss. The output token is decoded using beam search to generate the final translation.

[0153] The speech translation model trained using this embodiment can achieve:

[0154] 1. Precise Cross-Modal Fusion: Through linear projection and multi-head cross-attention, a deep mapping between audio features and text representations is achieved, improving the semantic alignment between speech and text. Compared to traditional pipelined methods, this system performs better in scenarios with rare words, proper nouns, and diverse accents, and has stronger robustness to noise and speech rate variations.

[0155] 2. Continuous and reliable global context memory: Based on a gating mechanism and a global memory network of Transformer encoder, it supports multi-turn dialogue scenarios, retains historical translation information, avoids context loss, and improves the consistency and integrity of long speech or cross-segment translation.

[0156] 3. Improve local generation and global consistency: The dynamic convolutional decoder captures details between local tokens and combines them with global memory attention to enhance intra-sentence coherence and overall semantic consistency, effectively reducing repetition, omissions and unnatural expressions, and improving the fluency and readability of the translation.

[0157] 4. Highly efficient fine-tuning: Employing LoRA lightweight fine-tuning, only key sub-modules are adjusted, significantly reducing training costs and memory usage. It is suitable for resource-constrained environments and rapid iteration, supports incremental training and online fine-tuning, and enhances domain adaptability.

[0158] 5. Low-latency streaming translation: Supports audio segmented input and continuous memory updates, outputting translations without waiting for the entire audio segment to finish, meeting real-time translation needs. Dynamic convolution and cross-modal attention computations are lightweight, ensuring fast response and supporting a flexible balance between latency and quality.

[0159] Example 2

[0160] This embodiment provides a video tag processing device including:

[0161] Step 1: Obtain the audio data of the language to be processed;

[0162] Step 2: Based on the audio data of the language to be processed and the initial text prompt, the target language translation text is obtained using the trained speech translation model. The speech translation model is trained using the speech translation model training method based on cross-modal attention, global memory and dynamic convolution described in Example 1. The initial text prompt is a preset text instruction or a translation task text prompt provided by the user.

[0163] This embodiment designs a speech translation model comprising a speech encoder, a text embedding layer, a cross-modal attention adapter, a large language model, a global memory network, a dynamic convolutional decoder, and an output layer. The cross-modal attention adapter projects audio features and fuses them with text embeddings through multi-head cross-attention; the global memory network updates and enhances historical memory based on gating mechanisms and the TransformerEncoder; the dynamic convolutional decoder extracts the decoded hidden representations through multi-scale convolutions and fuses them with memory to improve translation quality; and a LoRA fine-tuning strategy is combined to achieve efficient training and deployment. Through this design, deep fusion of speech and text, context-coherent continuous memory, and high-quality translation generation can be achieved, significantly improving end-to-end speech translation performance and reducing resource consumption.

[0164] Example 3

[0165] This embodiment provides a computer device, which can be a server or a terminal. The computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores training data for speech translation models or speech translation data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the speech translation model training method based on cross-modal attention, global memory, and dynamic convolution described in Embodiment 1, or the speech translation method based on cross-modal attention, global memory, and dynamic convolution described in Embodiment 2.

[0166] Example 4

[0167] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the speech translation model training method based on cross-modal attention, global memory, and dynamic convolution as described in Embodiment 1, or the speech translation method based on cross-modal attention, global memory, and dynamic convolution as described in Embodiment 2.

[0168] Example 5

[0169] This embodiment provides a computer program product, including a computer program that, when executed by a processor, implements the speech translation model training method based on cross-modal attention, global memory, and dynamic convolution as described in Embodiment 1, or the speech translation method based on cross-modal attention, global memory, and dynamic convolution as described in Embodiment 2.

[0170] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0171] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0172] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchain. The processors involved in the embodiments provided in this application may be, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc.

[0173] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0174] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for training a speech translation model based on cross-modal attention, global memory, and dynamic convolution, characterized in that, The training method for the speech translation model based on cross-modal attention, global memory, and dynamic convolution includes: Obtain a training dataset, wherein the training dataset contains multiple parallel samples, each sample including source language audio data, corresponding source language text transcription data, and corresponding target language text translation data; The samples in the training dataset are preprocessed to obtain source language audio feature sequences and text token sequences, wherein the text token sequences include source language text token sequences and target language text token sequences. Construct a speech translation model, wherein the speech translation model includes: A speech encoder is used to encode the source language audio feature sequence and output an audio feature representation; A text embedding layer is used to map the text token sequence into a text embedding representation; A cross-modal attention adapter, with its first input connected to the output of the speech encoder and its second input connected to the output of the text embedding layer, is used to fuse the audio feature representation and the text embedding representation based on a multi-head cross-attention mechanism, and output the fused multimodal feature representation; The large language model decoder, whose input is connected to the output of the cross-modal attention adapter, is used to generate a sequence of hidden states of the target language step by step based on the multimodal feature representation; A global memory network, whose input is connected to the output of the large language model decoder, is used to update the global memory matrix based on the hidden state sequence using a gating mechanism, and outputs the updated global memory matrix. A dynamic convolutional decoder, whose first input is connected to the output of the large language model decoder and whose second input is connected to the output of the global memory network, is used to perform multi-scale convolution on the hidden state sequence to extract local features, and to fuse the local features with the updated global memory matrix based on an attention mechanism to output an optimized target language hidden representation. The output layer, whose input is connected to the output of the dynamic convolutional decoder, is used to output the target language translated text based on the optimized target language hidden representation. The speech translation model is trained based on the source language audio feature sequence and the text token sequence until the convergence condition is met.

2. The speech translation model training method based on cross-modal attention, global memory, and dynamic convolution as described in claim 1, characterized in that, The samples in the training dataset are preprocessed to obtain source language audio feature sequences and text token sequences, specifically including: The source language audio data is resampled, framed, and its features are extracted to obtain the source language audio feature sequence; The source language text transcription data and the target language text translation data are processed by word segmentation and sub-wording to obtain the source language text token sequence and the target language text token sequence.

3. The method for training a speech translation model based on cross-modal attention, global memory, and dynamic convolution as described in claim 1, characterized in that, The cross-modal attention adapter specifically includes: A linear projection layer, whose input is connected to the output of the speech encoder, is used to project the dimension of the audio feature representation onto the dimension of the text feature representation to obtain the projected audio feature representation. The cross-attention module has its first input connected to the output of the linear projection layer and its second input connected to the output of the text embedding layer. This attention layer is used to perform multi-head cross-attention calculation with the text embedding representation as the query and the projected audio feature representation as the key and value, and outputs a cross-modal attention feature representation. The residual connection and layer normalization layer, with its first input connected to the output of the multi-head cross-attention layer and its second input connected to the output of the text embedding layer, is used to add the cross-modal attention feature representation and the text embedding representation, and perform layer normalization on the addition result to output the fused multimodal feature representation.

4. The method for training a speech translation model based on cross-modal attention, global memory, and dynamic convolution as described in claim 1, characterized in that, The global memory network specifically includes: The sequence compression module, whose input is connected to the output of the large language model decoder, is used to compress and project the hidden state of the current time step into a single vector representation and output the current context feature vector. The gating update module, whose input is connected to the output of the sequence compression module, is used to control the fusion ratio of historical information and current information through a gating mechanism based on the current context feature vector and the global memory matrix of the previous time step, and output the updated initial memory matrix. The context enhancement module, whose input is connected to the output of the gated update module, is used to perform deep encoding on the updated initial memory matrix through a self-attention mechanism and a feedforward neural network, and output the updated global memory matrix.

5. The method for training a speech translation model based on cross-modal attention, global memory, and dynamic convolution as described in claim 1, characterized in that, The dynamic convolutional decoder specifically includes: A multi-scale convolutional layer, whose input is connected to the output of the large language model decoder, is used to perform multi-scale convolutional features based on the hidden state sequence to obtain multi-scale features. The multi-scale convolutional layer includes multiple parallel groups of one-dimensional convolutional layers, each group of one-dimensional convolutional layers using convolutional kernels of different sizes. The feature fusion layer, whose input is connected to the output of the multi-scale convolutional layer, is used to perform fusion operations on the multi-scale features to generate a unified multi-scale fused feature representation. The memory attention fusion layer has its first input connected to the output of the feature fusion layer and its second input connected to the output of the global memory network. It is used to use the multi-scale fusion feature representation as a query and the global memory matrix as a key and value to perform multi-head attention calculation, realize the interaction and fusion of the current local context and global historical context information, and output attention-enhanced features. A linear output layer, whose input is connected to the output of the memory-attention fusion layer, is used to linearly transform the attention-enhancing features and project them onto a preset hidden dimension, outputting an optimized target language hidden representation.

6. The method for training a speech translation model based on cross-modal attention, global memory, and dynamic convolution as described in claim 1, characterized in that, The speech translation model is trained based on the source language audio feature sequence and the text token sequence until convergence is achieved, specifically including: Based on the parameter update strategy, the speech translation model is fine-tuned and trained using the source language audio feature sequence and the text token sequence until convergence is achieved. The parameter update strategy includes: injecting LoRA low-rank adaptation parameters into the key attention sub-modules in the large language model decoder; updating only the injected LoRA parameters and all parameters of the custom module; and keeping the remaining original parameters of the large language model decoder frozen. The custom module includes the cross-modal attention adapter, the global memory network, and the dynamic convolutional decoder.

7. A speech translation method based on cross-modal attention, global memory, and dynamic convolution, characterized in that, The speech translation method based on cross-modal attention, global memory, and dynamic convolution includes: Acquire the audio data of the language to be processed; Based on the audio data of the language to be processed and the initial text prompt, the target language translation text is obtained using a trained speech translation model. The speech translation model is trained using the speech translation model training method based on cross-modal attention, global memory and dynamic convolution as described in any one of claims 1-6. The initial text prompt is a preset text instruction or a translation task text prompt provided by the user.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the speech translation method based on cross-modal attention, global memory, and dynamic convolution as described in any one of claims 1-6, or the speech translation method based on cross-modal attention, global memory, and dynamic convolution as described in claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the speech translation method based on cross-modal attention, global memory, and dynamic convolution as described in any one of claims 1-6, or the speech translation method based on cross-modal attention, global memory, and dynamic convolution as described in claim 7.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the speech translation method based on cross-modal attention, global memory, and dynamic convolution as described in any one of claims 1-6, or the speech translation method based on cross-modal attention, global memory, and dynamic convolution as described in claim 7.

Images