A low-resource language-oriented speech large model adaptation method and device

Abstract

The application discloses a speech large model adaptation method and device for a low-resource language, and relates to the technical field of natural language processing. The method comprises the following steps: first, frame a speech signal, input an encoder with frozen parameters, and extract a time sequence acoustic feature sequence; then, pass through a feature enhancement module composed of linear projection, two-layer Transform and a feedforward network, capture global acoustic context, fuse fine-grained acoustic features through a target modal encoder and linear mapping, construct an adaptation structure based on a dynamic query transformer, utilize an explicit inductive bias of connection time sequence classification to guide dynamic generation of a query vector, and input a text generated by a large language model trained through a frozen training strategy. The application realizes efficient feature compression and cross-modal alignment of a variable-length speech signal in a low-resource scene, and realizes optimal convergence of model performance under very few samples.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method and apparatus for adapting large speech models for low-resource languages. Background Technology

[0002] With the rapid development of artificial intelligence technology, deep learning-based algorithms are reshaping various fields of information processing. Particularly in natural language processing, large language models have achieved milestone breakthroughs, demonstrating superior text understanding, logical reasoning, and generation capabilities. This paradigm's success is rapidly extending to multimodal domains. Among these, speech, as the most fundamental and natural medium of human communication, makes building high-performance, general-purpose large speech models a core competitive point for next-generation human-computer interaction technologies. Driven by this trend, end-to-end large speech models have become a research hotspot in the field of intelligent speech processing. Their core aim is to break down modal barriers and directly process continuous speech signals using the powerful general-purpose reasoning capabilities of large language models. To achieve this goal, building an efficient and accurate cross-modal adapter is crucial. Its role is to transform high-frequency, redundant acoustic features into discrete semantic representations that large language models can understand. However, current model designs are mainly geared towards high- and mid-range resource scenarios such as English and Chinese, typically employing simple linear projection layers or multilayer perceptrons to map the output of the speech encoder to the text space. However, speech signals have extremely high temporal resolution, and this simple linear mapping method lacks an effective information compression mechanism, resulting in excessively long sequences input to large language models containing a large amount of redundancy (such as silence and background noise). In high-resource environments, the model can rely on massive amounts of data to forcibly fit such long sequences, but in low-resource language scenarios, due to the extreme scarcity of training data, this "long sequence, high redundancy" feature input makes it difficult for the model to capture key semantics, easily leading to overfitting, and incurring huge inference computation costs. In the field of computer vision, in order to achieve more efficient feature compression and cross-modal alignment in data-constrained environments, researchers have introduced the query transformer (Q-Former) mechanism. The core advantage of this mechanism is that it uses a set of compact, learnable static query vectors to extract the most critical semantic information from the original high-dimensional features through a cross-attention mechanism, thereby significantly compressing the sequence length while retaining the core features. However, directly transferring this mechanism, designed for "static images," to the "dynamic speech" domain and applying it under low-resource conditions faces serious adaptive failures, mainly in the following two aspects: First, the temporal structural mismatch between "fixed-length queries" and "variable-length speech." Image content typically has a fixed spatial resolution, and its semantic distribution is relatively static, so a fixed number of static query vectors can effectively cover it. However, speech signals are inherently dynamic temporal streams with highly variable durations, whose duration fluctuates dramatically with speech rate, pauses, and content. Applying a fixed number of static query vectors directly to dynamic speech will lead to severe mismatches. For short speech, too many query vectors will introduce a large amount of redundant noise, diluting semantic density; while for long speech, a limited number of query vectors cannot fully cover all acoustic details, resulting in the loss of key information (such as word endings and intonation).This structural rigidity makes the standard Q-Former difficult to adapt to the dynamic changes in natural language streams. Secondly, the "implicit blind learning" mechanism collapses under low-resource conditions. The query vectors of the standard Q-Former are randomly initialized, and their physical meaning (i.e., which part of the speech is being studied) relies entirely on massive amounts of paired data for end-to-end implicit learning. In high-resource scenarios, the model has enough data to "teach" these query vectors how to align; however, in low-resource language scenarios, the extremely scarce labeled data is insufficient to drive this blind implicit learning process. Query vectors often fail to converge to effective acoustic boundaries, instead "memorizing" specific noise patterns in the training set, resulting in extremely poor model generalization ability, i.e., severe overfitting. In summary, existing linear adapters cannot solve the sequence redundancy problem, and directly borrowing the Q-Former mechanism from the image domain, due to its static fixed-length structure and implicit dependence on large amounts of data, cannot adapt to the variable length characteristics of speech and low-resource data environments. Therefore, a new adaptation mechanism is urgently needed that can dynamically generate query vectors based on speech content and introduce explicit inductive bias to reduce data dependence. Summary of the Invention

[0003] To address the problem in existing technologies where query vectors often fail to converge to effective acoustic boundaries in low-resource language scenarios, resulting in extremely poor model generalization ability and severe overfitting, this invention provides a method and apparatus for adapting large speech models for low-resource languages. The technical solution is as follows: On the one hand, a method for adapting large speech models for low-resource languages ​​is provided. This method is implemented by a device for adapting large speech models for low-resource languages, and includes: S1: Acquire the speech audio signal to be processed, perform frame segmentation processing, input it into the frozen speech encoder, and after temporal compression, obtain the original acoustic feature vector sequence containing temporal information. The frozen speech encoder is based on the mHuBERT-147 speech encoder, and is obtained through training using a frozen training strategy and forced settings. The frozen training strategy includes keeping all parameters of the speech encoder from being updated with gradients during model training. S2: Input the original acoustic feature vector sequence into the feature enhancement module to obtain the final enhanced features. The feature enhancement module includes a linear projection layer, a transformer enhancer, and a feedforward network layer. The transformer enhancer includes a two-layer Transformer structure, which uses a multi-head self-attention mechanism to capture the global acoustic context. S3: Using a target modal encoder and a linear mapping layer, the original acoustic feature vector sequence and the final enhanced features are mapped to obtain the final modal fusion features, which incorporate the context vector of fine-grained acoustic information. S4: A lightweight auxiliary automatic speech recognition head is used to perform connection-time classification-based mapping processing on the final modality fusion features. After posterior probability calculation and automatic filtering, and finally adaptive analysis, a variable number of dynamic query vectors are obtained. The adaptive analysis includes generating more dynamic query vectors for speech segments with fast speech speed or large information content, and generating less than a preset number of dynamic query vectors for speech segments with slow speech speed. S5: Based on the dynamic query vector and the final modal fusion features, cross-modal information aggregation calculation is performed using the cross-attention formula, information backtracking retrieval and alignment fusion are performed, and finally the data is concatenated and input into the final large language model that can perform cross-modal semantic alignment to obtain the final generated text string.

[0004] Preferably, the audio signal to be processed in step S1 is subjected to frame segmentation and input into a frozen speech encoder. After temporal compression, a sequence of original acoustic feature vectors containing temporal information is obtained. The frozen speech encoder is based on the mHuBERT-147 speech encoder and is trained using a frozen training strategy and forced settings. The frozen training strategy includes keeping all parameters of the speech encoder from being updated with gradients during model training, including: S11: The audio acquisition module acquires the speech audio signal to be processed to obtain the original speech waveform; S12: According to the preset frame length parameter, the original speech waveform is processed by frame segmentation to obtain a continuous acoustic frame sequence. S13: Construct a speech encoder, which includes a self-supervised model, comprising an mHuBERT-147 self-supervised model and a feature extraction function; S14: Train the self-supervised model to obtain a pre-trained self-supervised model. The training includes adopting a frozen training strategy, configuring model parameters, obtaining a parameter freeze instruction, and obtaining a frozen state speech encoder by forcibly setting the update gradient mechanism of the model parameters related to the speech encoder. The update gradient mechanism includes zero-value gradients. The parameter freeze instruction is used to keep all parameters of the speech encoder from being updated by gradients during model training. S15: Input the acoustic frame sequence into the frozen speech encoder, and after temporal compression, obtain the original acoustic feature vector sequence containing temporal information.

[0005] Preferably, in step S2, the original acoustic feature vector sequence is input into the feature enhancement module to obtain the final enhanced features. The feature enhancement module includes a linear projection layer, a transformer enhancer, and a feedforward network layer. The transformer enhancer includes a two-layer Transformer structure, which utilizes a multi-head self-attention mechanism to capture the global acoustic context, including: S21: Input the original acoustic feature vector sequence into the linear projection layer to obtain high-dimensional spatial features. The linear projection layer is used to map the original acoustic features to the internal dimensions of the adapter. S22: Input the high-dimensional spatial features into the transformer enhancer for context capture to obtain intermediate processing features; S23: Input the intermediate processing features into the feedforward network layer for nonlinear transformation to obtain the final enhanced features.

[0006] Preferably, step S3 employs a target modal encoder and a linear mapping layer to map the original acoustic feature vector sequence and the final enhanced features to obtain the final modal fusion features. These final modal fusion features fuse a context vector of fine-grained acoustic information, including: S31: Using a target modal encoder, the original acoustic feature vector sequence is encoded with target features to obtain a target query sequence, which serves as the query Q of the cross-attention mechanism; S32: Input the final enhanced features into the first linear mapping layer to obtain the key vector K, which is used to calculate the similarity with the target query sequence; S33: Input the final enhanced features into the second linear mapping layer to obtain the value vector V. The value vector V carries fine-grained acoustic information to be fused, including subtle interjections and emotional intonation. S34: Based on the target query sequence and the key vector K, the attention score is obtained through scaling dot product operation; S35: The Softmax function is used for normalization to obtain attention weights, which represent the degree of attention the target query sequence pays to information at different positions in the value vector V; S36: Input the attention weights and the value vector V into the weighted summation module to obtain a context vector that incorporates fine-grained acoustic information; S37: Input the context vector into the subsequent processing network to obtain the final modality fusion features.

[0007] Preferably, step S4 employs a lightweight assisted automatic speech recognition head to perform connection-time classification-based mapping processing on the final modality fusion features. After posterior probability calculation and automatic filtering, and finally adaptive analysis, a variable number of dynamic query vectors are obtained. The adaptive analysis includes generating more dynamic query vectors for fast-paced or information-rich speech segments, and generating fewer than a preset number of dynamic query vectors for slow-paced speech segments. S41: A lightweight auxiliary automatic speech recognition head is used to map the final modality fusion features to the vocabulary probability distribution, and then connects them to the characteristics of the temporal classification loss function. After posterior probability calculation, the posterior probability of the temporal classification for each frame is obtained. S42: Using dynamic shrinking logic, the final modality fusion features are automatically filtered based on the posterior probability of the connection temporal classification to obtain a set of high-information frames with significant connection temporal classification peaks. The dynamic shrinking logic is based on the characteristics of the connection temporal classification algorithm that generates probability spikes at effective phoneme positions and generates blank labels in non-information regions. The automatic filtering process includes discarding redundant blank frames. S43: Using the high-information-content frames, a query vector is constructed. After adaptive analysis, a variable number of dynamic query vectors are obtained. The adaptive analysis includes contextual information interaction and feature enhancement of key frames of the variable-length query sequence through a self-attention mechanism. The size of the index set of the variable-number dynamic query vectors is smaller than the total number of speech frames.

[0008] Preferably, in step S5, based on the dynamic query vector and the final modality fusion features, cross-modal information aggregation calculation is performed using a cross-attention formula, followed by information backtracking retrieval and alignment fusion. Finally, the data is concatenated and input into the final large language model capable of cross-modal semantic alignment to obtain the final generated text string, including: S51: Based on the dynamic query vector and the final modality fusion features, cross-modality information aggregation calculation is performed using the cross-attention formula to obtain the aggregated feature information; S52: Using the dynamic query vector, perform information backtracking retrieval on the aggregated feature information to obtain the detailed information that was compressed during the shrinkage process; S53: Through feature fusion processing, the aggregated feature information is aligned and fused with the detail information to generate semantic alignment features. The amount of data in the semantic alignment features is compared with the full enhanced feature sequence. S54: Input the semantic alignment features and the embedding vector of the text instructions into the splicing module to obtain a multimodal input sequence; S55: Input the multimodal input sequence into the final large language model that can perform cross-modal semantic alignment to obtain the final generated text string.

[0009] Preferably, the training process for the final large language model capable of cross-modal semantic alignment includes: S551: Select a pre-trained large language model and use low-rank adaptive technology to freeze the original parameters of a preset part. The pre-trained large language model includes the TiLamb-7B model based on Llama2 and incrementally pre-trained on Tibetan corpus. S552: Integrate the low-rank matrix into the attention module of the pre-trained large language model, and configure the rank, scaling factor and Dropout rate to obtain a trainable large language model. S553: ​​Based on the generation task and the alignment task, a joint objective function is obtained by combining loss functions, wherein the joint objective function includes generation loss and alignment loss; S554: Calculate the generation loss by calculating the cross-entropy loss of the trainable large language model in predicting the target text given audio features and instructions; S555: Calculate the connection-time classification loss of the automatic speech recognition head to improve the accuracy of the prediction results of phoneme boundaries and obtain the alignment loss. The improved accuracy of the prediction results of phoneme boundaries is used to accurately locate semantic peaks through shrinking logic to characterize key semantic information in speech. S556: The generation loss and alignment loss are added together to obtain the joint loss. The model weights are updated by forward calculation according to the update formula. When the joint objective function converges, the training ends and the final large language model capable of cross-modal semantic alignment is obtained.

[0010] On the other hand, a speech model adaptation device for low-resource languages ​​is provided. This device is applied to the speech model adaptation method for low-resource languages. The device includes: The speech and audio module is used to acquire the speech and audio signals to be processed, perform frame-by-frame processing, input the frozen speech encoder, and compress it in the time domain to obtain the original acoustic feature vector sequence containing temporal information. The frozen speech encoder is based on the mHuBERT-147 speech encoder and is obtained through training using a frozen training strategy and forced settings. The frozen training strategy includes keeping all parameters of the speech encoder from being updated with gradients during model training. Feature enhancement module: The original acoustic feature vector sequence is input into the feature enhancement module to obtain the final enhanced features. The feature enhancement module includes a linear projection layer, a transformer enhancer and a feedforward network layer. The transformer enhancer includes a two-layer Transformer structure. The two-layer Transformer structure uses a multi-head self-attention mechanism to capture the global acoustic context. Mapping processing module: Used to map the original acoustic feature vector sequence and the final enhanced features using a target modal encoder and a linear mapping layer to obtain the final modal fusion feature, which incorporates the context vector of fine-grained acoustic information; Adaptive Analysis Module: Used with a lightweight assisted automatic speech recognition head, the module performs connection-time classification-based mapping on the final modality fusion features. After posterior probability calculation and automatic filtering, the module undergoes adaptive analysis to obtain a variable number of dynamic query vectors. The adaptive analysis includes generating more dynamic query vectors for fast-paced or information-rich speech segments and generating fewer than a preset number of dynamic query vectors for slow-paced speech segments. The aggregation calculation module is used to perform cross-modal information aggregation calculation based on the dynamic query vector and the final modal fusion features, through the cross-attention formula, to perform information backtracking retrieval and alignment fusion, and finally to concatenate the data. The input is the final large language model that can perform cross-modal semantic alignment, and the final generated text string is obtained. Model training module: Used to select a pre-trained large language model, freeze the original parameters of a preset part using low-rank adaptive technique, define a joint objective function, update the model weights, and end training when the joint objective function converges, thus obtaining the final large language model that can perform cross-modal semantic alignment.

[0011] On the other hand, a speech model adaptation device for low-resource languages ​​is provided, the speech model adaptation device for low-resource languages ​​includes: a processor; a memory, the memory storing computer-readable instructions, which, when executed by the processor, implement the method described in any of the above-described speech model adaptation methods for low-resource languages.

[0012] On the other hand, a computer-readable storage medium is provided, characterized in that program code is stored in the computer-readable storage medium.

[0013] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: A lightweight speech adaptation architecture based on a dynamic query transformer (CTC) breaks down the temporal barrier between traditional static fixed-length queries and dynamic variable-length speech through a CTC-guided dynamic query generation mechanism, solving the problem of high-precision alignment of variable-length speech in low-resource environments. It introduces explicit inductive bias, using prior knowledge from the auxiliary recognition head to directly guide feature selection, replacing the implicit dependence on massive amounts of data in traditional methods. This significantly reduces the training threshold and overfitting risk in low-resource scenarios, achieving model performance convergence with very few samples. Furthermore, it constructs a dual-stream feature fusion mechanism of "dynamic contraction + panoramic backtracking," which significantly compresses the sequence length of the input large model and reduces inference computation overhead while fully preserving all features as key-value pairs. This ensures that fine-grained paralinguistic information such as emotion and intonation is not lost, achieving a balance between efficient computation and deep understanding. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart of a speech model adaptation method for low-resource languages ​​provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a large language model capable of cross-modal semantic alignment provided by an embodiment of the present invention; Figure 3 This is a block diagram of a speech model adaptation device for low-resource languages ​​provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a speech model adaptation device for low-resource languages ​​provided in an embodiment of the present invention. Detailed Implementation

[0016] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0017] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0018] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0019] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0020] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0021] This invention provides a method for adapting a large speech model for low-resource languages. This method can be implemented by a device for adapting a large speech model for low-resource languages, which can be a terminal or a server. Figure 1 The flowchart shown is for a large speech model adaptation method for low-resource languages. The processing flow of this method may include the following steps: Preferably, the audio signal to be processed is acquired, frame-by-frame processed, input into a frozen speech encoder, and compressed in the temporal domain to obtain the original acoustic feature vector sequence containing temporal information. The frozen speech encoder is based on the mHuBERT-147 speech encoder, trained using a frozen training strategy and forced settings. The frozen training strategy includes keeping all parameters of the speech encoder from being updated with gradients during model training, including: The audio acquisition module acquires the speech audio signal to be processed to obtain the original speech waveform, which includes speech data sampled at 16kHz. According to the preset frame length parameter, the original speech waveform is processed by frame segmentation to obtain a continuous acoustic frame sequence. A speech encoder is constructed, which includes a self-supervised model, comprising an mHuBERT-147 self-supervised model and a feature extraction function, wherein the feature extraction function is used to extract the hidden state sequence of the 9th layer. The self-supervised model is trained to obtain a pre-trained self-supervised model. The training includes adopting a freeze training strategy, configuring model parameters, obtaining a parameter freeze instruction, and obtaining a frozen speech encoder by forcibly setting the update gradient mechanism of the model parameters related to the speech encoder. The update gradient mechanism includes zero-value gradients. The parameter freeze instruction is used to keep all parameters of the speech encoder from being updated by gradients during model training. The acoustic frame sequence is input into a frozen speech encoder and compressed in the temporal domain to obtain an original acoustic feature vector sequence containing temporal information. The feature dimension of the original acoustic feature vector sequence is 768.

[0022] In some embodiments, the speech encoder employs an mHuBERT-147 self-supervised model pre-trained on large-scale multilingual data to process the raw speech waveform sampled at 16kHz. Through feature extraction function Mapped to the 9th layer hidden state sequence (where T is the number of downsampled frames and 768 is the feature dimension), and all model parameters are forcibly set. Update gradient Freeze training to effectively avoid catastrophic forgetting and minimize GPU memory usage in situations where data is scarce.

[0023] Preferably, the original acoustic feature vector sequence is input into the feature enhancement module to obtain the final enhanced features. The feature enhancement module includes a linear projection layer, a transformer enhancer, and a feedforward network layer. The transformer enhancer includes a two-layer Transformer structure, which utilizes a multi-head self-attention mechanism to capture the global acoustic context, including: The original acoustic feature vector sequence is input into a linear projection layer to obtain high-dimensional spatial features. The linear projection layer is used to map the original acoustic features to the internal dimensions of the adapter. The high-dimensional spatial features are input into the transformer enhancer for context capture to obtain intermediate processing features. The intermediate processing features are input into the feedforward network layer for nonlinear transformation to obtain the final enhanced features.

[0024] In some embodiments, the adapter first uses linear projection 768 dimensions Mapped to the adapter's internal dimensions. Then fed into a transformer enhancer containing two layers of Transformers.

[0025] Preferably, a target modal encoder and a linear mapping layer are used to map the original acoustic feature vector sequence and the final enhanced features to obtain the final modal fusion features. The final modal fusion features fuse a context vector of fine-grained acoustic information, including: A target modal encoder is used to encode the target features of the original acoustic feature vector sequence to obtain the target query sequence, which serves as the query Q of the cross-attention mechanism. The final enhanced features are input into the first linear mapping layer to obtain the key vector K, which is used to calculate the similarity with the target query sequence. The final enhanced features are input into the second linear mapping layer to obtain a value vector V. The value vector V carries fine-grained acoustic information to be fused, including subtle interjections and emotional intonation. Based on the target query sequence and the key vector K, an attention score is obtained through a scaled dot product operation; The attention weights are obtained by normalization using the Softmax function. These attention weights represent the degree of attention the target query sequence pays to information at different positions in the value vector V. The attention weights and the value vector V are input into the weighted summation module to obtain a context vector that incorporates fine-grained acoustic information. The context vector is then input into the subsequent processing network to obtain the final modality fusion features. In some embodiments, the final enhanced features They are fully preserved as keys (K) and values ​​(V) for subsequent cross-attention, ensuring that fine-grained acoustic details (such as intonation and emotion) are not lost.

[0026] Preferably, a lightweight auxiliary automatic speech recognition head is used to perform connection-time classification-based mapping processing on the final modality fusion features. After posterior probability calculation and automatic filtering, adaptive analysis is finally performed to obtain a variable number of dynamic query vectors. The adaptive analysis includes generating more dynamic query vectors for fast-paced or information-rich speech segments, and generating fewer than a preset number of dynamic query vectors for slow-paced speech segments. A lightweight assisted automatic speech recognition head is used to map the final modality fusion features to a vocabulary probability distribution, and then connects them to the characteristics of the temporal classification loss function. After posterior probability calculation, the posterior probability of the temporal classification for each frame is obtained. A dynamic shrinking logic is adopted to automatically filter the final modality fusion features based on the posterior probability of the connection temporal classification, resulting in a set of high-information frames with significant connection temporal classification peaks. The dynamic shrinking logic is based on the characteristics of the connection temporal classification algorithm that generates probability spikes at effective phoneme positions and blank labels in non-information regions. The automatic filtering process includes discarding redundant blank frames. Using the high-information-content frames, query vectors are constructed. Through adaptive analysis, a variable number of dynamic query vectors are obtained. The adaptive analysis includes contextual information interaction and feature enhancement of key frames in the variable-length query sequence through a self-attention mechanism. The size of the index set of the variable-number dynamic query vectors is smaller than the total number of speech frames.

[0027] In some embodiments, it should be noted that a lightweight assisted automatic speech recognition head is used to... Map the vocabulary size to the target language. Calculate the connection-temporal classification posterior probability for each frame and select the index set based on a threshold δ. To construct dynamic query sequences (where k) T).

[0028] It should be noted that a cross-attention layer is constructed to handle the generated dynamic queries. As a query, full data feature For the key value, use the cross-attention formula:

[0029] in, Let K be the projected weight matrix. Let V be the projection weight matrix, and the final enhanced feature be... , Let K be the dimension of the vector. The output is generated to effectively recover the compressed contextual acoustic and emotional information while achieving a compression rate of 2 to 10 times.

[0030] Preferably, based on the dynamic query vector and the final modality fusion features, cross-modal information aggregation calculation is performed using a cross-attention formula, followed by information backtracking retrieval and alignment fusion. Finally, the data is concatenated and input into the final large language model capable of cross-modal semantic alignment to obtain the final generated text string, including: Based on the dynamic query vector and the final modality fusion features, cross-modal information aggregation calculation is performed using the cross-attention formula to obtain the aggregated feature information; Using the dynamic query vector, information backtracking retrieval is performed on the aggregated feature information to obtain the detailed information that was compressed during the shrinking process; Through feature fusion processing, the aggregated feature information is aligned and fused with the detail information to generate semantic alignment features. The data volume of the semantic alignment features is 2 to 10 times that of the full enhanced feature sequence. The semantic alignment features and the embedding vector of the text instructions are input into the concatenation module to obtain a multimodal input sequence; The multimodal input sequence is fed into a large language model that can perform cross-modal semantic alignment to obtain the final generated text string.

[0031] Preferably, the training process for the final large language model capable of cross-modal semantic alignment includes: Select a pre-trained large language model and use low-rank adaptive technique to freeze the original parameters of a preset part. The pre-trained large language model includes the TiLamb-7B model based on Llama2 and incrementally pre-trained on Tibetan corpus. The low-rank matrix is ​​integrated into the attention module of the pre-trained large language model, and the rank, scaling factor and dropout rate are configured to obtain a trainable large language model. Based on the generation task and the alignment task, a joint objective function is obtained by combining loss functions, wherein the joint objective function includes generation loss and alignment loss; The generation loss is obtained by calculating the cross-entropy loss of the trainable large language model in predicting the target text given audio features and instructions; The connection-time classification loss of the automatic speech recognition head is calculated to improve the accuracy of the prediction results of phoneme boundaries, and the alignment loss is obtained. The improved accuracy of the prediction results of phoneme boundaries is used to accurately locate semantic peaks through shrinkage logic to characterize key semantic information in speech. The generation loss and alignment loss are added together to obtain the joint loss. The model weights are then updated by forward calculation according to the update formula. Training ends when the joint objective function converges, resulting in a final large language model capable of cross-modal semantic alignment.

[0032] In some embodiments, the base model is a TiLamb-7B model based on Llama 2 and incrementally pre-trained on a Tibetan corpus to ensure native language capability. LoRA technique is employed to accommodate low-resource fine-tuning. All original weights are frozen, and only rank is introduced. Scaling factor and Dropout rate The low-rank adaptation module utilizes a low-rank matrix. and According to the updated formula By performing forward computation on the weights, high-precision cross-modal semantic alignment can be achieved while greatly reducing computational overhead.

[0033] The acoustic features Z output by the adapter are concatenated with the embedding of the text instructions encoded by the tokenizer, and used as the input to the final large language model that can perform cross-modal semantic alignment.

[0034] It should be noted that the total loss It consists of two parts: generation loss : Computational large language model to predict target text given audio features and instructions The cross-entropy loss forces the model to learn to understand speech features and generate fluent text. Alignment loss. Calculate the connection-time classification loss of the automatic speech recognition head in the adapter. This is not merely an auxiliary task, but a structured constraint. It forces the automatic speech recognition head to accurately predict phoneme boundaries, thereby ensuring that the "shrinkage logic" can accurately locate semantic peaks. Without this loss, the shrinkage logic will fail, leading to chaotic selection of dynamic query vectors. In this embodiment, .

[0035] The above is an introduction to the method embodiments. The following describes the solution described in this application through device embodiments.

[0036] Figure 3 This is a block diagram illustrating a speech model adaptation apparatus for low-resource languages ​​according to an exemplary embodiment. The apparatus is used in a speech model adaptation method for low-resource languages. (Refer to...) Figure 3 The device includes a speech audio module 310, a feature enhancement module 320, a mapping processing module 330, an adaptive analysis module 340, and an aggregation calculation module 350.

[0037] Speech and audio module 310: used to acquire speech and audio signals to be processed, perform frame-by-frame processing, input to a frozen speech encoder, and after temporal compression, obtain the original acoustic feature vector sequence containing temporal information. The frozen speech encoder is based on the mHuBERT-147 speech encoder, and is obtained through training using a frozen training strategy and forced settings. The frozen training strategy includes keeping all parameters of the speech encoder from being updated with gradients during model training. Feature enhancement module 320: used to input the original acoustic feature vector sequence into the feature enhancement module to obtain the final enhanced features. The feature enhancement module includes a linear projection layer, a transformer enhancer and a feedforward network layer. The transformer enhancer includes a two-layer Transformer structure. The two-layer Transformer structure uses a multi-head self-attention mechanism to capture the global acoustic context. Mapping processing module 330: Used to map the original acoustic feature vector sequence and the final enhanced features using a target modal encoder and a linear mapping layer to obtain the final modal fusion feature, wherein the final modal fusion feature incorporates the context vector of fine-grained acoustic information; Adaptive analysis module 340: Used with a lightweight assisted automatic speech recognition head to perform connection-time classification-based mapping processing on the final modality fusion features, after posterior probability calculation and automatic filtering, and finally after adaptive analysis, to obtain a variable number of dynamic query vectors. The adaptive analysis includes generating more dynamic query vectors for speech segments with fast speech speed or large information content, and generating less than a preset number of dynamic query vectors for speech segments with slow speech speed. The aggregation calculation module 350 is used to perform cross-modal information aggregation calculation based on the dynamic query vector and the final modal fusion features, through the cross-attention formula, to perform information backtracking retrieval and alignment fusion, and finally to concatenate the data. The final large language model that can perform cross-modal semantic alignment is then input to obtain the final generated text string.

[0038] A speech model adaptation device for low-resource languages, the speech model adaptation device for low-resource languages ​​includes: a processor; a memory, the memory storing computer-readable instructions, which, when executed by the processor, implement the method described in any of the above-described speech model adaptation methods for low-resource languages.

[0039] Figure 4 This is a schematic diagram of the structure of a speech model adaptation device for low-resource languages ​​provided in an embodiment of the present invention, as shown below. Figure 4 As shown, speech model adaptation devices for low-resource languages ​​may include the above-mentioned... Figure 3 The illustrated speech model adaptation device for low-resource languages. Optionally, the speech model adaptation device 410 for low-resource languages ​​may include a first processor 2001.

[0040] Optionally, the speech model adaptation device 410 for low-resource languages ​​may also include a memory 2002 and a transceiver 2003.

[0041] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0042] The following is combined with Figure 4 The following is a detailed introduction to the various components of the speech model adaptation device 410 for low-resource languages: The first processor 2001 is the control center of the speech model adaptation device 410 for low-resource languages. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0043] Optionally, the first processor 2001 can perform various functions of the speech large model adapter device 410 for low-resource languages ​​by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0044] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0045] In a specific implementation, as one example, the speech large model adaptation device 410 for low-resource languages ​​may also include multiple processors, for example... Figure 4 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0046] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0047] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the speech large model adapter device 410 for low-resource languages. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0048] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0049] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0050] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected via the interface circuit of the speech model adapter 410 for low-resource languages. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0051] It should be noted that, Figure 4 The structure of the speech large model adaptation device 410 for low-resource languages ​​shown in the diagram does not constitute a limitation on the router. Actual knowledge structure recognition devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0052] Furthermore, the technical effects of the speech large model adaptation device 410 for low-resource languages ​​can be referred to the technical effects of the speech large model adaptation method for low-resource languages ​​described in the above method embodiments, and will not be repeated here.

[0053] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0054] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0055] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0056] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0057] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0058] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0059] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0060] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0061] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0062] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0063] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0064] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0065] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for adapting large speech models for low-resource languages, characterized in that, The method includes: S1: Acquire the speech audio signal to be processed, perform frame segmentation processing, input it into the frozen speech encoder, and after temporal compression, obtain the original acoustic feature vector sequence containing temporal information. The frozen speech encoder is based on the mHuBERT-147 speech encoder, and is obtained through training using a frozen training strategy and forced settings. The frozen training strategy includes keeping all parameters of the speech encoder from being updated with gradients during model training. S2: Input the original acoustic feature vector sequence into the feature enhancement module to obtain the final enhanced features. The feature enhancement module includes a linear projection layer, a transformer enhancer, and a feedforward network layer. The transformer enhancer includes a two-layer Transformer structure, which uses a multi-head self-attention mechanism to capture the global acoustic context. S3: Using a target modal encoder and a linear mapping layer, the original acoustic feature vector sequence and the final enhanced features are mapped to obtain the final modal fusion features, which incorporate the context vector of fine-grained acoustic information. S4: A lightweight assisted automatic speech recognition head is used to perform connection-time classification-based mapping processing on the final modality fusion features. After posterior probability calculation and automatic filtering, adaptive analysis is performed to obtain a variable number of dynamic query vectors. The adaptive analysis includes generating more dynamic query vectors for fast-paced or information-rich speech segments and fewer dynamic query vectors for slow-paced speech segments, including: S41: A lightweight auxiliary automatic speech recognition head is used to map the final modality fusion features to the vocabulary probability distribution, and then connects them to the characteristics of the temporal classification loss function. After posterior probability calculation, the posterior probability of the temporal classification for each frame is obtained. S42: Using dynamic shrinking logic, the final modality fusion features are automatically filtered based on the posterior probability of the connection temporal classification to obtain a set of high-information frames with significant connection temporal classification peaks. The dynamic shrinking logic is based on the characteristics of the connection temporal classification algorithm that generates probability spikes at effective phoneme positions and generates blank labels in non-information regions. The automatic filtering process includes discarding redundant blank frames. S43: Using the high-information-content frames, a query vector is constructed. After adaptive analysis, a variable number of dynamic query vectors are obtained. The adaptive analysis includes contextual information interaction and feature enhancement of key frames of the variable-length query sequence through a self-attention mechanism. The size of the index set of the variable-number dynamic query vectors is smaller than the total number of speech frames. S5: Based on the dynamic query vector and the final modality fusion features, cross-modal information aggregation calculation is performed using the cross-attention formula. Information backtracking retrieval and alignment fusion are then performed, and finally, the data is concatenated and input into the final large language model capable of cross-modal semantic alignment to obtain the final generated text string, including: S51: Based on the dynamic query vector and the final modality fusion features, cross-modality information aggregation calculation is performed using the cross-attention formula to obtain the aggregated feature information; S52: Using the dynamic query vector, perform information backtracking retrieval on the aggregated feature information to obtain the detailed information that was compressed during the shrinkage process; S53: Through feature fusion processing, the aggregated feature information and the detail information are aligned and fused to generate semantic alignment features. The amount of data of the semantic alignment features is significantly reduced compared to the full enhanced feature sequence. S54: Input the semantic alignment features and the embedding vector of the text instructions into the splicing module to obtain a multimodal input sequence; S55: Input the multimodal input sequence into the final large language model that can perform cross-modal semantic alignment to obtain the final generated text string.

2. The speech model adaptation method for low-resource languages ​​according to claim 1, characterized in that, The S1 acquires the speech audio signal to be processed, performs frame segmentation, inputs it to the frozen speech encoder, and after temporal compression, obtains the original acoustic feature vector sequence containing temporal information. The frozen speech encoder is based on the mHuBERT-147 speech encoder, and is obtained through training using a frozen training strategy and forced settings. The frozen training strategy includes keeping all parameters of the speech encoder from being updated with gradients during model training, including: S11: The audio acquisition module acquires the speech audio signal to be processed to obtain the original speech waveform; S12: According to the preset frame length parameter, the original speech waveform is processed by frame segmentation to obtain a continuous acoustic frame sequence. S13: Construct a speech encoder, which includes a self-supervised model, comprising an mHuBERT-147 self-supervised model and a feature extraction function; S14: Train the self-supervised model to obtain a pre-trained self-supervised model. The training includes adopting a frozen training strategy, configuring model parameters, obtaining a parameter freeze instruction, and obtaining a frozen state speech encoder by forcibly setting the update gradient mechanism of the model parameters related to the speech encoder. The update gradient mechanism includes zero-value gradients. The parameter freeze instruction is used to keep all parameters of the speech encoder from being updated by gradients during model training. S15: Input the acoustic frame sequence into the frozen speech encoder, and after temporal compression, obtain the original acoustic feature vector sequence containing temporal information.

3. The method for adapting large speech models for low-resource languages ​​according to claim 1, characterized in that, S2 inputs the original acoustic feature vector sequence into the feature enhancement module to obtain the final enhanced features. The feature enhancement module includes a linear projection layer, a transformer enhancer, and a feedforward network layer. The transformer enhancer includes a two-layer Transformer structure that uses a multi-head self-attention mechanism to capture the global acoustic context, including: S21: Input the original acoustic feature vector sequence into the linear projection layer to obtain high-dimensional spatial features. The linear projection layer is used to map the original acoustic features to the internal dimensions of the adapter. S22: Input the high-dimensional spatial features into the transformer enhancer for context capture to obtain intermediate processing features; S23: Input the intermediate processing features into the feedforward network layer for nonlinear transformation to obtain the final enhanced features.

4. The method for adapting large speech models for low-resource languages ​​according to claim 1, characterized in that, The S3 process employs a target modal encoder and a linear mapping layer to map the original acoustic feature vector sequence and the final enhanced features, obtaining the final modal fusion features. These final modal fusion features incorporate a context vector of fine-grained acoustic information, including: S31: Using a target modal encoder, the original acoustic feature vector sequence is encoded with target features to obtain a target query sequence, which serves as the query Q of the cross-attention mechanism; S32: Input the final enhanced features into the first linear mapping layer to obtain the key vector K, which is used to calculate the similarity with the target query sequence; S33: Input the final enhanced features into the second linear mapping layer to obtain the value vector V. The value vector V carries fine-grained acoustic information to be fused, including subtle interjections and emotional intonation. S34: Based on the target query sequence and the key vector K, the attention score is obtained through scaling dot product operation; S35: The Softmax function is used for normalization to obtain attention weights, which represent the degree of attention the target query sequence pays to information at different positions in the value vector V; S36: Input the attention weights and the value vector V into the weighted summation module to obtain a context vector that incorporates fine-grained acoustic information; S37: Input the context vector into the subsequent processing network to obtain the final modality fusion features.

5. The method for adapting large speech models for low-resource languages ​​according to claim 1, characterized in that, The training process for the final large language model capable of cross-modal semantic alignment includes: S551: Select a pre-trained large language model and use low-rank adaptive technology to freeze the original parameters of a preset part. The pre-trained large language model includes the TiLamb-7B model based on Llama2 and incrementally pre-trained on Tibetan corpus. S552: Integrate the low-rank matrix into the attention module of the pre-trained large language model, and configure the rank, scaling factor and Dropout rate to obtain a trainable large language model. S553: ​​Based on the generation task and the alignment task, a joint objective function is obtained by combining loss functions, wherein the joint objective function includes generation loss and alignment loss; S554: Calculate the generation loss by calculating the cross-entropy loss of the trainable large language model in predicting the target text given audio features and instructions; S555: Calculate the connection-time classification loss of the automatic speech recognition head to improve the accuracy of the prediction results of phoneme boundaries and obtain the alignment loss. The improved accuracy of the prediction results of phoneme boundaries is used to accurately locate semantic peaks through shrinking logic to characterize key semantic information in speech. S556: The generation loss and alignment loss are added together to obtain the joint loss. The model weights are updated by forward calculation according to the update formula. When the joint objective function converges, the training ends and the final large language model capable of cross-modal semantic alignment is obtained.

6. A speech model adaptation device for low-resource languages, wherein the speech model adaptation device for low-resource languages ​​is used to implement the speech model adaptation method for low-resource languages ​​as described in any one of claims 1-5, characterized in that, The device includes: The speech and audio module is used to acquire the speech and audio signals to be processed, perform frame-by-frame processing, input the frozen speech encoder, and compress it in the time domain to obtain the original acoustic feature vector sequence containing temporal information. The frozen speech encoder is based on the mHuBERT-147 speech encoder and is obtained through training using a frozen training strategy and forced settings. The frozen training strategy includes keeping all parameters of the speech encoder from being updated with gradients during model training. Feature enhancement module: The original acoustic feature vector sequence is input into the feature enhancement module to obtain the final enhanced features. The feature enhancement module includes a linear projection layer, a transformer enhancer and a feedforward network layer. The transformer enhancer includes a two-layer Transformer structure. The two-layer Transformer structure uses a multi-head self-attention mechanism to capture the global acoustic context. Mapping processing module: Used to map the original acoustic feature vector sequence and the final enhanced features using a target modal encoder and a linear mapping layer to obtain the final modal fusion feature, which incorporates the context vector of fine-grained acoustic information; Adaptive Analysis Module: This module uses a lightweight assisted automatic speech recognition head to perform connection-time classification-based mapping on the final modality fusion features. After posterior probability calculation and automatic filtering, it finally undergoes adaptive analysis to obtain a variable number of dynamic query vectors. The adaptive analysis includes generating more dynamic query vectors for speech segments with fast speech speed or high information content, and generating fewer dynamic query vectors for speech segments with slow speech speed. The aggregation calculation module is used to perform cross-modal information aggregation calculation based on the dynamic query vector and the final modal fusion features, using the cross-attention formula, to perform information backtracking retrieval and alignment fusion, and finally to concatenate the data. The result is then input into the final large language model that can perform cross-modal semantic alignment, and the final generated text string is obtained.

7. A speech model adaptation device for low-resource languages, characterized in that, The processor is adapted for a large speech model for low-resource languages; the memory stores computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 5.

Images