A low-resource language-oriented speech large model adaptation method and device

Abstract

The application discloses a speech large model adaptation method and device for a low-resource language, and relates to the technical field of natural language processing. The method comprises the following steps: first, frame a speech signal, input an encoder with frozen parameters, and extract a time sequence acoustic feature sequence; then, pass through a feature enhancement module composed of linear projection, two-layer Transform and a feedforward network, capture global acoustic context, fuse fine-grained acoustic features through a target modal encoder and linear mapping, construct an adaptation structure based on a dynamic query transformer, utilize an explicit inductive bias connected with time sequence classification to guide the dynamic generation of a query vector, and input the query vector into a large language model trained through a frozen training strategy to generate text. The application realizes efficient feature compression and cross-modal alignment of a variable-length speech signal in a low-resource scene, and realizes the best convergence of model performance under very few samples.

Images