Multi-model speech recognition method based on C-Adapter adapter
Through the combination of C-Adapter adapter and large language model, speech features are extracted and integrated with text information, the problems of low accuracy and slow speed in traditional speech recognition technology are solved, and efficient speech recognition is achieved.
Patent Information
- Application Number
- CN202411179763.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-12-06
AI Technical Summary
In traditional speech recognition technology, the accuracy of speech recognition is low and the recognition speed is slow, making it difficult to meet the requirements of speech conversion in real time.
Using a multi-model speech recognition method based on C-Adapter adapter, speech feature information is extracted through an acoustic encoder, and the C-Adapter adapter performs downsampling and one-dimensional convolution processing, and feature adjustment and normalization are performed in combination with the multi-head attention layer. Generate modal adaptive feature information, and splice it with text task representation information, and use large language models for recognition.
It improves the accuracy and speed of speech recognition, realizes the combination of speech recognition model and large language model, and meets the requirements of speech real-time conversion of text.
Smart Images

Figure CN119091862A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of speech recognition technology, and in particular to a multi-model speech recognition method based on a C-Adapter adapter. Background Art
[0002] Speech recognition refers to the recognition of the content of user input speech through a model. With the development of NLP large model technology, large language models based on large-scale text pre-training such as BERT, GPT, and Llama have achieved good performance in the field of natural language processing technology.
[0003] In traditional technology, speech recognition mainly extracts the audio features of the user's input speech through an encoder, performs feature analysis through an acoustic model, combines the feature vectors corresponding to adjacent speech frames, and matches the corresponding text information to obtain speech recognition results. This not only makes the speech recognition accuracy low, but also the recognition speed is relatively slow, which makes it difficult to meet the requirements of real-time speech-to-text conversion.
[0004] Therefore, a multi-model speech recognition method based on C-Adapter is proposed. Summary of the invention
[0005] In order to solve the above technical problems, the present invention provides a multi-model speech recognition method based on a C-Adapter adapter, which is used to solve the problems of low speech recognition accuracy and slow recognition speed in traditional technologies.
[0006] The embodiment of the present invention provides a multi-model speech recognition method based on a C-Adapter adapter, comprising:
[0007] Obtaining the original speech signal;
[0008] Extracting features from the original speech signal using an acoustic encoder to obtain speech feature information;
[0009] The C-Adapter performs downsampling and one-dimensional convolution processing according to the speech feature information to obtain low-dimensional feature information, and adjusts the attention mechanism of the low-dimensional feature information through a multi-head attention layer to obtain weighted feature information, perform normalization processing and feature enhancement, and generate modality adaptation feature information mapped to the speech recognition space;
[0010] Setting a speech recognition task character, mapping the speech recognition task character to a speech recognition space through a text vector model, and constructing text task representation information;
[0011] splicing the modal adaptation feature information and the text task representation information in the speech recognition space to construct speech information to be recognized;
[0012] The speech information to be recognized is recognized through a large language model to obtain a speech recognition result.
[0013] Preferably, the present invention provides a multi-model speech recognition method based on a C-Adapter adapter, the step of: extracting features from the original speech signal through an acoustic encoder to obtain speech feature information; comprising:
[0014] Obtain a pre-trained acoustic encoder by constructing a convolutional neural network layer, a mask layer, a Transformer layer, a speech processing layer, and a clustering analysis layer;
[0015] Acquire automatic speech recognition data in a training set, train the pre-trained acoustic encoder, and construct an acoustic encoder; optionally, include:
[0016] The convolutional neural network layer performs feature extraction on the automatic speech recognition data to obtain hidden feature information of the sample;
[0017] The mask layer randomly masks the sample sub-feature information in the sample hidden feature information according to a preset mask rate to obtain sample mask feature information;
[0018] Performing feature prediction on the sample mask feature information through a Transformer layer to obtain sample restoration feature information;
[0019] The speech processing layer pre-processes the automatic speech recognition data to obtain the pre-processed sample speech signal, and converts it into the frequency domain through Fourier transform for analysis to obtain sample spectrum feature information; the Mel filter performs energy superposition operation on the sample spectrum feature information to obtain Mel logarithmic feature information, obtains Mel cepstrum parameter information through discrete cosine transform for dynamic feature extraction, and generates Mel cepstrum parameter feature information;
[0020] The cluster analysis layer performs cluster analysis according to the Mel-frequency cepstrum parameter feature information to construct sample cluster codebook information;
[0021] Matching the unmasked sub-information in the sample recovery feature information with the clustering sub-information in the sample clustering codebook information, obtaining the masked clustering sub-information corresponding to the masked sub-information in the sample recovery feature information, and calculating the similarity distance;
[0022] The parameters of the Transformer layer are adjusted to optimize the similarity distance, and when a threshold condition is met, the pre-trained acoustic encoder is used as the acoustic encoder.
[0023] Preferably, the present invention provides a multi-model speech recognition method based on a C-Adapter adapter, the step of: extracting features from the original speech signal through an acoustic encoder to obtain speech feature information; comprising:
[0024] The multi-layer temporal convolutional network in the convolutional neural network layer performs an expansion convolution operation on the original speech signal, and the residual connection between the temporal convolutional networks is used to obtain hidden feature information, the hidden feature information is normalized by a normalization function, and the normalized hidden feature information is calculated by a GELU activation function to obtain hidden speech feature information;
[0025] The masking layer randomly masks the sub-feature information in the hidden speech feature information according to a preset masking rate to obtain masked feature information;
[0026] The encoder in the Transformer layer analyzes the global relationship of the mask feature information according to the mask feature information and the corresponding mask position, and predicts the sub-feature information of the mask through the decoder to obtain the speech feature information.
[0027] Preferably, the present invention provides a multi-model speech recognition method based on a C-Adapter adapter, the steps of: the C-Adapter adapter performs downsampling and one-dimensional convolution processing according to the speech feature information to obtain low-dimensional feature information, and adjusts the attention mechanism of the low-dimensional feature information through a multi-head attention layer to obtain weighted feature information, performs normalization processing and feature enhancement, and generates modality adaptation feature information mapped to the speech recognition space; including:
[0028] Build a pre-trained adapter, connect the acoustic encoder, pre-trained adapter, speech recognition space and large language model in sequence;
[0029] Get the automatic speech recognition data in the training set;
[0030] The acoustic encoder extracts features from the automatic speech recognition data to obtain sample speech feature information;
[0031] The pre-training adapter performs sample feature enhancement on the sample speech feature information, obtains sample modal feature information, and maps it to the speech recognition space;
[0032] Concatenating the sample modal feature information in the speech recognition space with the sample training identification information to construct sample speech recognition information;
[0033] The large language model performs recognition prediction based on the sample speech recognition information to obtain a sample recognition result;
[0034] A loss function is calculated based on the sample recognition result and the sample standard recognition result corresponding to the automatic speech recognition data to optimize the control parameters of the pre-trained adapter.
[0035] Preferably, the present invention provides a multi-model speech recognition method of a C-Adapter adapter, the steps of: the C-Adapter adapter performs downsampling and one-dimensional convolution processing according to the speech feature information to obtain low-dimensional feature information, and adjusts the attention mechanism of the low-dimensional feature information through a multi-head attention layer to obtain weighted feature information, performs normalization processing and feature enhancement, and generates modality adaptation feature information mapped to the speech recognition space; including:
[0036] Down-sampling the speech feature information through a sampling layer in the C-Adapter adapter to obtain a speech feature sampling signal;
[0037] Based on a one-dimensional convolutional neural network, feature compression is performed on the speech feature sampling signal to obtain low-dimensional feature information;
[0038] Calculating the attention value of the sub-feature information in the low-dimensional feature information through a multi-head attention layer, and performing weighted calculation to obtain weighted feature information;
[0039] Performing a residual connection according to the weighted feature information and the speech feature sampling signal, and processing through a first normalization layer to obtain normalized feature information;
[0040] Performing a fully connected transformation on the normalized feature information through a feedforward neural network model to obtain enhanced speech feature information;
[0041] A residual connection is performed based on the normalized feature information and the enhanced speech feature information, and the information is processed through a second normalization layer to obtain modal adaptation feature information, which is mapped into the speech recognition space.
[0042] Preferably, the present invention provides a multi-model speech recognition method of a C-Adapter adapter, the steps of: based on a one-dimensional convolutional neural network, performing feature compression on the speech feature sampling signal to obtain low-dimensional feature information; comprising:
[0043] Obtaining a query vector, a key vector and a value vector of the speech feature sampling signal;
[0044] Based on the one-dimensional convolutional neural network, the size, stride and padding of the convolution kernel are set to build a shared pooling layer;
[0045] Input the query vector, key vector and numerical vector into the shared pooling layer for feature compression, obtain low-dimensional feature information, and input it into the multi-head attention layer;
[0046]
[0047] Wherein, L is the dimension of the obtained low-dimensional feature information, n is the dimension of the feature vector matrix constructed according to the query vector, the key vector and the numerical vector, p represents the padding value, f represents the size of the convolution kernel, s represents the step value, The floor symbol.
[0048] Preferably, the present invention provides a multi-model speech recognition method of a C-Adapter adapter, wherein the steps include: setting speech recognition task characters, mapping the speech recognition task characters to the speech recognition space through a text vector model, and constructing text task representation information; comprising:
[0049] The speech recognition task character is set to "transcript";
[0050] A text vector model is built based on the Embedding layer to map the characters of the speech recognition task to the speech recognition space.
[0051] Preferably, the present invention provides a multi-model speech recognition method of a C-Adapter adapter, the step of: recognizing the speech information to be recognized by a large language model to obtain a speech recognition result; comprising:
[0052] The large language model extracts feature vectors according to the speech information to be recognized, obtains a speech query vector and a speech key-value pair vector, and caches the speech key-value pair vector in a key-value storage disk;
[0053] The self-decoding layer calculates the self-attention value of the voice query vector and the voice key-value pair vector through a self-attention sliding window, and performs weighted calculation according to the voice information to be recognized and the obtained self-attention value to generate weighted self-attention feature information. The first deep feedforward network performs nonlinear transformation to obtain self-decoding feature information;
[0054] According to the self-decoding feature information, a decoding query vector is extracted; a cross-attention value is calculated according to the decoding query vector and the speech key-value pair vector in the key-value storage disk through the cross-attention mechanism unit in the cross-decoding layer, and a weighted calculation is performed according to the self-decoding feature information and the obtained cross-attention value to generate weighted cross-attention feature information, and a second deep feedforward network performs classification prediction to obtain a speech recognition result.
[0055] Compared with the traditional technology, the beneficial effects of the present invention are: a multi-model speech recognition method based on C-Adapter, which realizes the recognition of user speech through an acoustic encoder, a C-Adapter and a large language model, solves the defect of obtaining speech recognition results by feature analysis based on an acoustic model in the traditional technology, extracts speech features through an acoustic encoder and a C-Adapter, generates text task representation information through a text vector model, performs modal fusion in the speech recognition space, and generates speech information to be recognized for processing based on a large language model; the above method realizes the combination of a speech recognition model and a large language model, and effectively improves the accuracy of speech recognition, increases the speed of speech recognition, and meets the requirements of real-time speech conversion into text while improving the speech recognition efficiency.
[0056] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.
[0057] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] Figure 1 A schematic flow chart of a multi-model speech recognition method based on a C-Adapter adapter provided by the present invention. DETAILED DESCRIPTION
[0059] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.
[0060] Embodiment 1:
[0061] The embodiment of the present invention provides a multi-model speech recognition method based on a C-Adapter adapter, referring to Figure 1 ,include:
[0062] Obtaining the original speech signal;
[0063] Extract features from the original speech signal through an acoustic encoder to obtain speech feature information;
[0064] The C-Adapter performs downsampling and one-dimensional convolution processing based on the speech feature information to obtain low-dimensional feature information, and adjusts the attention mechanism of the low-dimensional feature information through the multi-head attention layer to obtain weighted feature information, perform normalization processing and feature enhancement, and generate modality-adaptive feature information mapped to the speech recognition space;
[0065] Set the speech recognition task characters, map the speech recognition task characters to the speech recognition space through the text vector model, and construct text task representation information;
[0066] The modal adaptation feature information in the speech recognition space and the text task representation information are spliced to construct the speech information to be recognized;
[0067] The speech information to be recognized is recognized through a large language model to obtain the speech recognition result.
[0068] In the above embodiment, the acoustic encoder is used to extract features from the original speech signal to obtain speech feature information; the C-Adapter performs downsampling and one-dimensional convolution processing based on the speech feature information to obtain low-dimensional feature information, and the attention mechanism is adjusted on the low-dimensional feature information through the multi-head attention layer to obtain weighted feature information, perform normalization processing and feature enhancement, and generate modality adaptation feature information mapped to the speech recognition space;
[0069] Set the speech recognition task characters, map the speech recognition task characters to the speech recognition space through the text vector model, and construct text task representation information;
[0070] The modal adaptation feature information and text task representation information in the speech recognition space are spliced to construct the speech information to be recognized; the large language model recognizes the speech information to be recognized to obtain the speech recognition result.
[0071] The beneficial effects of the above technology are: the original speech signal is processed by the acoustic encoder and the C-Adapter adapter to realize the acquisition of the modal adaptation feature information of the speech modality, the speech recognition task characters are mapped to the speech recognition space by the text vector model, and the text task representation information of the text modality is generated; and the modal adaptation feature information and the text task representation information in the speech recognition space are spliced to construct the speech information to be recognized, and the fusion of multimodal information is realized, and the large language model is used for recognition to realize the acquisition of the speech recognition result; compared with the traditional technology, the above method realizes the recognition of the user's speech by the acoustic encoder, the C-Adapter adapter and the large language model, solves the defect of the traditional technology of obtaining the speech recognition result by feature analysis based on the acoustic model, extracts the speech features by the acoustic encoder and the C-Adapter adapter, generates the text task representation information by the text vector model, and performs modal fusion in the speech recognition space to generate the speech information to be recognized for processing based on the large language model; the above method realizes the combination of the speech recognition model and the large language model, and effectively improves the accuracy of speech recognition, improves the speed of speech recognition, and meets the requirements of real-time conversion of speech into text while improving the speech recognition efficiency.
[0072] Embodiment 2:
[0073] The embodiment of the present invention provides a multi-model speech recognition method based on a C-Adapter adapter, wherein the steps are: extracting features of an original speech signal through an acoustic encoder to obtain speech feature information; comprising:
[0074] Obtain a pre-trained acoustic encoder by constructing a convolutional neural network layer, a mask layer, a Transformer layer, a speech processing layer, and a clustering analysis layer;
[0075] Get the automatic speech recognition data in the training set, train the pre-trained acoustic encoder, and build the acoustic encoder; optional, including:
[0076] The convolutional neural network layer extracts features from the automatic speech recognition data to obtain hidden feature information of the samples;
[0077] The mask layer randomly masks the sample sub-feature information in the sample hidden feature information according to a preset mask rate to obtain the sample mask feature information;
[0078] The feature information of the sample mask is predicted through the Transformer layer to obtain the sample recovery feature information;
[0079] The speech processing layer preprocesses the automatic speech recognition data to obtain the preprocessed sample speech signal, and converts it to the frequency domain through Fourier transform for analysis to obtain the sample spectrum feature information; the Mel filter performs energy superposition operation on the sample spectrum feature information to obtain the Mel logarithmic feature information, and obtains the Mel cepstrum parameter information through discrete cosine transform for dynamic feature extraction to generate Mel cepstrum parameter feature information;
[0080] The clustering analysis layer performs clustering analysis based on the Mel-frequency cepstrum parameter feature information and constructs sample clustering codebook information;
[0081] Match the unmasked sub-information in the sample recovery feature information with the clustering sub-information in the sample clustering codebook information, obtain the masked clustering sub-information corresponding to the masked sub-information in the sample recovery feature information, and calculate the similarity distance;
[0082] Adjust the parameters of the Transformer layer to optimize the similarity distance, and use the pre-trained acoustic encoder as the acoustic encoder when the threshold condition is met.
[0083] In the above embodiments, a pre-trained acoustic encoder is obtained by constructing a convolutional neural network layer, a mask layer, a Transformer layer, a speech processing layer and a clustering analysis layer, and the acoustic encoder is obtained by training based on automatic speech recognition data.
[0084] In the above embodiments, the convolutional neural network layer extracts features from the automatic speech recognition data to obtain sample hidden feature information; the mask layer randomly masks the sample sub-feature information in the sample hidden feature information according to a preset masking rate to obtain sample mask feature information; and the Transformer layer predicts features on the sample mask feature information to obtain sample recovery feature information;
[0085] The speech processing layer preprocesses the automatic speech recognition data, obtains the preprocessed sample speech signal, and converts it to the frequency domain through Fourier transform for analysis to obtain the sample spectrum feature information; the Mel filter performs energy superposition operation on the sample spectrum feature information to obtain the Mel logarithmic feature information, obtains the Mel cepstrum parameter information through discrete cosine transform, performs dynamic feature extraction, and generates Mel cepstrum parameter feature information; the clustering analysis layer performs clustering analysis based on the Mel cepstrum parameter feature information to construct the sample clustering codebook information;
[0086] The unmasked sub-information in the sample recovery feature information is matched with the clustering sub-information in the sample clustering codebook information, the masked clustering sub-information corresponding to the masked sub-information in the sample recovery feature information is obtained, and the similarity distance is calculated; the similarity distance is optimized by adjusting the parameters of the Transformer layer.
[0087] In the above embodiments, the clustering analysis layer is built based on the K-means clustering algorithm.
[0088] In the above embodiments, the acoustic encoder may be implemented as a Hubert model.
[0089] In the above embodiments, for the initially constructed convolutional neural network layer, mask layer, Transformer layer, speech processing layer, and clustering analysis layer, unsupervised training is performed using the automatic speech recognition data in the training set. After the preset number of iterations, the sample recovery feature information output by the convolutional neural network layer, mask layer, and Transformer layer is similarly matched with the sample clustering codebook information, the parameters of the Transformer layer are optimized, and the feature extraction performance of the acoustic encoder is improved.
[0090] The beneficial effect of the above technology is that by constructing a pre-trained acoustic encoder, unsupervised training is performed based on automatic speech recognition data. While achieving fast training, the parameters of the Transformer layer are optimized based on the internal clustering analysis layer, the feature extraction performance of the pre-trained acoustic encoder is enhanced, and the acoustic encoder is obtained, so that it can realize feature extraction of the original speech signal.
[0091] Embodiment 3:
[0092] The embodiment of the present invention provides a multi-model speech recognition method based on a C-Adapter adapter, wherein the steps are: extracting features of an original speech signal through an acoustic encoder to obtain speech feature information; comprising:
[0093] The multi-layer temporal convolutional network in the convolutional neural network layer performs an expansion convolution operation on the original speech signal, and the residual connection between the temporal convolutional networks is used to obtain hidden feature information, and the hidden feature information is normalized by a normalization function, and the normalized hidden feature information is calculated by a GELU activation function to obtain hidden speech feature information;
[0094] The mask layer randomly masks the sub-feature information in the hidden speech feature information according to a preset masking rate to obtain masked feature information;
[0095] The encoder in the Transformer layer analyzes the global relationship of the mask feature information based on the mask feature information and the corresponding mask position, and the decoder predicts the sub-feature information of the mask to obtain the speech feature information.
[0096] In the above embodiments, the convolutional neural network layer performs an expansion convolution operation on the original speech signal through a multi-layer temporal convolutional network, which can realize the processing of original speech signals of variable length, and the setting of the normalization function avoids the gradient explosion in the processing process of the convolutional neural network layer, improves the processing stability, and can achieve faster convergence of hidden feature information through the GELU activation function, thereby obtaining hidden speech feature information.
[0097] In the above embodiments, the convolutional neural network layer of the acoustic encoder extracts hidden features of the original speech signal, generates hidden speech feature information, and randomly masks the sub-feature information in the hidden speech feature information according to a preset masking rate through the mask layer to obtain the masked feature information. The encoder in the Transformer layer analyzes the global relationship of the masked feature information based on the masked feature information and the corresponding mask position, and predicts the masked sub-feature information through the decoder to obtain the speech feature information.
[0098] The beneficial effect of the above technology is that through the convolutional neural network layer of the acoustic encoder, the hidden features of the original speech signal are extracted, and through the mask layer and the Transformer layer, feature masking and restoration are performed to obtain speech feature information.
[0099] Embodiment 4:
[0100] The embodiment of the present invention provides a multi-model speech recognition method based on a C-Adapter adapter, wherein the C-Adapter adapter performs downsampling and one-dimensional convolution processing according to speech feature information to obtain low-dimensional feature information, and performs attention mechanism adjustment on the low-dimensional feature information through a multi-head attention layer to obtain weighted feature information, perform normalization processing and feature enhancement, and generate modality adaptation feature information mapped to the speech recognition space; including:
[0101] Build a pre-trained adapter, connect the acoustic encoder, pre-trained adapter, speech recognition space and large language model in sequence;
[0102] Get the automatic speech recognition data in the training set;
[0103] The acoustic encoder extracts features from the automatic speech recognition data to obtain sample speech feature information;
[0104] The pre-trained adapter performs sample feature enhancement on the sample speech feature information, obtains the sample modal feature information, and maps it to the speech recognition space;
[0105] Concatenate the sample modal feature information in the speech recognition space with the sample training identification information to construct the sample speech recognition information;
[0106] The large language model performs recognition prediction based on the sample speech recognition information to obtain the sample recognition result;
[0107] Based on the sample recognition results and the sample standard recognition results corresponding to the automatic speech recognition data, the loss function is calculated to optimize the control parameters of the pre-trained adapter.
[0108] In the above embodiments, after constructing the pre-trained adapter, the acoustic encoder, the pre-trained adapter, the speech recognition space and the large language model are connected in sequence to build a model for training the pre-trained adapter.
[0109] In the above embodiment, the automatic speech recognition data is 960 hours of open source English librispeech speech recognition data.
[0110] In the above embodiments, feature extraction is performed on automatic speech recognition data through an acoustic encoder to obtain sample speech feature information; the pre-trained adapter performs sample feature enhancement on the sample speech feature information to obtain sample modal feature information and maps it to the speech recognition space; the sample modal feature information in the speech recognition space is concatenated with the sample training identification information to construct sample speech recognition information; the large language model performs recognition prediction based on the sample speech recognition information to obtain a sample recognition result; based on the sample recognition result and the sample standard recognition result corresponding to the automatic speech recognition data, the loss function is calculated to optimize the control parameters of the pre-trained adapter.
[0111] The beneficial effects of the above technology are: connecting the pre-trained adapter to the speech recognition architecture built by the method, realizing the training of the pre-trained adapter through the automatic speech recognition data in the training set, and improving the feature extraction performance of the pre-trained adapter by continuously optimizing the loss function of the sample recognition results and the sample standard recognition results, thereby obtaining the C-Adapter adapter.
[0112] Embodiment 5:
[0113] The embodiment of the present invention provides a multi-model speech recognition method of a C-Adapter adapter, comprising the steps of: the C-Adapter adapter performs downsampling and one-dimensional convolution processing according to speech feature information to obtain low-dimensional feature information, and performs attention mechanism adjustment on the low-dimensional feature information through a multi-head attention layer to obtain weighted feature information, performs normalization processing and feature enhancement, and generates modality adaptation feature information mapped to the speech recognition space; comprising:
[0114] Down-sampling the speech feature information through the sampling layer in the C-Adapter adapter to obtain a speech feature sampling signal;
[0115] Based on a one-dimensional convolutional neural network, the speech feature sampling signal is compressed to obtain low-dimensional feature information;
[0116] The attention value of the sub-feature information in the low-dimensional feature information is calculated through the multi-head attention layer, and weighted calculation is performed to obtain weighted feature information;
[0117] Performing residual connection according to the weighted feature information and the speech feature sampling signal, and processing through the first normalization layer to obtain normalized feature information;
[0118] The normalized feature information is fully connected through a feedforward neural network model to obtain enhanced speech feature information;
[0119] Residual connection is performed based on the normalized feature information and the enhanced speech feature information, and the information is processed through the second normalization layer to obtain the modal adaptation feature information and map it into the speech recognition space.
[0120] In the above embodiments, the speech feature information is downsampled through the sampling layer in the C-Adapter adapter to obtain a speech feature sampling signal; based on a one-dimensional convolutional neural network, the speech feature sampling signal is feature compressed to obtain low-dimensional feature information; the multi-head attention layer calculates the attention value of the sub-feature information in the low-dimensional feature information, and performs weighted calculation to obtain weighted feature information; residual connection is performed based on the weighted feature information and the speech feature sampling signal, and the first normalization layer is used to process the information to obtain normalized feature information; the feedforward neural network model performs a full connection transformation on the normalized feature information to obtain enhanced speech feature information; residual connection is performed based on the normalized feature information and the enhanced speech feature information, and the second normalization layer is used to process the information to obtain modal adaptation feature information, which is mapped to the speech recognition space.
[0121] The beneficial effects of the above technology are: through the sampling layer and one-dimensional convolutional neural network in the C-Adapter adapter, undersampling and feature compression of long-sequence speech feature information are achieved, the processing speed of speech feature information is improved, and the global long-term features of low-dimensional feature information are captured through the multi-head attention layer. After the first normalization layer, the feedforward neural network model and the second normalization layer are processed, the acquisition of modal adaptation feature information is achieved.
[0122] Embodiment 6:
[0123] The embodiment of the present invention provides a multi-model speech recognition method of a C-Adapter adapter, comprising the steps of: based on a one-dimensional convolutional neural network, performing feature compression on a speech feature sampling signal to obtain low-dimensional feature information; comprising:
[0124] Obtaining a query vector, a key vector and a value vector of a speech feature sampling signal;
[0125] Based on the one-dimensional convolutional neural network, the size, stride and padding of the convolution kernel are set to build a shared pooling layer;
[0126] The query vector, key vector and numerical vector are input into the shared pooling layer for feature compression, low-dimensional feature information is obtained, and then input into the multi-head attention layer;
[0127]
[0128] Among them, L is the dimension of the obtained low-dimensional feature information, n is the dimension of the feature vector matrix constructed according to the query vector, key vector and numerical vector, p represents the padding value, f represents the size of the convolution kernel, and s represents the step value. The floor symbol.
[0129] In the above embodiments, the query vector, key vector and value vector of the speech feature sampling signal are obtained, and feature compression is performed through a shared pooling layer to obtain low-dimensional feature information.
[0130] The beneficial effects of the above technology are: building a shared pooling layer based on a one-dimensional convolutional neural network, processing query vectors, key vectors and numerical vectors, and obtaining low-dimensional feature information to improve the processing efficiency of the multi-head attention layer on low-dimensional feature information, solving the inconvenience of needing to process query vectors, key vectors and numerical vectors separately in traditional technologies, while reducing the complexity of the adapter code, further simplifying the architecture and improving the processing efficiency of the C-Adapter adapter on speech feature information.
[0131] Embodiment 7:
[0132] The embodiment of the present invention provides a multi-model speech recognition method of a C-Adapter adapter, comprising the steps of: setting speech recognition task characters, mapping the speech recognition task characters to a speech recognition space through a text vector model, and constructing text task representation information; comprising:
[0133] The character for the speech recognition task is set to "transcript";
[0134] Build a text vector model based on the Embedding layer to map the characters of the speech recognition task to the speech recognition space.
[0135] In the above embodiment, "transcript" is set as the speech recognition task character, a text vector model is built based on the Embedding layer, and the speech recognition task character is mapped to the speech recognition space.
[0136] Embodiment 8:
[0137] The embodiment of the present invention provides a multi-model speech recognition method of a C-Adapter adapter, comprising the steps of: recognizing speech information to be recognized by using a large language model to obtain a speech recognition result; comprising:
[0138] The large language model extracts feature vectors based on the speech information to be recognized, obtains the speech query vector and the speech key-value pair vector, and caches the speech key-value pair vector to the key-value storage disk;
[0139] The self-decoding layer calculates the self-attention value of the voice query vector and the voice key-value pair vector through the self-attention sliding window, and performs weighted calculation based on the voice information to be recognized and the obtained self-attention value to generate weighted self-attention feature information. The first deep feedforward network performs nonlinear transformation to obtain self-decoding feature information.
[0140] According to the self-decoding feature information, the decoding query vector is extracted; the cross-attention value is calculated according to the decoding query vector and the voice key-value pair vector in the key-value storage disk through the cross-attention mechanism unit in the cross-decoding layer, and weighted calculation is performed according to the self-decoding feature information and the obtained cross-attention value to generate weighted cross-attention feature information. The second deep feedforward network performs classification prediction to obtain the speech recognition result.
[0141] In the above embodiments, the large language model adopts a "self-decoding layer + cross-decoding layer" structure to recognize the speech information to be recognized.
[0142] In the above embodiments, after acquiring the speech information to be recognized, feature vector extraction is performed to obtain a speech query vector and a speech key-value pair vector, and the self-decoding layer calculates the attention value according to the speech query vector and the speech key-value pair vector through the self-attention sliding window, performs weighted calculation, generates weighted self-attention feature information, and performs nonlinear transformation through the first deep feedforward network to obtain self-decoding feature information; according to the self-decoding feature information, the decoding query vector is extracted, and the cross-attention mechanism unit in the cross-decoding layer calculates according to the decoding query vector and the speech key-value pair vector in the key-value storage disk to obtain the cross-attention value, and performs weighted calculation to generate weighted cross-attention feature information, and performs classification prediction through the second deep feedforward network to obtain the speech recognition result.
[0143] In one embodiment, the large language model may be implemented by calling an existing speech recognition model.
[0144] The beneficial effects of the above technology are: through the self-decoding layer and the cross-decoding layer, the recognition of the speech information to be recognized is realized, and the speech recognition result is obtained; and the large language model adopts the above structure, while the self-decoding layer retains the autoregressive property, it captures a wider range of contextual information through the cross-decoding layer, thereby improving the recognition accuracy, and the cross-decoding layer uses the speech key-value pair vector cached in the key-value storage disk to process, which reduces the memory space occupied, effectively improves the running speed of the large language model, and thereby improves the efficiency of speech recognition.
[0145] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A multi-model speech recognition method based on C-Adapter, characterized in that: include: Obtaining the original speech signal; Extracting features from the original speech signal using an acoustic encoder to obtain speech feature information; The C-Adapter performs downsampling and one-dimensional convolution processing according to the speech feature information to obtain low-dimensional feature information, and adjusts the attention mechanism of the low-dimensional feature information through a multi-head attention layer to obtain weighted feature information, perform normalization processing and feature enhancement, and generate modality adaptation feature information mapped to the speech recognition space; Setting a speech recognition task character, mapping the speech recognition task character to a speech recognition space through a text vector model, and constructing text task representation information; splicing the modal adaptation feature information and the text task representation information in the speech recognition space to construct speech information to be recognized; The speech information to be recognized is recognized through a large language model to obtain a speech recognition result.
2. According to claim 1, a multi-model speech recognition method based on C-Adapter adapter is characterized in that: The step of extracting features from the original speech signal through an acoustic encoder to obtain speech feature information comprises: Obtain a pre-trained acoustic encoder by constructing a convolutional neural network layer, a mask layer, a Transformer layer, a speech processing layer, and a clustering analysis layer; Acquire automatic speech recognition data in a training set, train the pre-trained acoustic encoder, and construct an acoustic encoder; optionally, include: The convolutional neural network layer performs feature extraction on the automatic speech recognition data to obtain hidden feature information of the sample; The mask layer randomly masks the sample sub-feature information in the sample hidden feature information according to a preset mask rate to obtain sample mask feature information; Performing feature prediction on the sample mask feature information through a Transformer layer to obtain sample restoration feature information; The speech processing layer pre-processes the automatic speech recognition data to obtain the pre-processed sample speech signal, and converts it into the frequency domain through Fourier transform for analysis to obtain sample spectrum feature information; the Mel filter performs energy superposition operation on the sample spectrum feature information to obtain Mel logarithmic feature information, obtains Mel cepstrum parameter information through discrete cosine transform for dynamic feature extraction, and generates Mel cepstrum parameter feature information; The cluster analysis layer performs cluster analysis according to the Mel-frequency cepstrum parameter feature information to construct sample cluster codebook information; Matching the unmasked sub-information in the sample recovery feature information with the clustering sub-information in the sample clustering codebook information, obtaining the masked clustering sub-information corresponding to the masked sub-information in the sample recovery feature information, and calculating the similarity distance; The parameters of the Transformer layer are adjusted to optimize the similarity distance, and when a threshold condition is met, the pre-trained acoustic encoder is used as the acoustic encoder.
3. The multi-model speech recognition method based on C-Adapter according to claim 2, characterized in that: The step of extracting features from the original speech signal through an acoustic encoder to obtain speech feature information comprises: The multi-layer temporal convolutional network in the convolutional neural network layer performs an expansion convolution operation on the original speech signal, and the residual connection between the temporal convolutional networks is used to obtain hidden feature information, the hidden feature information is normalized by a normalization function, and the normalized hidden feature information is calculated by a GELU activation function to obtain hidden speech feature information; The masking layer randomly masks the sub-feature information in the hidden speech feature information according to a preset masking rate to obtain masked feature information; The encoder in the Transformer layer analyzes the global relationship of the mask feature information according to the mask feature information and the corresponding mask position, and predicts the sub-feature information of the mask through the decoder to obtain the speech feature information.
4. The multi-model speech recognition method based on C-Adapter according to claim 1, characterized in that: The steps include: the C-Adapter performs downsampling and one-dimensional convolution processing according to the speech feature information to obtain low-dimensional feature information, and performs attention mechanism adjustment on the low-dimensional feature information through a multi-head attention layer to obtain weighted feature information, perform normalization processing and feature enhancement, and generate modality adaptation feature information mapped to the speech recognition space; including: Build a pre-trained adapter, connect the acoustic encoder, pre-trained adapter, speech recognition space and large language model in sequence; Get the automatic speech recognition data in the training set; The acoustic encoder extracts features from the automatic speech recognition data to obtain sample speech feature information; The pre-training adapter performs sample feature enhancement on the sample speech feature information, obtains sample modal feature information, and maps it to the speech recognition space; Concatenating the sample modal feature information in the speech recognition space with the sample training identification information to construct sample speech recognition information; The large language model performs recognition prediction based on the sample speech recognition information to obtain a sample recognition result; A loss function is calculated based on the sample recognition result and the sample standard recognition result corresponding to the automatic speech recognition data to optimize the control parameters of the pre-trained adapter.
5. The multi-model speech recognition method of a C-Adapter according to claim 4, characterized in that: The steps include: the C-Adapter performs downsampling and one-dimensional convolution processing according to the speech feature information to obtain low-dimensional feature information, and performs attention mechanism adjustment on the low-dimensional feature information through a multi-head attention layer to obtain weighted feature information, perform normalization processing and feature enhancement, and generate modality adaptation feature information mapped to the speech recognition space; including: Down-sampling the speech feature information through a sampling layer in the C-Adapter adapter to obtain a speech feature sampling signal; Based on a one-dimensional convolutional neural network, feature compression is performed on the speech feature sampling signal to obtain low-dimensional feature information; Calculating the attention value of the sub-feature information in the low-dimensional feature information through a multi-head attention layer, and performing weighted calculation to obtain weighted feature information; Performing a residual connection according to the weighted feature information and the speech feature sampling signal, and processing through a first normalization layer to obtain normalized feature information; Performing a fully connected transformation on the normalized feature information through a feedforward neural network model to obtain enhanced speech feature information; A residual connection is performed based on the normalized feature information and the enhanced speech feature information, and the information is processed through a second normalization layer to obtain modal adaptation feature information, which is mapped into the speech recognition space.
6. The multi-model speech recognition method of a C-Adapter according to claim 5, characterized in that: The step of: performing feature compression on the speech feature sampling signal based on a one-dimensional convolutional neural network to obtain low-dimensional feature information; comprises: Obtaining a query vector, a key vector and a value vector of the speech feature sampling signal; Based on the one-dimensional convolutional neural network, the size, stride and padding of the convolution kernel are set to build a shared pooling layer; Input the query vector, key vector and numerical vector into the shared pooling layer for feature compression, obtain low-dimensional feature information, and input it into the multi-head attention layer; Wherein, L is the dimension of the obtained low-dimensional feature information, n is the dimension of the feature vector matrix constructed according to the query vector, the key vector and the numerical vector, p represents the padding value, f represents the size of the convolution kernel, s represents the step value, The floor symbol.
7. The multi-model speech recognition method of a C-Adapter according to claim 1, characterized in that: The step of setting a speech recognition task character, mapping the speech recognition task character to a speech recognition space through a text vector model, and constructing text task representation information comprises: The speech recognition task character is set to "transcript"; A text vector model is built based on the Embedding layer to map the characters of the speech recognition task to the speech recognition space.
8. The multi-model speech recognition method of a C-Adapter according to claim 1, characterized in that: The step of recognizing the speech information to be recognized by using a large language model to obtain a speech recognition result comprises: The large language model extracts feature vectors according to the speech information to be recognized, obtains a speech query vector and a speech key-value pair vector, and caches the speech key-value pair vector in a key-value storage disk; The self-decoding layer calculates the self-attention value of the voice query vector and the voice key-value pair vector through a self-attention sliding window, and performs weighted calculation according to the voice information to be recognized and the obtained self-attention value to generate weighted self-attention feature information. The first deep feedforward network performs nonlinear transformation to obtain self-decoding feature information; According to the self-decoding feature information, a decoding query vector is extracted; a cross-attention value is calculated according to the decoding query vector and the speech key-value pair vector in the key-value storage disk through the cross-attention mechanism unit in the cross-decoding layer, and a weighted calculation is performed according to the self-decoding feature information and the obtained cross-attention value to generate weighted cross-attention feature information, and a second deep feedforward network performs classification prediction to obtain a speech recognition result.
Citation Information
Cited By
Sub-graph detection method and device and electronic equipment
CN120318531A
Subgraph detection method, apparatus and electronic device
CN120318531B
Speech recognition method, device, equipment, computer readable medium and program product
CN121331108A
