Speech recognition method, device and equipment based on end-to-end cross-language large model

Abstract

The invention relates to a speech recognition method, device and equipment based on an end-to-end cross-language large model. The method comprises the following steps: constructing a cross-language large model; obtaining a training data set formed by multiple groups of voice text label pairs, and inputting the training data set into the cross-language large model for end-to-end training until a trained cross-language large model is obtained; a to-be-recognized voice signal is input into the trained cross-language model, and filter feature and wav2vec2 feature extraction and splicing, text embedded feature representation, dimension alignment, splicing and decoding translation are carried out through a voice coding module, a text coding module, a feature alignment module and a decoding module in the model in sequence. And outputting and obtaining the speech recognition text of the target language. By adopting the method, the accuracy and robustness of speech recognition can be improved, and cross-language speech recognition is realized.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a speech recognition method, apparatus and device based on an end-to-end cross-language large model. Background Art

[0002] Speech recognition technology is the process of automatically converting speech signals into text. As one of the important technologies for human-machine interaction, it is widely used in the fields of intelligent assistants and voice control. The speech recognition process generally includes the following steps:

[0003] (1) Voice signal acquisition: Voice signals are input through a microphone, the sound waves are converted into voltage signals, and then sampled through an A / D (analog-to-digital conversion) device, thereby converting the continuous voltage signals into digital signals that can be processed by computers.

[0004] (2) Feature extraction: Extract key features from the collected speech signal, such as acoustic features, time features, sequence features, etc. These features are the basis for subsequent speech recognition model analysis and processing and can reflect the main information in the speech signal.

[0005] (3) Model training: The core of the speech recognition system lies in model training. The traditional two-stage model includes an acoustic model and a language model. The acoustic model matches the extracted acoustic features with the physical features of the speech signal to score each acoustic unit. The language model is based on linguistic theory and predicts the probability of different words or phrase sequences to help understand and generate text that fits the context. Model training usually uses large-scale data sets to improve the recognition accuracy and generalization ability of the model through pre-training models or statistical methods.

[0006] (4) Decoding: The decoding stage generates the most likely text output by combining the results of the acoustic model and the language model, ultimately completing the conversion from speech to text.

[0007] With the development of deep learning technology, acoustic models and language models have gradually changed from traditional hidden Markov model series to deep learning models based on large model neural networks. Acoustic and language models based on deep learning have shown significant performance improvements when processing complex speech signals. However, due to the architectural design of the two-stage model, there is a feature mismatch problem between the acoustic model and the language model, which limits the upper limit of the model's recognition.

[0008] To solve this problem, end-to-end speech recognition models such as Paraformer have emerged in recent years. Unlike traditional two-stage models, end-to-end models directly map input speech signals to output text, reducing the error accumulation caused by intermediate steps. However, such models still face certain challenges when dealing with scenarios such as new words, hot words, and cross-language recognition. In order to improve its performance in these areas, it is often necessary to introduce additional language models for auxiliary decoding.

[0009] Therefore, although speech recognition technology has made significant progress driven by deep learning, there is still much room for improvement in accuracy, generalization ability, and cross-language adaptability. Summary of the invention

[0010] Based on this, it is necessary to provide a speech recognition method, device and equipment based on an end-to-end cross-language large model to address the above technical problems.

[0011] A speech recognition method based on an end-to-end cross-language large model, the method comprising:

[0012] Build a large cross-language model, which consists of a speech encoding module, a text encoding module, a feature alignment module, and a decoding module;

[0013] A training data set consisting of multiple groups of speech-text label pairs is obtained, and the training data set is input into a cross-language large model for end-to-end training until a trained cross-language large model is obtained; wherein, in each group of speech-text label pairs, a cross-language recognition prompt word is spliced ​​at the front end of the text label, and the cross-language recognition prompt word is used to prompt the model to translate the text label into text in the target language during the training process;

[0014] The speech signal to be recognized is input into the trained cross-language model, and the speech encoding module is used to extract and concatenate filter features and wav2vec2 features of the input speech signal to obtain a speech feature vector; the text encoding module is used to represent the text label corresponding to the speech signal according to the cross-language recognition prompt word, and a text embedding feature vector is obtained; the feature alignment module is used to align the dimension of the speech feature vector with the dimension of the text embedding feature vector; the decoding module is used to concatenate and decode the text embedding feature vector output by the text encoding module and the speech feature vector output by the feature alignment module after the dimension is aligned, and the speech recognition text in the target language is output.

[0015] In one embodiment, the voice coding module extracts and concatenates filter features and wav2vec2 features of the input voice signal to obtain a voice feature vector, including:

[0016] The speech coding module consists of a filter feature extraction module, a wav2vec2 pre-training model, and a feature concatenation module;

[0017] The filter feature extraction module is used to perform pre-emphasis, framing, windowing, fast Fourier transform, Mel spectrum filtering and logarithmic operation on the input speech signal in sequence, and output the filter feature;

[0018] The wav2vec2 pre-trained model is used to perform convolutional coding and context representation on the input speech signal in sequence, and output wav2vec2 features;

[0019] The feature concatenation module is used to concatenate the filter features and wav2vec2 features in feature dimensions and output a speech feature vector.

[0020] In one embodiment, the wav2vec2 pre-trained model consists of an encoder network, a context network, and a discretization network;

[0021] The encoder network extracts features from the input speech signal through several convolutional layers and outputs convolutional features; the context network represents the context of the convolutional features through several time step encoders and outputs wav2vec2 features; the discretization network discretizes the convolutional features, calculates the loss based on the discretized convolutional features and the wav2vec2 features, and trains the wav2vec2 pre-trained model based on the loss.

[0022] In one embodiment, the filter feature and the wav2vec2 feature are concatenated in feature dimension, and the output is a speech feature vector, including:

[0023] Through a single convolution layer, the filter features are convolved in a feature dimension according to the set convolution step size to obtain the transformed filter features;

[0024] Based on the feature dimension of the transformed filter feature, the neighboring points of the wav2vec2 feature are fused with a set step size to obtain the transformed wav2vec2 feature, and the feature dimension of the transformed filter feature is kept consistent with that of the transformed wav2vec2 feature;

[0025] The transformed filter features are concatenated with the transformed wav2vec2 features to output the speech feature vector.

[0026] In one of the embodiments, the feature alignment module uses a multi-layer perceptron to map the dimension of the speech feature vector to the dimension of the text embedding feature vector, keeping the dimensions of the two consistent.

[0027] In one of the embodiments, after the decoding module performs feature vector splicing on the text embedding feature vector output by the text encoding module and the dimensionally aligned speech feature vector output by the feature alignment module, the obtained spliced ​​feature vector is input into the large language model for decoding and translation, and the speech recognition text in the target language is output.

[0028] A speech recognition device based on an end-to-end cross-language large model, the device comprising:

[0029] A model building unit is used to build a large cross-language model. The large cross-language model consists of a speech encoding module, a text encoding module, a feature alignment module, and a decoding module.

[0030] The model training unit is used to obtain a training data set consisting of multiple groups of speech-text label pairs, and input the training data set into the cross-language large model for end-to-end training until a trained cross-language large model is obtained; wherein, in each group of speech-text label pairs, a cross-language recognition prompt word is spliced ​​at the front end of the text label, and the cross-language recognition prompt word is used to prompt the model to translate the text label into text in the target language during the training process;

[0031] The speech recognition unit is used to input the speech signal to be recognized into the trained cross-language model, and perform filter feature and wav2vec2 feature extraction and splicing on the input speech signal through the speech encoding module to obtain a speech feature vector; perform text embedding feature representation on the text label corresponding to the speech signal according to the cross-language recognition prompt word through the text encoding module to obtain a text embedding feature vector; align the dimension of the speech feature vector with the dimension of the text embedding feature vector through the feature alignment module; perform feature vector splicing and decoding translation on the text embedding feature vector output by the text encoding module and the speech feature vector after dimension alignment output by the feature alignment module through the decoding module, and output the speech recognition text in the target language.

[0032] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0033] Build a large cross-language model, which consists of a speech encoding module, a text encoding module, a feature alignment module, and a decoding module;

[0034] A training data set consisting of multiple groups of speech-text label pairs is obtained, and the training data set is input into a cross-language large model for end-to-end training until a trained cross-language large model is obtained; wherein, in each group of speech-text label pairs, a cross-language recognition prompt word is spliced ​​at the front end of the text label, and the cross-language recognition prompt word is used to prompt the model to translate the text label into text in the target language during the training process;

[0035] The speech signal to be recognized is input into the trained cross-language model, and the speech encoding module is used to extract and concatenate filter features and wav2vec2 features of the input speech signal to obtain a speech feature vector; the text encoding module is used to represent the text label corresponding to the speech signal according to the cross-language recognition prompt word, and a text embedding feature vector is obtained; the feature alignment module is used to align the dimension of the speech feature vector with the dimension of the text embedding feature vector; the decoding module is used to concatenate and decode the text embedding feature vector output by the text encoding module and the speech feature vector output by the feature alignment module after the dimension is aligned, and the speech recognition text in the target language is output.

[0036] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0037] Build a large cross-language model, which consists of a speech encoding module, a text encoding module, a feature alignment module, and a decoding module;

[0038] A training data set consisting of multiple groups of speech-text label pairs is obtained, and the training data set is input into a cross-language large model for end-to-end training until a trained cross-language large model is obtained; wherein, in each group of speech-text label pairs, a cross-language recognition prompt word is spliced ​​at the front end of the text label, and the cross-language recognition prompt word is used to prompt the model to translate the text label into text in the target language during the training process;

[0039] The speech signal to be recognized is input into the trained cross-language model, and the speech encoding module is used to extract and concatenate filter features and wav2vec2 features of the input speech signal to obtain a speech feature vector; the text encoding module is used to represent the text label corresponding to the speech signal according to the cross-language recognition prompt word, and a text embedding feature vector is obtained; the feature alignment module is used to align the dimension of the speech feature vector with the dimension of the text embedding feature vector; the decoding module is used to concatenate and decode the text embedding feature vector output by the text encoding module and the speech feature vector output by the feature alignment module after the dimension is aligned, and the speech recognition text in the target language is output.

[0040] Compared with the prior art, the above-mentioned speech recognition method, device and apparatus based on end-to-end cross-language large model have the following technical effects:

[0041] 1. This application introduces cross-language recognition prompt words to be spliced ​​into the front end of the text label. Based on the cross-language recognition prompt words, the model can be prompted to translate the text label into the text of the target language during the model training process, so that the constructed cross-language large model has the ability of multi-language recognition, that is, it can recognize the audio of one language as the text of another language. In addition, the cross-language recognition prompt words will affect the parameter learning of each layer of the cross-language large model, but will not add additional learning parameters, simplifying the design and training process of the model.

[0042] 2. This application uses a method of jointly extracting filter features and wav2vec2 features to extract speech features. The combined extraction of the two methods can obtain more comprehensive and diverse speech features, which can better meet the recognition task requirements of different languages. And by splicing the feature dimensions of the two speech features, combining the basic acoustic characteristics in the filter features and the advanced semantic characteristics in the wav2vec2 features, the model can use more information for learning during the training process, without the need for iterative updates of the language model and auxiliary work such as hot word and new word discovery, which improves the ability of speech feature representation and significantly improves the accuracy and robustness of speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flowchart of a speech recognition method based on an end-to-end cross-language large model in one embodiment;

[0044] Figure 2 is a structural diagram of a speech coding module in one embodiment;

[0045] Figure 3 A schematic diagram of a Mel filter bank in one embodiment;

[0046] Figure 4 A schematic diagram of the structure of a wav2vec2 pre-training model in one embodiment;

[0047] Figure 5 A schematic diagram of performing speech recognition based on a trained cross-language model in one embodiment;

[0048] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] In one embodiment, Figure 1As shown, a speech recognition method based on an end-to-end cross-language large model is provided, comprising the following steps:

[0051] Step S1, constructing a large cross-language model, which consists of a speech encoding module, a text encoding module, a feature alignment module and a decoding module.

[0052] like Figure 2 As shown, the speech coding module consists of a filter feature extraction module, a wav2vec2 pre-training model and a feature splicing module, which is mainly used to represent the input speech signal with a speech feature vector.

[0053] The filter feature extraction module is used to perform pre-emphasis, framing, windowing, fast Fourier transform, Mel spectrum filtering and logarithmic operation on the input speech signal in sequence, and output the filter feature. Specifically:

[0054] Pre-emphasis: used to compensate for the loss of high-frequency components of speech signals and enhance the high-frequency components of signals.

[0055] Framing: It is used to make the speech signal stable in a short time. The speech is framed in a short time to meet the stable condition of FFT (Fast Fourier Transform).

[0056] Windowing: After the speech is framed, it is necessary to multiply the window function in the time domain to increase the continuity of the left and right ends of the frame, reduce spectrum leakage, and prevent the two ends of the window from changing sharply but transitioning smoothly to 0, so that the cut speech waveform frame slowly drops to 0, reducing the truncation effect of the speech frame.

[0057] FFT: used to perform Fourier transform on the windowed speech signal to obtain the frequency domain distribution of the speech signal.

[0058] Mel spectrum filtering: The spectrum characteristics obtained by FFT are linearly distributed, which does not conform to the human ear's perception of real speech signals. The human ear is more sensitive to low-frequency sounds and will focus more on certain specific frequency points, which is similar to a filter bank, namely the Mel filter bank. Figure 3 As shown in the figure, it uses logarithmic transformation to map the linear spectrum obtained by fast Fourier transform to the Mel nonlinear spectrum based on auditory perception. After the Mel filter bank is constructed, to obtain the Mel cepstrum energy, it is also necessary to perform Mel filtering on the frequency spectrum after fast Fourier transform to calculate the energy and total energy value of each order filter. At the same time, since the human ear's perception of volume is not linear, it is also necessary to take the logarithm of the energy to obtain the final output filter feature.

[0059] The wav2vec2 pre-trained model is used to perform convolutional coding and context representation on the input speech signal in sequence, and output wav2vec2 features. The structure of the wav2vec2 pre-trained model used in this application is as follows: Figure 4 As shown, it consists of an encoder network, a context network, and a discretization network. Specifically, Figure 4 The encoder network at the bottom extracts features from the input speech signal through 7 convolutional layers, and outputs the convolutional features; the context network on the left represents the context of the convolutional features through several time step encoders (Transformer), and outputs the wav2vec2 features; the discretization network on the right discretizes the convolutional features, calculates the loss based on the discretized convolutional features and the wav2vec2 features, and trains the wav2vec2 pre-training model based on the loss. Among them, the discretization network only participates in the calculation during model training. After the training is completed, it is only necessary to obtain the wav2vec2 features based on the encoder network and the context network output. Furthermore, in order to better represent the speech features of different languages, the wav2vec2 pre-training model used in this application adopts a model version that has been pre-trained with multilingual corpora, which will play a role in the following feature alignment module.

[0060] This application adopts the method of jointly extracting filter features and wav2vec2 features to extract speech features. The combined extraction of the two methods can obtain more comprehensive and diverse speech features, which can better adapt to the recognition task requirements of different languages.

[0061] Among them, the feature concatenation module is used to concatenate the filter features and wav2vec2 features in feature dimensions, and output a speech feature vector. Specifically, the filter features are first convolved in feature dimensions through a single convolution layer according to a set convolution step size to obtain the transformed filter features; then, based on the feature dimensions of the transformed filter features, the wav2vec2 features are fused with neighboring points of a set step size to obtain the transformed wav2vec2 features, and the feature dimensions of the transformed filter features and the transformed wav2vec2 features are kept consistent; finally, the transformed filter features are concatenated with the transformed wav2vec2 features to output a speech feature vector.

[0062] Specifically, for a 1-second speech signal with a sampling rate of 16k, assuming that the number of Mel-triangle filter groups is 26, the length of the analysis window winlen is measured in seconds, which is 0.01s (10ms), and the FFT size nfft is 512 by default. The resulting filter feature dimension is (100, 26), which can be understood as compressing the 1-second speech signal into 100 feature points, and the feature dimension of each point is 26, that is, each point integrates about 1000 / 100 about 10ms of features.

[0063] After obtaining the filter features, the feature dimension convolution transformation is first calculated. The convolution layer is designed as a single-layer convolution structure with a convolution kernel size of (7,2) and a convolution step size of 3. For an input feature dimension of (100,26), the output feature dimension is (32,9), that is, each point fuses features of about 30ms.

[0064] Furthermore, for a 1-second speech signal with a sampling rate of 16000, the final wav2vec2 feature dimension is (256, 256). In order to keep consistent with the feature dimension of the above filter feature after CNN transformation, a mean calculation is performed on the 8 adjacent points (essentially a convolution operation with a kernel of 8*1), and the final feature dimension is (256 / 8, 256) = (32, 256).

[0065] Finally, the filter features with a feature dimension of (32,9) and the wav2vec2 features with a feature dimension of (32,256) are concatenated, and the feature dimension becomes (32,265).

[0066] By concatenating the feature dimensions of the two speech features, combining the basic acoustic characteristics of the filter features and the advanced semantic characteristics of the wav2vec2 features, the model can use more information for learning during the training process, without the need for iterative updates of the language model and auxiliary work such as hot word and new word discovery, thereby improving the ability to represent speech features and significantly improving the accuracy and robustness of speech recognition.

[0067] Among them, the feature alignment module uses a multi-layer perceptron to map the dimension of the speech feature vector to the dimension of the text embedding feature vector, keeping the dimensions of the two consistent.

[0068] Among them, after the decoding module concatenates the text embedding feature vector output by the text encoding module and the dimensionally aligned speech feature vector output by the feature alignment module, the concatenated feature vector is input into the large language model for decoding and translation, and the output text is required to be the text of the language required to be translated by the cross-language recognition prompt word.

[0069] Step S2, obtaining a training data set consisting of multiple groups of speech-text label pairs, and inputting the training data set into a cross-language large model for end-to-end training until a trained cross-language large model is obtained; wherein, in each group of speech-text label pairs, a cross-language recognition prompt word is spliced ​​at the front end of the text label, and the cross-language recognition prompt word is used to prompt the model to translate the text label into text in the target language during the training process.

[0070] Specifically, examples of speech-text label pairs are as follows:

[0071] {"audio":"1.mp3","text":"Please identify the audio as [English] text label: I love playing the guitar and piano!"}

[0072] {"audio":"1.mp3","text":"Please identify the audio as [Chinese] Text label: I like playing guitar and piano!"}

[0073] Compared with traditional speech recognition training data sets, this application improves the text label part of the speech-text label pair, and adds a cross-language recognition prompt word at the front end. The cross-language recognition prompt word can be any text with similar meanings such as "Please recognize the audio as [languagecode] text", and translate the original text label into the text of the corresponding languagecode. The main purpose of this operation is to pass this instruction to the cross-language large model so that the cross-language large model can follow the instruction to learn. It should be noted that the cross-language recognition prompt word is in text form and is only input as a text embedding at the model input layer. It is not a learnable vector embedded in the high-level structure of the model (increasing the amount of training parameters). In fact, the cross-language recognition prompt word embedding method of this application will affect the parameter learning of each layer of the large model, but will not add additional learning parameters. While enabling the constructed cross-language large model to have the ability of multi-language recognition, it simplifies the design and training process of the model.

[0074] Furthermore, the overall architecture of the cross-language large model constructed by this application is consistent with general large language models such as llama, GPT, etc., so the model training method is also the same as other large language models. And compared with the previous speech synthesis model training, the training method of this application is completely end-to-end, that is, from audio input to recognized text output, only one large model is needed.

[0075] Step S3, input the speech signal to be recognized into the trained cross-language model, and perform filter feature and wav2vec2 feature extraction and splicing on the input speech signal through the speech encoding module to obtain a speech feature vector; perform text embedding feature representation on the text label corresponding to the speech signal according to the cross-language recognition prompt word through the text encoding module to obtain a text embedding feature vector; align the dimension of the speech feature vector with the dimension of the text embedding feature vector through the feature alignment module; perform feature vector splicing and decoding translation on the text embedding feature vector output by the text encoding module and the speech feature vector output by the feature alignment module after dimension alignment through the decoding module, and output the speech recognition text in the target language. The process of speech recognition with the trained cross-language model is as follows: Figure 5 shown.

[0076] In one embodiment, a speech recognition device based on an end-to-end cross-language large model is provided, comprising:

[0077] A model building unit is used to build a large cross-language model. The large cross-language model consists of a speech encoding module, a text encoding module, a feature alignment module, and a decoding module.

[0078] The model training unit is used to obtain a training data set consisting of multiple groups of speech-text label pairs, and input the training data set into the cross-language large model for end-to-end training until a trained cross-language large model is obtained; wherein, in each group of speech-text label pairs, a cross-language recognition prompt word is spliced ​​at the front end of the text label, and the cross-language recognition prompt word is used to prompt the model to translate the text label into text in the target language during the training process;

[0079] The speech recognition unit is used to input the speech signal to be recognized into the trained cross-language model, and perform filter feature and wav2vec2 feature extraction and splicing on the input speech signal through the speech encoding module to obtain a speech feature vector; perform text embedding feature representation on the text label corresponding to the speech signal according to the cross-language recognition prompt word through the text encoding module to obtain a text embedding feature vector; align the dimension of the speech feature vector with the dimension of the text embedding feature vector through the feature alignment module; perform feature vector splicing and decoding translation on the text embedding feature vector output by the text encoding module and the speech feature vector after dimension alignment output by the feature alignment module through the decoding module, and output the speech recognition text in the target language.

[0080] For the specific limitations of the speech recognition device based on the end-to-end cross-language large model, please refer to the limitations of the speech recognition method based on the end-to-end cross-language large model above, which will not be repeated here. Each module in the above-mentioned speech recognition device based on the end-to-end cross-language large model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0081] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a speech recognition method based on an end-to-end cross-language large model is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0082] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0083] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0084] Build a large cross-language model, which consists of a speech encoding module, a text encoding module, a feature alignment module, and a decoding module;

[0085] A training data set consisting of multiple groups of speech-text label pairs is obtained, and the training data set is input into a cross-language large model for end-to-end training until a trained cross-language large model is obtained; wherein, in each group of speech-text label pairs, a cross-language recognition prompt word is spliced ​​at the front end of the text label, and the cross-language recognition prompt word is used to prompt the model to translate the text label into text in the target language during the training process;

[0086] The speech signal to be recognized is input into the trained cross-language model, and the speech encoding module is used to extract and concatenate filter features and wav2vec2 features of the input speech signal to obtain a speech feature vector; the text encoding module is used to represent the text label corresponding to the speech signal according to the cross-language recognition prompt word, and a text embedding feature vector is obtained; the feature alignment module is used to align the dimension of the speech feature vector with the dimension of the text embedding feature vector; the decoding module is used to concatenate and decode the text embedding feature vector output by the text encoding module and the speech feature vector output by the feature alignment module after the dimension is aligned, and the speech recognition text in the target language is output.

[0087] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0088] Build a large cross-language model, which consists of a speech encoding module, a text encoding module, a feature alignment module, and a decoding module;

[0089] A training data set consisting of multiple groups of speech-text label pairs is obtained, and the training data set is input into a cross-language large model for end-to-end training until a trained cross-language large model is obtained; wherein, in each group of speech-text label pairs, a cross-language recognition prompt word is spliced ​​at the front end of the text label, and the cross-language recognition prompt word is used to prompt the model to translate the text label into text in the target language during the training process;

[0090] The speech signal to be recognized is input into the trained cross-language model, and the speech encoding module is used to extract and concatenate filter features and wav2vec2 features of the input speech signal to obtain a speech feature vector; the text encoding module is used to represent the text label corresponding to the speech signal according to the cross-language recognition prompt word, and a text embedding feature vector is obtained; the feature alignment module is used to align the dimension of the speech feature vector with the dimension of the text embedding feature vector; the decoding module is used to concatenate and decode the text embedding feature vector output by the text encoding module and the speech feature vector output by the feature alignment module after the dimension is aligned, and the speech recognition text in the target language is output.

[0091] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0092] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A speech recognition method based on an end-to-end cross-language large model, characterized in that: The method comprises: Constructing a large cross-language model, wherein the large cross-language model consists of a speech encoding module, a text encoding module, a feature alignment module, and a decoding module; A training data set consisting of multiple groups of speech-text label pairs is obtained, and the training data set is input into a cross-language large model for end-to-end training until a trained cross-language large model is obtained; wherein, in each group of speech-text label pairs, a cross-language recognition prompt word is spliced ​​at the front end of the text label, and the cross-language recognition prompt word is used to prompt the model to translate the text label into text in the target language during the training process; The speech signal to be recognized is input into the trained cross-language model, and the filter features and wav2vec2 features are extracted and spliced ​​on the input speech signal through the speech encoding module to obtain a speech feature vector; the text label corresponding to the speech signal is represented by text embedding features according to the cross-language recognition prompt word through the text encoding module to obtain a text embedding feature vector; the dimension of the speech feature vector is aligned with the dimension of the text embedding feature vector through the feature alignment module; the text embedding feature vector output by the text encoding module and the speech feature vector after the dimension is aligned output by the feature alignment module are spliced ​​and decoded and translated through the decoding module to output the speech recognition text in the target language.

2. The method according to claim 1, characterized in that The speech encoding module extracts and concatenates filter features and wav2vec2 features of the input speech signal to obtain a speech feature vector, including: The speech coding module is composed of a filter feature extraction module, a wav2vec2 pre-training model and a feature splicing module; The filter feature extraction module is used to perform pre-emphasis, framing, windowing, fast Fourier transform, Mel spectrum filtering and logarithmic operation processing on the input speech signal in sequence, and output the filter feature; The wav2vec2 pre-trained model is used to perform convolution coding and context representation on the input speech signal in sequence, and output wav2vec2 features; The feature concatenation module is used to concatenate the filter features and the wav2vec2 features in terms of feature dimensions, and output a speech feature vector.

3. The method according to claim 2, characterized in that The wav2vec2 pre-training model consists of an encoder network, a context network, and a discretization network; The encoder network extracts features of the input speech signal through several convolutional layers and outputs convolutional features; the context network represents the context of the convolutional features through several time step encoders and outputs wav2vec2 features; the discretization network discretizes the convolutional features, calculates the loss based on the discretized convolutional features and the wav2vec2 features, and trains the wav2vec2 pre-training model based on the loss.

4. The method according to claim 2, characterized in that: The filter features and wav2vec2 features are concatenated in feature dimensions to output a speech feature vector, including: Performing a feature dimension convolution transformation on the filter feature according to a set convolution step size through a single convolution layer to obtain a transformed filter feature; Based on the feature dimension of the transformed filter feature, the wav2vec2 feature is subjected to neighboring point fusion with a set step size to obtain the transformed wav2vec2 feature, and the feature dimension of the transformed filter feature is kept consistent with that of the transformed wav2vec2 feature; The transformed filter features and the transformed wav2vec2 features are concatenated and output to obtain a speech feature vector.

5. The method according to claim 1, characterized in that The feature alignment module uses a multi-layer perceptron to map the dimension of the speech feature vector to the dimension of the text embedding feature vector, keeping the dimensions of the two consistent.

6. The method according to claim 1, characterized in that After the decoding module performs feature vector splicing on the text embedding feature vector output by the text encoding module and the dimensionally aligned speech feature vector output by the feature alignment module, the obtained spliced ​​feature vector is input into the large language model for decoding and translation, and the speech recognition text in the target language is output.

7. A speech recognition device based on an end-to-end cross-language large model, characterized in that: The device comprises: A model building unit, used to build a large cross-language model, wherein the large cross-language model consists of a speech encoding module, a text encoding module, a feature alignment module, and a decoding module; The model training unit is used to obtain a training data set consisting of multiple groups of speech-text label pairs, and input the training data set into the cross-language large model for end-to-end training until a trained cross-language large model is obtained; wherein, in each group of speech-text label pairs, a cross-language recognition prompt word is spliced ​​at the front end of the text label, and the cross-language recognition prompt word is used to prompt the model to translate the text label into text in the target language during the training process; The speech recognition unit is used to input the speech signal to be recognized into the trained cross-language model, and perform filter feature extraction and splicing on the input speech signal through the speech encoding module to obtain a speech feature vector; perform text embedding feature representation on the text label corresponding to the speech signal according to the cross-language recognition prompt word through the text encoding module to obtain a text embedding feature vector; align the dimension of the speech feature vector with the dimension of the text embedding feature vector through the feature alignment module; perform feature vector splicing and decoding translation on the text embedding feature vector output by the text encoding module and the speech feature vector after dimension alignment output by the feature alignment module through the decoding module, and output the speech recognition text in the target language.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images