A multi-encoder based speech recognition method

Abstract

The application discloses a kind of based on multi-encoder's call voice recognition method, first to call voice according to sound channel information speaker classification;The classified voice is divided according to speaking time and blank information and is sorted;The preprocessed voice is sorted, and is converted into 80-dimensional Fbank feature;Fbank feature is spliced according to the previous sentence, the source of recognition sentence and the next sentence as context information;The spliced voice and target sentence are respectively input into two different encoders, and the context information is fused according to different ways, to obtain the hidden space feature containing context information;Again input into history decoder, and the text features recognized before are fused in history decoder, to obtain the feature information containing context;Finally, the decoding operation is carried out to obtain the recognition result of target voice.The application fully exploits the context information in the call, makes up for the lack of context information for traditional speech recognition, and improves the precision of call voice recognition.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of pattern recognition, and particularly relates to a method for recognizing call speech. BACKGROUND

[0002] With the continuous development of speech recognition, more and more recognition models are being proposed, among which typical models such as models based on Transformer and models based on Transducer can achieve good application effects on speech recognition. However, in most real scenarios, people's speech is in the form of dialogue, such as call speech. The whole world produces a large amount of call speech data every day. However, because of the privacy protection and lack of labeled data of call speech data at the present stage, there has been no model specially for recognizing this field of call speech. Now many commercial solutions for call speech recognition are divided into single speech recognition and do not consider the supplement of contextual information to the recognition effect in the call.

[0003] Contextual information is an important factor in call speech recognition, which contains the theme of the current call, related names and words that often appear in the theme. The traditional speech recognition mostly uses the WFST mapping method to solve the utilization of these contexts, which trains the specified hot words to achieve the use of contextual features. However, such mapping method can only be used for single scene, and the specified hot words also need to be evaluated by an expert system. In order to solve this problem, people explore the method used in machine translation and question answering system in NLP, most of which are improved based on the Transformer model. However, the model has a fatal problem in the long sequence problem, that is, the length of the sequence directly affects the high computational load and memory cost. SUMMARY

[0004] In order to overcome the deficiencies of the prior art, the application provides a call speech recognition method based on multiple encoders, which first classifies the call speech according to channel information; the classified speech is segmented and sorted according to speaking time and blank information; the sorted speech is preprocessed and converted into 80-dimensional Fbank features; the Fbank features are spliced according to the previous sentence, the target sentence and the next sentence as the source of context information; the spliced speech and the target sentence are respectively input into two different encoders, and the context information is fused in different ways to obtain hidden space features containing context information; then input into the history decoder, and fuse with the previously recognized text features in the history decoder to obtain feature information containing context; finally, the decoding operation is performed to obtain the recognition result of the target speech. The application fully excavates the context information in the call, makes up for the lack of context information in traditional speech recognition, and improves the accuracy of call speech recognition.

[0005] The technical solution adopted by the application to solve the technical problems comprises the following steps:

[0006] Step 1: classifying the call speech according to channel information;

[0007] Step 2: segmenting the classified speech with blank information and sorting according to the speaking start time;

[0008] Step 3: preprocessing the sorted speech, converting the acoustic signal of the speech into 80-dimensional Fbank features, and counting the words in the annotated text of the call speech into the word library;

[0009] Step 4: splicing the Fbank features according to the previous sentence, the target sentence and the next sentence as the source of context information; inputting the spliced speech and the target sentence into two different encoders, and fusing the context information according to external fusion or internal fusion to obtain hidden space features containing context information;

[0010] Step 5: inputting the hidden space features into the history decoder, fusing the recognized text features in the history decoder to obtain feature information containing context;

[0011] Step 6: inputting the context-containing feature information obtained in step 5 into the decoder for masked decoding operation to obtain decoding feature information;

[0012] Step 7: inputting the decoding feature information into a preset normalization exponential function to obtain the recognition result of the target speech.

[0013] Further, the step 2 is specifically:

[0014] Segmentation is performed based on three indicators: minimum volume, maximum time interval between two sentences, and minimum duration of a single audio. The specific segmentation process is as follows:

[0015] Step 2-1: There will be background noise in the blank sound segment, and the segments with a volume lower than the minimum volume will be filtered out;

[0016] Step 2-2: To ensure the minimum volume does not delete the boundary sound, that is, the ending sound after a sentence, the end of the sound is controlled by the maximum time interval between two sentences;

[0017] Step 2-3: Use the minimum time length of a single audio to remove the portion of the call speech that is louder than the minimum volume but shorter than the set threshold and does not contain valid information, ultimately obtaining the segmented audio.

[0018] Step 2-4: Sort the audio segments segmented by the indicators by the start time of speaking.

[0019] Furthermore, the specific implementation steps of step 3 are: pre-processing using the output mode of Fbank, the process of which includes:

[0020] Step 3-1: Perform pre-emphasis processing and filter the mid- and low-frequency bands of the voice signal through a high-pass filter;

[0021] Step 3-2: Perform frame processing on the speech signal that has passed the high-pass filter, that is, cut the variable-length audio into small segments of fixed length to obtain the framed speech signal wav[X1,X2,X3,…,Xn];

[0022] Step 3-3: Set a sliding Hamming window function to perform Fourier transform on the framed speech signal to convert the speech data from a time domain signal to a frequency domain signal;

[0023] wav_frame=hamming(wav,10ms)

[0024] wav_data=fft(wav_frame)

[0025] Step 3-4: The acquired frequency domain signal is filtered through the logarithmic Mel spectrum to complete the Fbank calculation and obtain the Fbank feature with an 80-dimensional spectrum:

[0026] wav Fbank =log_mel(wav_data).

[0027] Step 3-5: Count the words that appear in the marked text corresponding to the call voice into the vocabulary, and replace these words with digital indexes to ensure that there are no repetitions.

[0028] Further, the specific implementation steps of step 4 are:

[0029] Step 4-1: splice the Fbank features according to the previous sentence, the next sentence and the target sentence to form context information Can wav ;

[0030] Con wav = Cat(wav1, wav2, wav3)

[0031] wherein wav1, wav2, wav3 represent the target sentence, the previous sentence and the next sentence of the target sentence respectively; Cat(.) represents splicing;

[0032] Step 4-2: select one of external fusion or internal fusion for feature fusion to obtain the latent space feature;

[0033] Step 4-2-1: external fusion, composed of Longformer encoder and Conformer encoder;

[0034] input the context information Con wav into the Longformer encoder to obtain its high-level feature C out in the latent space; input the target sentence wav1 into the Conformer encoder to obtain its high-level feature S out in the latent space; then input the two high-level features into an attention layer, and then input the high-level feature C feature obtained in the attention layer and S out obtained in the Conformer encoder into the Decoder layer for decoding operation through a gating mechanism, and finally input together into the Decoder layer for decoding operation, represented as follows:

[0035] S out = Conformer(wav1)

[0036] C out = Longformer(Con wav )

[0037] C feature = Attention(Q = S out , K = C out , V = C out )

[0038] C feature = cat(S out , C out )

[0039] g = sigmoid(C feature )

[0040] S′ out =g*C feature +(1-g)*C feature

[0041] wherein Attention is an attention layer, Q, K, V are attention parameters, and sigmoid is an activation function;

[0042] Step 4-2-2: internal fusion;

[0043] The context information Con wav is input into the Longformer encoder to obtain its high-level feature C out in the hidden space; the target sentence wav1 is input into the Conformer encoder to obtain its high-level feature S out in the hidden space; then the two high-level features are input into the Decoder layer, and are respectively decoded and trained with the corresponding text labels, and finally the high-level features C′ out and S′ out obtained by decoding and training are respectively obtained.

[0044] The fusion is performed through a gating mechanism, and finally the recognition result is obtained through dimension change of a fully connected layer;

[0045] S out =Conformer(wav2)

[0046] C out =Longformer(Con wav )

[0047] C′ out =Attention(Q=label,K=C out ,V=C out )

[0048] S′ out =Attention(Q=label,K=S out ,V=S out )

[0049] C feature =cat(C′ out ,S′ out )

[0050] g=sigmoid(C feature )

[0051] S″ out =g*C feature +(1-g)*C feature

[0052] Among them, Attention is the attention layer, Q, K, V are attention parameters, sigmoid is the activation function, and label is the text annotation.

[0053] Furthermore, the specific implementation steps of step 5 are:

[0054] Step 5-1: Input the latent space features into the decoder layer of the history decoder for decoding;

[0055] Step 5-2: The history decoder caches the historical information decoded during the decoding process as global context information att_catch for the current speech decoding, where att represents the decoded output feature of the decoder;

[0056] att,att_catch=Drcoder(S″ out )

[0057] Step 5-3: Combine the global context information cached from the previous sentence with the high-level features S of the target sentence out Feature fusion through gating mechanism;

[0058] C feature =cat(S″ out , att_catch)

[0059] g=sigmoid(C feature )

[0060] S″′ out =g*C feature +(1-g)*C feature .

[0061] Furthermore, the specific implementation steps of step 6 are:

[0062] The feature information obtained in step 5 is input into the Decoder layer for decoding training, and the next information is iteratively predicted through the mask mechanism;

[0063] S mask =mask(S″′) out )

[0064] hiden_input=Attion(Q=S mask ,K=S mask ,V=S mask )

[0065] The mask is a mask mechanism.

[0066] Furthermore, the specific implementation steps of step 7 are:

[0067] Step 7-1: input the decoding feature information obtained in step 6 into a full connection layer, the input dimension is the size of the hidden space, and the output dimension is the size of the vocabulary;

[0068] u i = linear(hiden_input, vocab_size)

[0069] wherein hiden_input is the size of the hidden space, vocab_size is the size of the vocabulary, and linear is the full connection layer;

[0070] Step 7-2: input the conversion result of the feature representation into a preset normalization exponential function to obtain the recognition result of the target sentence voice, and the preset normalization exponential function is as follows:

[0071]

[0072] wherein Y is the recognition result of the target voice, T represents the category of the vocabulary, and u i represents the conversion result of the feature representation.

[0073] Preferably, the channel information is left and right channels.

[0074] Preferably, in step 3-2, the fixed-length small segment is one frame of 10-30 ms.

[0075] The present application has the following beneficial effects:

[0076] The present application provides two feature fusion methods and historical decoders, which sufficiently mine the context information in the call to make up for the lack of context information in traditional speech recognition. Thus, the accuracy of call speech recognition is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0077] Figure 1 is a flowchart of a call speech recognition method based on a multi-encoder according to the present application.

[0078] Figure 2 is a context information external fusion method flowchart of a call speech recognition based on a multi-encoder according to the present application.

[0079] Figure 3 is a context information internal fusion method flowchart of a call speech recognition based on a multi-encoder according to the present application.

[0080] Figure 4 is a historical decoder method flowchart of a call speech recognition based on a multi-encoder according to the present application. DETAILED DESCRIPTION

[0081] The application will be further described below with reference to the accompanying drawings and examples.

[0082] The application aims to provide a multi-encoder-based call voice recognition method to effectively improve the accuracy of call voice recognition.

[0083] A multi-encoder-based call voice recognition method, comprising the following steps:

[0084] Step 1: classifying call voice according to channel information;

[0085] Step 2: sorting the classified voice according to the start time of speech after cutting the voice with blank information;

[0086] Step 3: preprocessing the sorted voice, converting the acoustic signal of the voice into 80-dimensional Fbank features, and counting the words in the labeled text of the call voice into the word library;

[0087] Step 4: concatenating the Fbank features according to the previous sentence, the target sentence and the next sentence as the source of context information; inputting the concatenated voice and the target sentence into two different encoders respectively, and performing context information fusion according to external fusion or internal fusion to obtain hidden space features containing context information;

[0088] Step 5: inputting the hidden space features into a history decoder to fuse with the recognized text features in the history decoder to obtain feature information containing context;

[0089] Step 6: inputting the feature information containing context obtained in step 5 into a decoder to perform masked decoding operation to obtain decoding feature information;

[0090] Step 7: inputting the decoding feature information into a preset normalization exponential function to obtain the recognition result of the target voice.

[0091] Further, the step 2 specifically comprises:

[0092] According to the three indicators of minimum volume, maximum time interval between two sentences and minimum time length of a single audio, the specific cutting process is as follows:

[0093] Step 2-1: There will be background noise in the blank sound segment, which is filtered by the minimum volume, and the segment smaller than the minimum volume is deleted;

[0094] Step 2-2: To ensure that the minimum volume does not delete the boundary sound, i.e. the tail sound at the end of a sentence, the end of the sound is controlled by the maximum time interval between two sentences.

[0095] Step 2-3: Use the minimum time length of a single audio to remove the portion of the call speech that is louder than the minimum volume but shorter than the set threshold and does not contain valid information, ultimately obtaining the segmented audio.

[0096] Step 2-4: Sort the audio segments segmented by the indicators by the start time of speaking.

[0097] Furthermore, the specific implementation steps of step 3 are: pre-processing using the output mode of Fbank, the process of which includes:

[0098] Step 3-1: Perform pre-emphasis processing and filter the mid- and low-frequency bands of the voice signal through a high-pass filter;

[0099] Step 3-2: Perform frame processing on the speech signal that has passed the high-pass filter, that is, cut the variable-length audio into small segments of fixed length to obtain the framed speech signal wav[X1,X2,X3,…,Xn];

[0100] Step 3-3: Set a sliding Hamming window function to perform Fourier transform on the framed speech signal to convert the speech data from a time domain signal to a frequency domain signal;

[0101] wav_frame=hamming(wav,10ms)

[0102] wav_data=fft(wav_frame)

[0103] Step 3-4: The acquired frequency domain signal is filtered through the logarithmic Mel spectrum to complete the Fbank calculation and obtain the Fbank feature with an 80-dimensional spectrum:

[0104] wav Fbank =log_mel(wav_data).

[0105] Step 3-5: Count the words that appear in the marked text corresponding to the call voice into the vocabulary, and replace these words with digital indexes to ensure that there are no repetitions.

[0106] Furthermore, the specific implementation steps of step 4 are:

[0107] Step 4-1: Concatenate the Fbank features according to the previous and next sentences of the target sentence to form the context information Con wav ;

[0108] Con wav =Cat(wav1,wav2,wav3)

[0109] Wherein, wav1, wav2, wav3 represent the target sentence, the previous sentence and the next sentence of the target sentence respectively; Cat(.) represents concatenation;

[0110] Step 4-2: select one of external fusion or internal fusion for feature fusion to obtain the latent space feature;

[0111] Step 4-2-1: external fusion, composed of Longformer encoder and Conformer encoder;

[0112] The context information Con wav is input into the Longformer encoder to obtain its high-level feature C out in the latent space; the target sentence wav1 is input into the Conformer encoder to obtain its high-level feature S out in the latent space; then the two high-level features are input into an attention layer, and the high-level feature C feature obtained in the attention layer is fused with S out obtained in the Conformer encoder through a gating mechanism, and finally input into the Decoder layer for decoding operation, which is represented as follows:

[0113] S out =Conformer(wav1)

[0114] C out =Longformer(Con wav )

[0115] C feature =Attention(Q=S out ,K=C out ,V=C out )

[0116] C feature =cat(S out ,C out )

[0117] g=sigmoid(C feature )

[0118] S′ out =g*C feature +(1-g)*C feature

[0119] Wherein, Attention is an attention layer, Q, K, V are attention parameters, and sigmoid is an activation function;

[0120] Step 4-2-2: internal fusion;

[0121] Contextual information Con wav is input into the Longformer encoder to obtain its high-level features C in the hidden space out ; the target sentence wav1 is input into the Conformer encoder to obtain its high-level features S in the hidden space out ; then the two high-level features are input into the Decoder layer, respectively with its corresponding text label for decoding training, and finally the high-level features C' out and S' out obtained by decoding training are respectively obtained

[0122] Through the gating mechanism, finally through the full connection layer to change the dimension, finally obtain the recognition result

[0123] S out = Conformer(wav2)

[0124] C out = Longformer(Con wav )

[0125] C' out = Attention(Q = label, K = C out , V = C out )

[0126] S' out = Attention(Q = label, K = S out , V = S out )

[0127] C feature = cat(C' out , S' out )

[0128] g = sigmoid(C feature )

[0129] S" out = g * G feature + (1-g) * C feature

[0130] Where Attention is the attention layer, Q, K, V are attention parameters, sigmoid is the activation function, and label is the text label.

[0131] Further, the specific implementation steps of step 5 are:

[0132] Step 5-1: input the hidden space features into the decoder layer of the history decoder to perform decoding operation

[0133] Step 5-2: The history decoder is to cache the history information decoded in the decoding process as the global context information att catch of the current speech decoding, and att represents the decoding output feature of the decoder;

[0134] att, att catch = Decoder (S" out )

[0135] Step 5-3: The global context information cached in the previous sentence is fused with the high-level feature S out of the target sentence through a gating mechanism;

[0136] C feature = cat (S" out , att catch)

[0137] g = sigmoid (C feature )

[0138] S" cut = g * C feature + (1-g) * C feature .

[0139] Further, the specific implementation steps of step 6 are:

[0140] The feature information obtained in step 5 is input into the Decoder layer for decoding training, and the next information is iteratively predicted through the mask mechanism;

[0141] S mask = mask (S" out )

[0142] hiden_input = Atttion (Q = S mask , K = S mask , V = S mask )

[0143] Where mask is a mask mechanism.

[0144] Further, the specific implementation steps of step 7 are:

[0145] Step 7-1: The decoding feature information obtained in step 6 is input into the fully connected layer, and the input dimension is the size of the hidden space, and the output dimension is the size of the vocabulary;

[0146] u i = linear (hiden_input, vocab_size)

[0147] where hiden_input is the size of the hidden space, vocab_size is the size of the vocabulary, and linear is a full connection layer.

[0148] Step 7-2: input the conversion result of the feature representation into a preset normalization exponential function to obtain a recognition result of the target sentence voice, and the preset normalization exponential function is as follows:

[0149]

[0150] where Y is the recognition result of the target voice, T represents the category of the vocabulary, and u i represents the conversion result of the feature representation. Specific embodiments:

[0152] Figure 1 is a flowchart of a call voice recognition method based on a multi-encoder according to an embodiment of the present application. As shown in the figure, the call voice recognition method based on the multi-encoder includes the following steps: Figure 1

[0153] In step 1, call recordings are input into an input end.

[0154] In step 2, data processing is performed on the input call recordings.

[0155] In an embodiment of the present application, the voice recognition device performs preprocessing on multiple call recordings to classify speakers, sort speaking times, and process blank segments.

[0156] Step 2.1: classify the call recordings according to channel information;

[0157] Step 2.2: divide the classified voice according to three indicators, i.e., minimum volume, maximum time interval between two sentences, and minimum time length of a single audio;

[0158] Step 2.3: finally, sort the voice according to speaking times;

[0159] In step 3, the sorted voice is preprocessed to convert the acoustic signal of the voice into 80-dimensional Fbank features, and the process includes:

[0160] Step 3.1: pre-emphasis processing is to filter the low-frequency band in the voice signal through a high-pass filter;

[0161] Step 3.2: frame processing is performed on the voice signal passing through the high-pass filter, i.e., the indefinite-length audio is divided into small segments of fixed length, generally 10-30 ms per frame, and then the frame-processed voice signal wav[X1, X2, X3, …, Xn] is obtained.

[0162] ​Step 3.3, set a sliding Hamming window function to perform Fourier transform (corresponding to the discrete Fourier transform of digital audio) on these framed sentences, to convert the speech data from time domain signal to frequency domain signal;

[0163] Step 3.4, the obtained frequency domain signal is filtered by log mel spectrum to complete the calculation of Fbank, and the frequency spectrum is 80-dimensional wav_data(batch_size, wav_input, 80).

[0164] Step 3 can be represented by the following calculation formula:

[0165] wav_frame=hamming(wav,10ms)

[0166] wav_data=fft(wav_frame)

[0167] wav Fbank =log_mel(wav_data)

[0168] In step 4, the preprocessed Fbank features are spliced according to the previous sentence, the recognized sentence and the next sentence as the source of context information; the spliced speech and the speech to be recognized are respectively input into two different encoders, and the context information is fused in different ways to obtain the hidden space features containing context information. The specific steps are as follows:

[0169] Step 4.1, the previous sentence, the next sentence and the target sentence are spliced to form context information Con_wav;

[0170] Con wav =Cat(wav1,wav2,wav3)

[0171] Step 4.2, two feature fusion methods are proposed: external fusion and internal fusion.

[0172] Please refer to Figure 2 , Figure 2 the structure diagram of the context information external fusion method of the speech recognition based on multiple encoders of the application;

[0173] Figure 2 The network structure diagram shown by the network structure diagram is composed of two encoders, and the context feature fusion is performed in the Encoder layer. First, the spliced speech is input into the Longformer encoder to obtain its high-level features C out in the hidden space, and then the speech to be recognized is input into the Conformer encoder to obtain its high-level features S out, then the two high-level features are input into an attention layer, and then the high-level features C feature obtained in the attention layer are obtained out are fused through a gating mechanism and are finally input into the Decoder layer for decoding operation.

[0174] S out =Conformer(wav1)

[0175] C out =Longformer(Con wav )

[0176] C feature =Attention(Q=S out ,K=C out ,V=C out )

[0177] C feature =cat(S out ,C out )

[0178] g=sigmoid(C feature )

[0179] S′ out =g*C feature +(1-g)*C feature

[0180] wherein Attention is an attention layer, Q, K, and V are attention parameters, cat is a concatenation layer, and sigmoid is an activation function

[0181] Referring to Figure 3 , the network structure diagram of the context information internal fusion method of the speech recognition based on the multi-encoder call;

[0182] Figure 3 The network structure diagram shows that internal fusion: the context feature fusion is performed in the Decoder layer, and the premise is that the concatenated speech is input into the Longformer encoder in the Encoder layer to obtain the high-level feature C out in the hidden space, then the speech to be recognized is input into the Conformer encoder to obtain the high-level feature S out in the hidden space, then they are input into the Decoder layer, and the label value decoding training is respectively performed, and finally the high-level features C′ out and S′ out, and finally the recognition result is obtained through the dimension change of the full connection layer.

[0183] S out = Conformer(wav2)

[0184] C out = Longformer(Con wav )

[0185] C′ out = Attention(Q = label, K = C out , V = C out )

[0186] S′ out = Attention(Q = label, K = S out , V = S out )

[0187] C feature = cat(C′ out , S′ out )

[0188] g = sigmoid(C feature )

[0189] S″ out = g * C feature + (1 - g) * C feature

[0190] Where Attention is the attention layer, Q, K, and V are attention parameters, label is the label dimension, cat is the concatenation layer, and sigmoid is the activation function.

[0191] In step 5, the feature representation is input into the history decoder we propose, where it is fused with previously recognized text features to obtain context-inclusive feature information. The specific steps are as follows:

[0192] Step 5.1: The high-level features output by the encoder are input into the decoder layer for decoding operation, which iteratively predicts the next unit by masking future information.

[0193] Step 5.2: The history decoder we propose is to cache the historical information decoded in the decoding process as global context information att_catch for the current speech decoding.

[0194] att, att_catch = Decoder(S″ out )

[0195] Step 5.3, when the current sentence is identified, the global context information cached in the last sentence is fused with the high-level features of the current sentence through a gating mechanism.

[0196] C feature = cat(S" out , att_catch)

[0197] g = sigmoid(C feature )

[0198] S" out = g * C feature + (1-g) * C feature .

[0199] In step 6, the features are input into the decoder for masked decoding operation to obtain decoding feature information. The specific steps are as follows:

[0200] Step 6.1, the fused features are input into the Decoder layer for decoding training, and the model cannot see the future information through the masking mechanism, so as to iteratively predict the next information.

[0201] S mask = mask(S" out )

[0202] hiden_input = Atttion(Q = S mask , K = S mask , V = S mask )

[0203] Where mask is a masking mechanism.

[0204] In step 7, the decoding feature information is input into a preset normalization exponential function to obtain the recognition result of the target speech. The specific steps are as follows:

[0205] Step 7.1, the obtained features are input into a fully connected layer, the input dimension is the size of the hidden space, and the output dimension is the size of the vocabulary;

[0206] u i = linear(hiden_input, vocab_size)

[0207] Where hiden_input is the size of the hidden space, vocab_size is the size of the vocabulary, and linear is a fully connected layer.

[0208] Step 7.2, the conversion result of the feature representation is input into a preset normalization exponential function to obtain the recognition result of a specific target speech, and the preset normalization exponential function is as follows:

[0209]

[0210] where Y is the recognition result of the target speech, and T represents the category of the vocabulary.

Claims

1. A method for speech recognition based on multiple encoders, characterized in that: The steps include: Step 1: Classify the call voice according to the channel information; Step 2: Segment the classified speech with blank information and sort it according to the start time of speaking; Step 3: Preprocess the sorted speech, convert the speech acoustic signal into 80-dimensional Fbank features, and count the words in the annotated text of the call speech into the vocabulary; Step 4: Concatenate the Fbank features according to the previous sentence, target sentence, and next sentence as the source of contextual information; input the concatenated speech and target sentence into two different encoders respectively, and fuse the contextual information according to external fusion or internal fusion to obtain latent space features containing contextual information; Step 5: Input the latent space features into the history decoder, where they are fused with the recognized text features to obtain feature information containing context; Step 6: Input the feature information containing the context obtained in step 5 into the decoder for masked decoding to obtain decoded feature information; Step 7: Input the decoded feature information into a preset normalized exponential function to obtain the recognition result of the target speech.

2. The method for call speech recognition based on multiple encoders according to claim 1, characterized in that: The step 2 is specifically as follows: Segmentation is performed based on three indicators: minimum volume, maximum time interval between two sentences, and minimum duration of a single audio. The specific segmentation process is as follows: Step 2-1: There will be background noise in the blank sound segment, and the segments with a volume lower than the minimum volume will be filtered out; Step 2-2: To ensure the minimum volume does not delete the boundary sound, that is, the ending sound after a sentence, the end of the sound is controlled by the maximum time interval between two sentences; Step 2-3: Use the minimum time length of a single audio to remove the portion of the call speech that is louder than the minimum volume but shorter than the set threshold and does not contain valid information, ultimately obtaining the segmented audio. Step 2-4: Sort the audio segments segmented by the indicators by the start time of speaking.

3. The method for call speech recognition based on multiple encoders according to claim 2, characterized in that: The specific implementation steps of step 3 are: pre-processing using the output mode of Fbank, the process includes: Step 3-1: Perform pre-emphasis processing and filter the mid- and low-frequency bands of the voice signal through a high-pass filter; Step 3-2: Perform frame processing on the speech signal that has passed the high-pass filter, that is, cut the variable-length audio into small segments of fixed length to obtain the framed speech signal wav[X1,X2,X3,…,Xn]; Step 3-3: Set a sliding Hamming window function to perform Fourier transform on the framed speech signal to convert the speech data from a time domain signal to a frequency domain signal; wav_frame=hamming(wav, 10ms) wav_data=fft(wav_frame) Step 3-4: The acquired frequency domain signal is filtered through the logarithmic Mel spectrum to complete the Fbank calculation and obtain the Fbank feature with an 80-dimensional spectrum: wav Fbank =log_mel(wav_data) Step 3-5: Count the words that appear in the marked text corresponding to the call voice into the vocabulary, and replace these words with digital indexes to ensure that there are no repetitions.

4. The method for call speech recognition based on multiple encoders according to claim 3, characterized in that: The specific implementation steps of step 4 are: Step 4-1: Concatenate the Fbank features according to the previous and next sentences of the target sentence to form the context information Con wav ; Con wav =Cat(wav1,wav2,wav3) Among them, wav1, wav2, and wav3 represent the target sentence, the sentence before the target sentence, and the sentence after the target sentence respectively; Cat(.) represents splicing; Step 4-2: Select either external fusion or internal fusion to perform feature fusion and obtain latent space features; Step 4-2-1: External fusion, consisting of Longformer encoder and Conformer encoder; The context information Con wav Input into the Longformer encoder to obtain its high-level features C in the latent space out ; Input the target sentence wav1 into the Conformer encoder to obtain its high-level features S in the latent space out ; Then these two high-level features are input into an attention layer together, and then the high-level feature C obtained in the attention layer is feature And the Conformer encoder gets S out The fusion is performed through the gating mechanism and finally input into the Decoder layer for decoding operation as follows: S out =Conformer(wav1) C out =Longformer(Con wav ) C feature =Attention(Q=S out ,K=C out ,V=C out ) C feature =cat(S out ,C out ) g=sigmoid(C feature ) S′ out =g*C feature +(1-g)*C feature Among them, Attention is the attention layer, Q, K, V are attention parameters, and sigmoid is the activation function; Step 4-2-2: internal fusion; The context information Con wav Input into the Longformer encoder to obtain its high-level features C in the latent space out ; Input the target sentence wav1 into the Conformer encoder to obtain its high-level features S in the latent space out ; Then these two high-level features are input into the Decoder layer, and decoding training is performed with their corresponding text annotations respectively. Finally, the high-level features C′ obtained by the decoding training are out and S′ out , The fusion is performed through the gating mechanism, and finally the dimension is changed through the fully connected layer to obtain the recognition result; S out =Conformer(wav2) C out =Longformer(Con wav ) C′ out =Attention(Q=label,K=C out ,V=C out ) S′ out =Attention(Q=label,K=S out ,V=S out ) C feature =cat(C′ out ,S′ out ) g=sigmoid(C feature ) S″ out =g*C feature +(1-g)*C feature Among them, Attention is the attention layer, Q, K, V are attention parameters, sigmoid is the activation function, and label is the text annotation.

5. The method for call speech recognition based on multiple encoders according to claim 4, characterized in that: The specific implementation steps of step 5 are: Step 5-1: Input the latent space features into the decoder layer of the history decoder for decoding; Step 5-2: The history decoder caches the historical information decoded during the decoding process as global context information att_catch for the current speech decoding, where att represents the decoded output feature of the decoder; att, att_catch=Decoder(S″ out ) Step 5-3: Combine the global context information cached from the previous sentence with the high-level features S of the target sentence out Feature fusion through gating mechanism; C feature =cat(S″ out ,att_catch) g=sigmoid(C feature ) S″′ out =g*C feature +(1-g)*C feature 。 6. The method for call speech recognition based on multiple encoders according to claim 5, characterized in that: The specific implementation steps of step 6 are: The feature information obtained in step 5 is input into the Decoder layer for decoding training, and the next information is iteratively predicted through the mask mechanism; S mask =mask(S″′ out ) hiden_input=Atttion(Q=S mask ,K=S mask ,V=S mask ) The mask is a mask mechanism.

7. The method for call speech recognition based on multiple encoders according to claim 6, characterized in that: The specific implementation steps of step 7 are: Step 7-1: Input the decoded feature information obtained in step 6 into the fully connected layer. The input dimension is the size of the latent space, and the output dimension is the size of the vocabulary. u i =linear(hiden_input,vocab_size) Among them, hiden_input is the size of the latent space, vocab_size is the vocabulary size, and linear is the fully connected layer; Step 7-2: Input the conversion result of the feature representation into a preset normalized exponential function to obtain the recognition result of the target sentence speech. The preset normalized exponential function is as follows: Among them, Y is the recognition result of the target speech, T represents the category of the vocabulary, and u i Represents the conversion result of the feature representation.

8. The method for call speech recognition based on multiple encoders according to claim 1, characterized in that: The channel information is the left and right channels.

9. The method for call speech recognition based on multiple encoders according to claim 3, characterized in that: In step 3-2, the fixed-length segment is a frame of 10-30 ms.

Images