A multi-lingual speech recognition method with off-line and on-line cooperation

Abstract

The present application relates to the technical field of speech recognition, and discloses a multi-language speech recognition method based on off-line and online cooperation. The method comprises the following steps: extracting the mel-frequency cepstral coefficient of a current speech frame, inputting the language classification network to obtain a language confidence vector; when the maximum value is lower than a preset threshold, extracting the bottom layer representation vector output by the first convolutional layer of the end-side acoustic model, splicing the bottom layer representation vector with the mel-frequency cepstral coefficient, and uploading the bottom layer representation vector to a cloud-side server; inputting the bottom layer representation vector into the attention module of the cloud-side acoustic model as a position prior bias, narrowing the search space of the acoustic candidate state under the constraint of the position prior bias, and decoding the mel-frequency cepstral coefficient to output a recognized text. The present application reduces the calculation resource consumption of the cloud-side acoustic model, shortens the decoding response delay, avoids the recognition errors caused by the model capacity limitation of the end-side, and improves the recognition accuracy and system stability in a multi-language environment.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology and discloses an offline / online collaborative multilingual speech recognition method. Background Technology

[0002] The existing implementation of online-offline collaborative speech recognition in multilingual scenarios involves the edge device acquiring speech signals and running a lightweight general-purpose acoustic model for initial decoding. When the language confidence score output by the edge model falls below a set threshold or the current language is determined to be outside the local vocabulary, the edge device triggers a cloud-based collaborative mechanism, directly packaging and uploading the raw speech data to the cloud server. Upon receiving the raw speech data, the cloud server, using its vast local multilingual acoustic model, performs a complete Viterbi decoding search, starting from the global acoustic candidate states and combining them with the multilingual language model, to output the final multilingual recognized text.

[0003] When dealing with multilingual or non-mainstream language scenarios, the aforementioned existing technical solutions lack specific constraint information from the front-end acoustic processing of the cloud-based multilingual acoustic model after receiving the raw speech data uploaded from the edge. The cloud model must perform joint optimization of acoustic scores and language model scores within the entire unrestricted global acoustic candidate state space. This indiscriminate full search leads to a sharp increase in computational resource consumption when processing languages ​​not covered by the edge, and the path search latency during the decoding process is consequently lengthened, failing to meet the real-time processing requirements of low-latency responses in online and offline collaborative systems in multilingual scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide an offline / online collaborative multilingual speech recognition method that can effectively solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A collaborative online and offline multilingual speech recognition method includes: continuously extracting the Mel frequency cepstral coefficients of the current speech frame from the end-side acoustic front end, and inputting the Mel frequency cepstral coefficients into the end-side lightweight language classification network to obtain a language confidence vector;

[0007] Determine whether the maximum value in the language confidence vector exceeds a preset confidence threshold;

[0008] If the maximum value exceeds the preset confidence threshold, the edge multilingual acoustic model performs streaming decoding on the current speech frame and outputs the recognized text.

[0009] If the maximum value does not exceed the preset confidence threshold, the bottom-level representation vector of the current speech frame output by the first convolutional layer of the acoustic model on the edge is extracted, the bottom-level representation vector is concatenated with the Mel frequency cepstral coefficients of the current speech frame, and uploaded to the cloud server through the communication link.

[0010] The cloud-side server inputs the received underlying representation vector into the attention module of the cloud-side multilingual acoustic model as a positional prior bias. Under the constraint of the positional prior bias, the cloud-side multilingual acoustic model narrows the search space of acoustic candidate states, decodes the Mel frequency cepstral coefficients of the current speech frame, and outputs the recognized text.

[0011] Preferably, the step of obtaining the language confidence vector includes: concatenating the Mel frequency cepstral coefficients of the current speech frame with the Mel frequency cepstral coefficients of the previous N historical speech frames cached in the end-side circular buffer to generate a concatenated spectral feature matrix, where N is a positive integer;

[0012] The spliced ​​spectral feature matrix is ​​input into the multi-head self-attention layer of the edge-side lightweight language classification network to extract the inter-frame correlation weights of the spliced ​​spectral feature matrix in the time dimension.

[0013] The spliced ​​spectral feature matrix carrying the inter-frame correlation weights is input into the fully connected output layer of the edge-side lightweight language classification network, and mapped to generate the language confidence vector containing each preset language dimension.

[0014] Preferably, the step of extracting the underlying representation vector includes: taking the mean pooling result of the convolutional feature map output by the first convolutional layer of the end-side acoustic model in the channel dimension as the initial representation vector;

[0015] Perform scalar quantization operation based on a preset quantization codebook on the initial representation vector, and map the initial representation vector to the index sequence of the corresponding codeword in the preset quantization codebook;

[0016] The index sequence is variable-length encoded to generate the underlying representation vector in binary bitstream form;

[0017] Before concatenating the underlying representation vector with the Mel frequency cepstral coefficients of the current speech frame, the underlying representation vector in binary bitstream form is restored to a floating-point vector.

[0018] Preferably, the step of uploading to the cloud server via the communication link includes: concatenating the underlying representation vector and the Mel frequency cepstral coefficients of the current speech frame in the feature dimension to generate a hybrid feature data packet;

[0019] Obtain the current round-trip latency of the network link between the endpoint and the cloud server;

[0020] When the round-trip time of the current network link is less than a preset delay threshold, the hybrid feature data packet is transmitted through the low-latency channel of the communication link.

[0021] When the round-trip time of the current network link is greater than or equal to the preset delay threshold, the underlying representation vector in the hybrid feature data packet is subjected to secondary downsampling processing, and the hybrid feature data packet after secondary downsampling processing is transmitted through the ordinary data channel of the communication link.

[0022] Preferably, the step of inputting the received underlying representation vector into the attention module of the cloud-side multilingual acoustic model as a positional prior bias includes: within the attention module of the cloud-side multilingual acoustic model, using the query vector generated by linear mapping of the Mel frequency cepstral coefficients of the current speech frame as the basic query matrix;

[0023] The key vector generated by linear mapping of the underlying representation vector is multiplied by the basic query matrix to generate the bias attention score matrix.

[0024] The biased attention score matrix is ​​superimposed on the original attention score matrix of the cloud-side multilingual acoustic model to generate the superimposed attention score matrix.

[0025] The superimposed attention score matrix is ​​subjected to masking and Softmax normalization to generate the position prior bias.

[0026] Preferably, the step of the cloud-side multilingual acoustic model narrowing the search space of acoustic candidate states and performing decoding under the constraint of the position prior bias includes: applying the position prior bias to the acoustic output posterior probability distribution of the cloud-side multilingual acoustic model, setting the probability values ​​corresponding to candidate phonemes below a preset probability threshold in the acoustic output posterior probability distribution to zero and generating a truncated posterior probability distribution;

[0027] The truncated posterior probability distribution is input into a pre-constructed weighted finite-state converter, which integrates a multilingual pronunciation dictionary and a multilingual language model.

[0028] Inside the weighted finite state converter, a Viterbi pruning search is performed along the active path on the truncated posterior probability distribution to output the recognized text.

[0029] Preferably, in the step of extracting the inter-frame correlation weights of the spliced ​​spectral feature matrix in the time dimension, a language relative position encoding matrix is ​​introduced inside the multi-head self-attention layer of the end-side lightweight language classification network;

[0030] Calculate the absolute time difference between the i-th frame and the j-th frame in the spliced ​​spectral feature matrix, and map the absolute time difference to the corresponding vector in the language relative position encoding matrix;

[0031] Add the corresponding vector to the key vector of the spliced ​​spectral feature matrix to generate a bias key vector;

[0032] The attention score is calculated using the bias key vector and the query vector of the spliced ​​spectral feature matrix, and the inter-frame association weight is determined by the attention score, where i and j are both positive integers and i is not equal to j.

[0033] Preferably, in the step of performing scalar quantization operation based on a preset quantization codebook on the initial representation vector, the preset quantization codebook is maintained on the terminal side, and the preset quantization codebook is periodically updated through the following steps: obtaining multiple historical initial representation vectors corresponding to multiple historical speech frames in the current period;

[0034] Calculate the cluster center of the multiple historical initial representation vectors, and remove the codewords in the current preset quantization codebook that are farthest from the cluster center;

[0035] Insert the vector corresponding to the cluster center at the removal position to generate the updated preset quantization codebook;

[0036] When performing scalar quantization, the Euclidean distance between the initial representation vector and each codeword in the updated preset quantization codebook is calculated, and the codeword with the smallest Euclidean distance is selected as the quantization result.

[0037] Preferably, in the step of superimposing the bias attention score matrix onto the original attention score matrix of the cloud-side multilingual acoustic model to generate the superimposed attention score matrix, the historical attention entropy value output by the cloud-side multilingual acoustic model at the previous decoding time is obtained.

[0038] A dynamic temperature scaling factor is calculated based on the historical attention entropy value, wherein the historical attention entropy value is negatively correlated with the dynamic temperature scaling factor.

[0039] The elements in the bias attention score matrix are multiplied and scaled using the dynamic temperature scaling factor to generate a scaled bias matrix.

[0040] The scaled bias matrix is ​​added element-wise to the original attention score matrix to output the superimposed attention score matrix.

[0041] Preferably, in the step of performing Viterbi pruning search on the truncated posterior probability distribution along the active path inside the weighted finite state converter, a candidate word grid is constructed at each word-ending node of the weighted finite state converter.

[0042] In the candidate word grid, candidate paths whose cumulative path score is greater than the difference between the current global maximum path score and the preset retention difference are retained, and candidate paths in the candidate word grid whose cumulative path score is less than the difference between the current global maximum path score and the preset retention difference are truncated.

[0043] The word-ending hidden states of the retained candidate paths are used as the starting hidden states for Viterbi pruning search in the weighted finite state converter for the next speech frame until the speech stream ends, and the recognized text is generated by backtracking along the optimal retained path.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0045] 1. When the edge device encounters a speech frame with low language confidence, this invention extracts the low-level representation vector output from the first convolutional layer of the edge acoustic model. This low-level representation vector is then concatenated with Mel-frequency cepstral coefficients and uploaded to the cloud server. The cloud server injects the low-level representation vector into the attention module of the multilingual acoustic model as a positional prior bias, and uses this bias to superimpose and intervene in the original attention score matrix. This processing method ensures that the cloud-based multilingual acoustic model, during decoding, is constrained by the orientation of the feature vectors extracted by the edge shallow network, directly shielding the score weights of irrelevant acoustic candidate states, reducing the search space of acoustic candidate states, decreasing the optimization computation load in the cloud when processing low-confidence multilingual speech, and shortening the cloud decoding response time.

[0046] 2. This invention increases the amount of classification input information for language classification networks in complex speech environments by concatenating the current speech frame with historical speech frames in a circular buffer at the edge to generate a spectral feature matrix, combined with the extraction of inter-frame correlation weights by a multi-head self-attention layer. In the low-level representation vector extraction stage, a preset quantization codebook is used for scalar quantization and periodic updates, reducing the data transmission bandwidth usage of the communication link between the edge and cloud sides. In the cloud decoding stage, the low-level representation vector is mapped to a bias key vector and multiplied with the basic query matrix to generate a bias attention score matrix. This bias matrix is ​​then adjusted using a dynamic temperature scaling factor and superimposed, ensuring that the weight distribution of the cloud attention module aligns with the actual speech feature trends at the edge. In the acoustic posterior probability distribution processing, candidate phonemes below the probability threshold are zeroed out to generate a truncated posterior probability distribution. Viterbi pruning is then performed in the weighted finite state converter using a preset retention difference, eliminating invalid candidate paths and reducing state memory overhead during cloud decoding. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating the overall process of online / offline collaborative multilingual speech recognition for this invention.

[0048] Figure 2 This is a flowchart of the lightweight language classification network processing on the edge side of the present invention;

[0049] Figure 3 This is a flowchart of the underlying representation vector quantization and codebook update process of this invention;

[0050] Figure 4 This is a flowchart of the hybrid feature transmission process for network state awareness according to the present invention;

[0051] Figure 5 This is a flowchart illustrating the generation process of the prior bias for the cloud-side attention module position in this invention.

[0052] Figure 6 This is a flowchart of the Viterbi pruning decoding process for the cloud-side weighted finite state converter of the present invention. Detailed Implementation

[0053] Please refer to the attached document. Figure 1This embodiment provides an offline / online collaborative multilingual speech recognition method. The end-side device acquires a continuous speech stream signal through an audio acquisition channel. The end-side acoustic front-end performs frame-by-frame processing on the speech stream signal, continuously extracting the Mel-frequency cepstral coefficients of the current speech frame. The end-side acoustic front-end performs pre-emphasis processing on the input speech stream signal to compensate for the high-frequency components attenuated during transmission and acquisition. The pre-emphasis processed speech signal is divided into continuous speech frames, with a preset overlap interval maintained between adjacent speech frames. Windowing processing is performed on each frame of the speech signal to suppress spectral leakage. Subsequently, a Fast Fourier Transform is performed on the windowed speech frame to convert the time-domain speech signal into a frequency-domain linear power spectrum. The linear power spectrum is input into a preset Mel filter bank to obtain the Mel-scale spectral energy distribution. Logarithmic operation and discrete cosine transform are performed on the Mel spectral energy distribution to extract the Mel-frequency cepstral coefficients of the corresponding speech frame. The mapping relationship between Mel frequency and linear frequency is defined by the following formula:

[0054]

[0055] in, Mel frequency, measured in megahertz (mJ). The frequency is linear, measured in Hertz. This formula maps the linear frequency scale to the Mel frequency scale perceived by human hearing, providing a frequency scale benchmark for the construction of Mel filter banks and the extraction of Mel frequency cepstral coefficients.

[0056] Furthermore, the extracted Mel-frequency cepstral coefficients of the current speech frame are input into the edge-side lightweight language classification network to obtain a language confidence vector. The edge-side lightweight language classification network receives the Mel-frequency cepstral coefficients of the current speech frame and, through feature extraction and mapping processing, outputs a language confidence vector with the same dimensions as the preset number of languages. Each element in the language confidence vector corresponds to the classification confidence of a preset language. The edge-side inference control unit extracts the maximum value from the language confidence vector and compares this maximum value with a preset confidence threshold. Based on the comparison result, the corresponding decoding processing branch is triggered.

[0057] If the maximum value exceeds the preset confidence threshold, the edge-side multilingual acoustic model performs streaming decoding on the current speech frame and outputs the recognized text. The edge-side multilingual acoustic model adopts a streaming end-to-end acoustic model based on connectionist temporal classification. The model contains stacked one-dimensional convolutional layers and bidirectional gated recurrent unit layers. It receives the Mel-frequency cepstral coefficients of the current speech frame, outputs the posterior probability of phonemes frame by frame, and combines the corresponding language pronunciation dictionary and n-gram grammar language model stored locally on the edge to perform a frame-by-frame Viterbi search to generate streaming recognized text, which is output to the edge-side application layer in real time.

[0058] If the maximum value does not exceed the preset confidence threshold, the low-level representation vector output by the first convolutional layer of the current speech frame in the edge acoustic model is extracted. This low-level representation vector is then concatenated with the Mel-frequency cepstral coefficients of the current speech frame and uploaded to the cloud server via the communication link. The first convolutional layer of the edge acoustic model is the model's first feature extraction layer. It receives the Mel-frequency cepstral coefficients of the current speech frame as input, performs one-dimensional convolution operations, and outputs a multi-channel convolutional feature map. Channel-dimensional aggregation processing is then performed on the convolutional feature map to extract the low-level representation vector. The edge data packaging unit concatenates the low-level representation vector with the Mel-frequency cepstral coefficients of the current speech frame along the feature dimension, generating a unified format transmission data packet. This transmission data packet is then uploaded to the cloud server via the communication link established between the edge and the cloud server.

[0059] The cloud-side server inputs the received low-level representation vector into the attention module of the cloud-side multilingual acoustic model as a positional prior bias. Under the constraint of the positional prior bias, the cloud-side multilingual acoustic model narrows the search space of acoustic candidate states, decodes the Mel-frequency cepstral coefficients of the current speech frame, and outputs the recognized text. The receiving unit of the cloud-side server performs parsing processing on the transmitted data packets, separating the Mel-frequency cepstral coefficients and the low-level representation vector of the current speech frame. The Mel-frequency cepstral coefficients are input into the encoder input of the cloud-side multilingual acoustic model, while the low-level representation vector is input into the attention module of the cloud-side multilingual acoustic model. The cloud-side multilingual acoustic model adopts an end-to-end speech recognition architecture based on Transformer, including an encoder, an attention module, and a decoder. The encoder is composed of multiple convolutional sub-layers and a Transformer encoder layer stacked together, and the attention module is located between the encoder and the decoder.

[0060] The attention module of the cloud-based multilingual acoustic model performs linear mapping on the underlying representation vectors to generate a priori position bias for the corresponding speech frame. This priori position bias is then superimposed on the attention score calculation process of the attention module, applying directional constraints to the attention weight distribution of the acoustic features output by the encoder. Weights corresponding to candidate states irrelevant to the acoustic features of the current speech frame are masked, thus narrowing the search space for acoustic candidate states. The decoder of the cloud-based multilingual acoustic model receives the constrained acoustic features and, combined with a decoding network integrating a multilingual pronunciation dictionary and a multilingual language model, performs a decoding search, outputting the recognized text for the corresponding speech frame. This recognized text is then transmitted back to the edge device via a communication link.

[0061] Table 1. Comparison of characteristic parameters at each stage of the entire speech signal processing workflow.

[0062] Feature processing stage Feature Dimension Physical meaning Data format Flow direction Mel frequency cepstral coefficient extraction 1-40 dimensions Characterizing the frequency domain energy distribution features of speech frames, covering the Mel frequency range that is sensitive to human hearing. 32-bit floating-point vector From edge acoustic front-end to language classification network, edge acoustic model, cloud server Language confidence vector Consistent with the number of preset languages Each dimension corresponds to the classification confidence score of a single preset language, representing the probability that the current speech frame belongs to the corresponding language. 32-bit floating-point vector End-side language classification network to inference control unit Low-level representation vector 1-64 dimensions The low-level acoustic representation of the speech frame extracted from the first convolutional layer of the end-side acoustic model preserves the basic temporal and frequency domain correlation features of the speech. 32-bit floating-point vector or binary encoded bitstream From edge acoustic model to data packaging unit, cloud server attention module Acoustic posterior probability distribution Consistent with the preset number of phonemes Each dimension corresponds to the output probability of a single phoneme, representing the likelihood of the phoneme corresponding to the current speech frame. 32-bit floating-point vector Acoustic model encoder to decoder network Recognize text Variable-length character sequences Textualized output of the speech signal UTF-8 encoded character sequence Decode network to end-side application layer

[0063] This table clarifies the dimensional definitions, physical meanings, data formats, and flow directions of output features at each key stage in the entire process of speech signal processing on both the device and cloud sides. It provides a unified format constraint for feature data interaction between the device and cloud sides, ensuring the integrity and consistency of semantic information during feature flow.

[0064] In this embodiment, the edge device extracts the Mel-frequency cepstral coefficients of the speech frame through an acoustic front-end, outputs a language confidence vector through a lightweight language classification network, and selects either edge-side local streaming decoding or cloud-side collaborative decoding based on the confidence threshold. In the cloud-side collaborative decoding branch, the edge device extracts the low-level representation vector output from the first convolutional layer of the acoustic model, concatenates it with the Mel-frequency cepstral coefficients, and uploads it to the cloud. The cloud device injects the low-level representation vector into an attention module to generate a positional prior bias. This positional prior bias constrains the search space of the acoustic candidate states, completing the decoding process of the speech frame. This embodiment provides directional constraints for cloud-side decoding through the low-level representation extracted by the edge device, avoiding the cloud device from performing an indiscriminate full search within the global acoustic candidate state space.

[0065] Please refer to the attached document. Figure 2 In a preferred embodiment, the edge device caches the Mel-frequency cepstral coefficients (MFCCs) of historical speech frames through a circular buffer, providing contextual feature support for language classification. The edge-side circular buffer is a fixed-length circular storage structure with a storage depth of N frames (where N is a positive integer), continuously caching the MFCCs of the N consecutive historical speech frames preceding the current speech frame. After each new speech frame's MFCC extraction is completed, the buffer performs an enqueue and dequeue operation, storing the latest speech frame features at the end of the buffer while removing the oldest historical feature from the beginning of the buffer. This ensures that the buffer always retains the historical features of the N consecutive frames preceding the current frame, preventing buffer overflow and temporal disorder of historical features.

[0066] The Mel-frequency cepstral coefficients (MFCCs) of the current speech frame are concatenated with the MFCCs of the previous N historical speech frames stored in the edge-side circular buffer to generate a concatenated spectral feature matrix. The MFCCs of the current speech frame are then used as the features of the (N+1)th frame and concatenated with the MFCCs of the previous N historical speech frames stored in the circular buffer, performing a head-to-tail concatenation along the time dimension to generate a concatenated spectral feature matrix of dimension (N+1)×D, where D is the feature dimension of a single frame's MFCCs. During the concatenation process, dimension alignment is performed on all feature frames to ensure that the dimensions of each frame's features are consistent, avoiding dimension mismatch issues during concatenation.

[0067] The concatenated spectral feature matrix is ​​input into the multi-head self-attention layer of the edge-side lightweight language classification network to extract the inter-frame correlation weights in the temporal dimension. The multi-head self-attention layer of the edge-side lightweight language classification network receives the concatenated spectral feature matrix as input, performs layer normalization on the matrix, and then generates a query matrix, key matrix, and value matrix through three independent linear mapping layers. These matrices are then input into the multi-head self-attention computation unit, which outputs a feature matrix carrying the inter-frame correlation weights. The multi-head self-attention computation process is defined by the following formula:

[0068]

[0069]

[0070] in, The number of attention heads in a multi-head self-attention layer; The query matrix is ​​generated by linear mapping from the concatenated spectral feature matrix; The key matrix is ​​generated by linear mapping from the spliced ​​spectral feature matrix; The value matrix is ​​generated by linear mapping from the spliced ​​spectral feature matrix; , , The first The query matrix, key matrix, and value matrix corresponding to each attention head; The dimension of the key vector corresponding to a single attention head; For multi-head output, a linear mapping weight matrix is ​​used. This is a splicing operation for multi-attention head output.

[0071] Within the multi-head self-attention layer of the lightweight language classification network on the edge, a language relative position encoding matrix is ​​introduced to enhance the ability to capture inter-frame temporal correlations. The absolute temporal difference between the i-th and j-th frames in the concatenated spectral feature matrix is ​​calculated and mapped to a corresponding vector in the language relative position encoding matrix, where i and j are both positive integers and i is not equal to j. The corresponding vector is added to the key vector of the concatenated spectral feature matrix to generate a bias key vector. An attention score is calculated using the bias key vector and the query vector of the concatenated spectral feature matrix, and the inter-frame correlation weights are determined based on the attention score. The calculation process of the relative position encoding is defined by the following formula:

[0072]

[0073]

[0074] in, To construct the time index corresponding to the current query frame in the spectral feature matrix, This refers to the time index corresponding to the key-value frames in the spliced ​​spectral feature matrix; For the first Frame and the The absolute time difference between frames; The matrix is ​​a pre-defined language relative position encoding matrix, where the row index corresponds to the absolute time difference and the column index corresponds to the encoding vector dimension. For the first Frame and the Relative position encoding vectors between frames; For the first The original key vector corresponding to the frame; This is the bias key vector after adding relative position encoding.

[0075] The first The query vector and bias key vector corresponding to each frame are multiplied by a dot product to generate attention scores for the corresponding frame pairs. Normalization is then applied to the attention scores of all frame pairs to obtain the inter-frame association weights. The magnitude of the association weights represents the semantic association between the speech features of two frames; higher weights indicate a stronger semantic association. The concatenated spectral feature matrix carrying the inter-frame association weights is then input into a feedforward network layer for feature transformation. The feedforward network layer employs a two-layer linear mapping structure, with a Gaussian error linear unit activation function added between the two linear mapping layers to complete the non-linear transformation of the features.

[0076] The concatenated spectral feature matrix carrying inter-frame correlation weights is input into the fully connected output layer of the lightweight language classification network on the input side, and mapped to generate a language confidence vector containing each preset language dimension. The input dimension of the fully connected output layer is consistent with the output dimension of the feedforward network layer, and the output dimension is consistent with the number of preset languages. The output of the fully connected layer is mapped through the sigmoid activation function to generate a language confidence vector with each dimension having a value between 0 and 1. Each element in the language confidence vector corresponds to the classification confidence of a preset language.

[0077] Table 2. Parameter Configuration Table for Multi-Head Self-Attention Layer of End-Side Lightweight Language Classification Network

[0078] Attention Head Number Single-head feature dimension Linear mapping input dimension Linear mapping output dimension Functional positioning 1 8D 40-dimensional 8D Capturing inter-frame fundamental frequency correlation features over a short time span 2 8D 40-dimensional 8D Capturing inter-frame formant correlation features over a short time span 3 8D 40-dimensional 8D Capturing inter-frame syllable association features over a medium time span 4 8D 40-dimensional 8D Capturing inter-frame tonal correlation features over a medium time span 5 8D 40-dimensional 8D Capturing inter-frame prosodic features over a long time span 6 8D 40-dimensional 8D Capture inter-frame speech rate correlation features over a long period of time

[0079] This table clarifies the parameter configuration and functional positioning of each attention head in the multi-head self-attention layer of the lightweight language classification network on the edge. Through the division of labor design of multiple attention heads, it captures the inter-frame correlation features of speech over different time spans and in different dimensions, providing multi-dimensional feature support for language classification. At the same time, the low-dimensional design of a single head controls the computational overhead on the edge and adapts to the inference performance constraints of the edge device.

[0080] In this embodiment, Mel-frequency cepstral coefficients of historical speech frames are cached in an edge-side circular buffer. The features of the current speech frame are then concatenated with those of historical frames to generate a concatenated spectral feature matrix, providing richer contextual information for language classification. A language relative position encoding matrix is ​​introduced into the multi-head self-attention layer of the lightweight language classification network. A relative position encoding vector is generated based on the absolute temporal difference between frames. After biasing the key vector, an attention score is calculated to obtain the inter-frame association weights. Finally, the language confidence vector is generated through mapping via a fully connected output layer. This embodiment enhances the language classification network's ability to capture inter-frame semantic associations by introducing historical frame contextual information and relative position encoding, improving the stability of language classification results in complex speech environments.

[0081] Please refer to the attached document. Figure 3 In another preferred embodiment, the edge device performs aggregation and quantization processing on the convolutional feature map output by the first convolutional layer of the edge acoustic model to generate a low-level representation vector. The first convolutional layer of the edge acoustic model is a one-dimensional convolutional layer, receiving a Mel-frequency cepstral coefficient matrix with a receiving dimension of T×D as input, where T is the duration of the speech frame and D is the feature dimension of the Mel-frequency cepstral coefficients in a single frame. The first convolutional layer performs convolution operations on the input features through a preset number of one-dimensional convolutional kernels. The convolutional kernels perform sliding window operations along the time dimension, and the output dimension is T×C× The convolutional feature map, where C is the number of output channels of the convolutional layer. This represents the single-channel feature dimension of the convolution output.

[0082] The average pooling result of the convolutional feature map output from the first convolutional layer of the end-side acoustic model is extracted along the channel dimension and used as the initial representation vector. Global average pooling is then performed on the convolutional feature map along the channel dimension, aggregating the multi-channel convolutional feature maps into a single-channel feature vector. The aggregation process is defined by the following formula:

[0083]

[0084] in, The initial representation vector corresponding to the t-th frame of speech is the first... The numerical value of the dimension; In the convolutional feature map output by the first convolutional layer, the t-th frame, the c-th channel, and the... The eigenvalues ​​corresponding to the dimension; C is the total number of channels in the convolutional feature map; For the time index of the speech frame. Indexed by feature dimensions, Channel indexing. Through channel-dimensional mean pooling, the multi-channel convolutional feature maps are aggregated into a single-channel initial representation vector, preserving the underlying acoustic features of the speech frame while reducing the feature dimensionality.

[0085] A scalar quantization operation based on a preset quantization codebook is performed on the initial representation vector, mapping the initial representation vector to an index sequence of corresponding codewords in the preset quantization codebook. The preset quantization codebook is maintained locally on the client side, containing M preset codewords, each codeword being a floating-point vector with the same dimension as the initial representation vector, where M is a positive integer. For each feature value in the initial representation vector, a scalar quantization operation is performed. Specifically, the Euclidean distance between the initial representation vector and each codeword in the quantization codebook is calculated, and the index corresponding to the codeword with the smallest Euclidean distance is selected as the quantization result for that feature value. This process is repeated across all dimensions of the initial representation vector to generate an index sequence with the same dimensions as the initial representation vector. The Euclidean distance is defined by the following formula:

[0086]

[0087] in, This is the initial representation vector; This refers to the m-th codeword in the quantization codebook. Let d' be the value of the d'th dimension of the initial representation vector; Let d' be the value of the d'th dimension of the m-th codeword; The dimension of the initial representation vector; Let be the Euclidean distance between the initial representation vector and the m-th codeword.

[0088] The index sequence is variable-length encoded to generate a low-level representation vector in binary bitstream form. A pre-defined variable-length encoding table is stored locally on the device. This table is generated based on the statistical distribution of massive speech features, assigning shorter binary encoding lengths to index values ​​with higher occurrence probabilities and longer binary encoding lengths to those with lower occurrence probabilities. The generated index sequence is input into the variable-length encoding table, and element-wise variable-length encoding is performed, mapping fixed-length index values ​​to variable-length binary bitstreams. The encoded bitstreams of all dimensions are then concatenated to generate the low-level representation vector in binary bitstream form, reducing the transmission volume of feature data.

[0089] Before concatenating the underlying representation vector with the Mel-frequency cepstral coefficients of the current speech frame, the underlying representation vector in binary bitstream form is restored to a floating-point vector. The data processing unit on the edge performs inverse variable-length encoding on the underlying representation vector in binary bitstream form. Based on a preset variable-length encoding table, the binary bitstream is mapped back to the corresponding index sequence. Then, based on the current preset quantization codebook, each index value in the index sequence is mapped to a floating-point vector of the corresponding codeword, completing the restoration of the underlying representation vector. This ensures that the dimension of the restored underlying representation vector matches that of the Mel-frequency cepstral coefficients, guaranteeing the feasibility of subsequent concatenation operations.

[0090] A preset quantization codebook is maintained on the device side and updated periodically through a preset process. A fixed codebook update cycle is set on the device side, and an update operation is triggered once after each update cycle. Multiple historical initial representation vectors corresponding to multiple historical speech frames within the current cycle are obtained, and all historical initial representation vectors are used as clustering input to perform... Mean value clustering is used to calculate the corresponding number of cluster centers. The calculation process for cluster centers is defined by the following formula:

[0091]

[0092] in, Let be the cluster center corresponding to the k-th cluster; The set of historical initial representation vectors contained in the k-th cluster; The number of historical initial representation vectors contained in the k-th cluster; This is a single historical initial representation vector within a cluster.

[0093] Calculate the Euclidean distance between each codeword in the current preset quantization codebook and all cluster centers. Select the codeword furthest from all cluster centers and remove it from the quantization codebook. Insert the calculated cluster center vector into the corresponding position of the removed codeword to generate an updated preset quantization codebook. During scalar quantization, calculate the Euclidean distance between the initial representation vector and each codeword in the updated preset quantization codebook, and select the codeword with the smallest Euclidean distance as the quantization result. During codebook updates, perform consistency checks on the updated quantization codebook to ensure that all codewords in the codebook have the same dimension and no duplicate codewords, thus avoiding mapping errors during quantization.

[0094] Table 3 Comparison of Preset Quantization Codebook Codeword Index and Vector Mapping Interval

[0095] Codeword Index Corresponding vector numerical range Index binary code length Upper limit of quantization error Applicable feature distribution scenarios 0 [-2.0,-1.5) 3 people 0.25 Low-frequency feature distribution of silence segments in speech frames 1 [-1.5,-1.0) 3 people 0.25 Distribution of negative high amplitude features of voiced segments in speech frames 2 [-1.0,-0.5) 2 people 0.25 Negative mid-amplitude feature distribution of speech frame transition segments 3 [-0.5,0.0) 1 person 0.25 Negative low-amplitude feature distribution of general segments of speech frames 4 [0.0,0.5) 1 person 0.25 Positive low-amplitude feature distribution of common segments in speech frames 5 [0.5,1.0) 2 people 0.25 Positive mid-amplitude feature distribution of speech frame transition segments 6 [1.0,1.5) 3 people 0.25 Positive high amplitude feature distribution of voiced segments in speech frames 7 [1.5,2.0] 3 people 0.25 High-frequency feature distribution of unvoiced segments in speech frames

[0096] The table clarifies the vector value range, encoding length, quantization error upper limit, and applicable scenarios corresponding to each codeword index in the preset quantization codebook. By allocating shorter binary encoding lengths to low-amplitude feature ranges with higher probability of occurrence, the volume of the binary bit stream after variable-length encoding is further reduced while controlling quantization error, thereby reducing the transmission bandwidth occupation between the end side and the cloud side.

[0097] In this embodiment, mean pooling is performed on the channel-dimensional convolutional feature map output from the first convolutional layer of the edge acoustic model to generate an initial representation vector. Based on a preset quantization codebook, scalar quantization and variable-length encoding are performed on the initial representation vector to generate a low-level representation vector in binary bitstream form. Before transmission, this low-level representation vector is restored to a floating-point vector for feature concatenation. Simultaneously, the edge performs clustering based on historical initial representation vectors within the current period, periodically updating the quantization codebook to ensure it adapts to the distribution characteristics of the current speech features. This embodiment reduces the transmission volume of the low-level representation vector through scalar quantization and variable-length encoding, minimizing bandwidth usage in the communication link between the edge and cloud sides. Furthermore, periodic codebook updates control feature information loss during the quantization process.

[0098] Please refer to the attached document. Figure 4 In another preferred embodiment, the edge device selects the corresponding transmission strategy based on the network link status and uploads the hybrid feature data packet to the cloud server. The edge data packaging unit restores the underlying representation vector of the floating-point vector and the Mel frequency cepstral coefficients of the current speech frame, performs a head-to-tail concatenation operation on the feature dimension, and generates a dimension of The mixed feature vector, where D is the feature dimension of the single-frame Mel-frequency cepstral coefficients. This represents the feature dimension of the underlying representation vector. The hybrid feature vectors of multiple consecutive frames are encapsulated into a hybrid feature data packet. The header of the data packet includes the timestamp and sequence number of the corresponding audio frame, used for timing alignment and reordering after reception by the cloud server.

[0099] The current round-trip time (RTT) of the network link between the endpoint and the cloud server is obtained. Specifically, the endpoint's link state detection unit continuously acquires the RTT between the endpoint and the cloud server. Specifically, the link state detection unit periodically sends probe packets to the cloud server, records the sending time of the probe packets and the receiving time of the corresponding response packets, and calculates the current RTT value. The link state detection unit performs a moving average processing on the RTT values ​​obtained from multiple consecutive measurements to eliminate measurement errors caused by network jitter, obtaining a smoothed current network link RTT.

[0100] When the current network link round-trip latency is less than a preset latency threshold, hybrid characteristic data packets are transmitted through the low-latency channel of the communication link. The low-latency channel uses a preset low-latency transmission protocol, sets the data packet priority to the highest level, and enables a minimized queuing strategy during transmission to reduce queuing and forwarding latency, ensuring that the end-to-end transmission latency of the hybrid characteristic data packets meets real-time requirements.

[0101] When the current network link round-trip latency is greater than or equal to a preset latency threshold, a second downsampling process is performed on the underlying representation vector in the hybrid feature data packet. The downsampling-processed hybrid feature data packet is then transmitted through the ordinary data channel of the communication link. Specifically, the second downsampling process involves performing interval sampling on the feature dimension of the underlying representation vector, retaining a preset proportion of the feature dimension values, reducing the dimension of the underlying representation vector, and generating the downsampling-processed hybrid feature data packet. The ordinary data channel uses a common transmission protocol, supports packet loss retransmission and flow control mechanisms, and is adapted to high-latency, low-bandwidth network link environments to ensure reliable transmission of the hybrid feature data packet.

[0102] Please refer to the attached document. Figure 5 The cloud-side server inputs the received low-level representation vector into the attention module of the cloud-side multilingual acoustic model as a positional prior bias. Within the attention module of the cloud-side multilingual acoustic model, the query vector generated by linearly mapping the Mel-frequency cepstral coefficients of the current speech frame is used as the basic query matrix. The receiving unit of the cloud-side server performs parsing processing on the received mixed feature data packets, separating the Mel-frequency cepstral coefficients and the low-level representation vector of the current speech frame. The Mel-frequency cepstral coefficients are input into the encoder of the cloud-side multilingual acoustic model. The encoder performs multi-layer feature transformation processing on the Mel-frequency cepstral coefficients, outputting an encoder feature matrix. The encoder feature matrix is ​​then linearly mapped to generate a query vector, an original key vector, and a value vector, with the query vector serving as the basic query matrix.

[0103] The key vector, generated by linear mapping of the underlying representation vector, is multiplied by the base query matrix to produce a biased attention score matrix. The separated underlying representation vector is then input into an independent linear mapping layer to generate a key vector matching the dimensions of the base query matrix. This key vector is then multiplied by the base query matrix to produce the biased attention score matrix. The calculation process is defined by the following formula:

[0104]

[0105] in, This is the biased attention score matrix; The basic query matrix is ​​generated by processing the Mel frequency cepstral coefficients of the current speech frame through an encoder and a linear mapping. The key vector is generated by linear mapping of the underlying representation vector; superscript This is a matrix transpose operation.

[0106] The biased attention score matrix is ​​superimposed on the original attention score matrix of the cloud-based multilingual acoustic model to generate the superimposed attention score matrix. The original attention score matrix of the cloud-based multilingual acoustic model is obtained by the dot product of the base query matrix and the original key vector generated by the encoder feature mapping. Before the superposition operation, the historical attention entropy value output by the cloud-based multilingual acoustic model at the previous decoding time is obtained, and the dynamic temperature scaling factor is calculated based on the historical attention entropy value, where the historical attention entropy value and the dynamic temperature scaling factor are negatively correlated. The calculation process of the dynamic temperature scaling factor is defined by the following formula:

[0107]

[0108] in, This is the dynamic temperature scaling factor; This is the minimum preset scaling factor; This is the maximum preset scaling factor; This is the historical attention entropy value output at the previous decoding time step; This is a preset entropy threshold value; It is a natural constant.

[0109] A dynamic temperature scaling factor is used to multiply and scale each element of the bias attention score matrix to generate a scaled bias matrix. The scaled bias matrix is ​​then added element-wise to the original attention score matrix to output the superimposed attention score matrix. The dynamic temperature scaling factor adjusts the strength of the bias attention score matrix. When the historical attention entropy value is high, it indicates that the attention distribution at the previous decoding time step is relatively dispersed; reducing the dynamic temperature scaling factor weakens the effect of the bias attention score matrix and avoids feature information loss due to over-constraint. Conversely, when the historical attention entropy value is low, it indicates that the attention distribution at the previous decoding time step is relatively concentrated; increasing the dynamic temperature scaling factor enhances the effect of the bias attention score matrix and strengthens the constraint effect on the search space.

[0110] A masking operation and Softmax normalization are performed on the superimposed attention score matrix to generate a priori position biases. An upper triangular masking operation is then performed on the superimposed attention score matrix to mask the attention scores corresponding to speech frames after the current decoding time, preventing future information leakage and ensuring the causality of streaming decoding. Subsequently, Softmax normalization is performed on the masked superimposed attention score matrix to generate a priori position biases. The calculation process is defined by the following formula:

[0111]

[0112] in, The generated position is a priori bias; This is the attention score matrix after stacking; This is an upper triangular masking operation used to mask attention scores at future time steps; This is a normalized activation function that maps the elements of the input matrix to values ​​between 0 and 1, with the sum of all elements being 1.

[0113] The cloud-based multilingual acoustic model narrows the search space for acoustic candidate states under the constraint of a prior position bias. It decodes the Mel-frequency cepstral coefficients of the current speech frame and outputs the recognized text. The prior position bias is applied to the acoustic output posterior probability distribution of the cloud-based multilingual acoustic model. The probability values ​​of candidate phonemes below a preset probability threshold in the acoustic output posterior probability distribution are zeroed out, generating a truncated posterior probability distribution. A weighted summation operation is performed between the prior position bias and the acoustic feature matrix output by the encoder to generate a constrained acoustic feature matrix. This constrained acoustic feature matrix is ​​input into the output layer of the decoder, mapping to generate the acoustic output posterior probability distribution. The dimension of the acoustic output posterior probability distribution is consistent with the size of the preset multilingual phoneme set, with each element corresponding to the output probability of a candidate phoneme. The acoustic output posterior probability distribution is truncated, a process defined by the following formula:

[0114]

[0115] in, To truncate candidate phonemes in the posterior probability distribution The corresponding probability value; Candidate phonemes in the posterior probability distribution of the original acoustic output The corresponding probability value; This is a preset probability threshold.

[0116] Please refer to the attached document. Figure 6 The truncated posterior probability distribution is input into a pre-constructed weighted finite-state converter (WGSG), which integrates a multilingual pronunciation dictionary and a multilingual language model. The WGSG employs a four-layer cascaded structure: layer H is a Hidden Markov Model (HMM) topology layer, corresponding to the state transition relationships of phonemes; layer C is a context-dependent phoneme mapping layer, mapping context-independent phonemes to context-dependent phonemes; layer L is a multilingual pronunciation dictionary layer, storing the phoneme-to-word mapping relationships for all preset languages; and layer G is a multilingual language model layer, employing an n-gram grammar language model to represent the sequence transition probabilities between words. These four layers are fused into a single decoding network through the composite operation of the WGSG, achieving end-to-end mapping from acoustic phonemes to text sequences.

[0117] Within the weighted finite-state converter, a Viterbi pruning search is performed along the active path on the truncated posterior probability distribution to output the recognized text. During the Viterbi pruning search, the cumulative path score corresponding to each state is calculated frame by frame. The calculation process of the cumulative path score is defined by the following formula:

[0118]

[0119] in, The cumulative path score is the value corresponding to the j-th state at time t. The cumulative path score is the value of the i-th state at time t-1. Let be the transition probability from the i-th state to the j-th state; Input the observation value for the j-th state at time t. The output probability; This is for retrieving the maximum value.

[0120] At each word-ending node of the weighted finite-state converter, a candidate word grid is constructed. The candidate word grid contains all candidate paths reaching that node and their corresponding cumulative path scores. Within the candidate word grid, the difference between the current global maximum path score and a preset retention threshold is calculated as the path retention threshold. Candidate paths with cumulative path scores greater than the retention threshold are retained, while candidate paths with cumulative path scores less than the retention threshold are truncated, and the memory resources corresponding to the truncated paths are released. The word-ending hidden states of the retained candidate paths are used as the starting hidden states for Viterbi pruning search in the weighted finite-state converter for the next speech frame. This decoding and pruning process is repeated until the speech stream ends. A backtracking operation is then performed along the finally retained optimal path to generate the corresponding recognized text, which is then transmitted back to the end device via a communication link.

[0121] Table 4. Mapping relationships and state constraints at each level of the weighted finite state converter.

[0122] Hierarchical identifier level name Input elements Output elements State constraint rules H layer Hidden Markov Model Topology Layer Context-dependent phonemes HMM state sequence Each phoneme corresponds to a 3-state HMM topology, allowing only forward transitions between states. C layer Context-dependent phoneme mapping layer Context-independent phonemes Context-dependent phonemes A triphone context model is used, constraining the context window to one phoneme on each side. L layer Multilingual pronunciation dictionary layer Phoneme sequence Vocabulary Unit Only phoneme-to-word mappings existing in preset multilingual pronunciation dictionaries are allowed. G layer Multilingual language model layer Word sequence text sequence A ternary grammar language model is used to constrain the transition probability threshold of word sequences.

[0123] This table clarifies the mapping relationships and state constraint rules of the four levels of the weighted finite state converter (HCLG). Through a hierarchical cascaded structure, it completes the mapping from acoustic phonemes to text sequences. At the same time, through the state constraint rules of each level, it restricts invalid state transition paths, provides basic state space constraints for Viterbi pruning search, and further narrows the search range in the decoding process.

[0124] In this embodiment, the endpoint selects the corresponding transmission channel and transmission strategy based on the round-trip latency of the current network link. In high-latency network environments, it performs secondary downsampling on the underlying representation vector to adapt to different network link states. The cloud side performs a dot product operation between the key vector generated by the linear mapping of the underlying representation vector and the basic query matrix to generate a bias attention score matrix. Based on the historical attention entropy value at the previous decoding time, it calculates a dynamic temperature scaling factor, performs scaling on the bias attention score matrix, and then superimposes it with the original attention score matrix. After masking and normalization, it generates a priori position bias.

[0125] A prior positional bias is applied to the acoustic output posterior probability distribution. Candidate phoneme probabilities below a threshold are zeroed out, generating a truncated posterior probability distribution. This truncated distribution is then input into a weighted finite-state converter that integrates a multilingual pronunciation dictionary and a language model. Viterbi pruning search is performed within the converter, filtering active paths using a path retention threshold from the candidate word grid. Finally, the recognized text is generated by backtracking. This embodiment adjusts the strength of the prior positional bias using a dynamic temperature scaling factor. Combined with acoustic posterior probability truncation and Viterbi pruning, this further reduces the acoustic candidate state search space during cloud-side decoding, decreasing cloud-side computational resource consumption and decoding response latency.

Claims

1. A multilingual speech recognition method for online and offline collaboration, characterized in that, include: The end-side acoustic front end continuously extracts the Mel frequency cepstral coefficients of the current speech frame and inputs the Mel frequency cepstral coefficients into the end-side lightweight language classification network to obtain the language confidence vector; Determine whether the maximum value in the language confidence vector exceeds a preset confidence threshold; If the maximum value exceeds the preset confidence threshold, the edge multilingual acoustic model performs streaming decoding on the current speech frame and outputs the recognized text. If the maximum value does not exceed the preset confidence threshold, the bottom-level representation vector of the current speech frame output by the first convolutional layer of the acoustic model on the edge is extracted, the bottom-level representation vector is concatenated with the Mel frequency cepstral coefficients of the current speech frame, and uploaded to the cloud server through the communication link. The cloud-side server inputs the received underlying representation vector into the attention module of the cloud-side multilingual acoustic model as a positional prior bias. Under the constraint of the positional prior bias, the cloud-side multilingual acoustic model narrows the search space of acoustic candidate states, decodes the Mel frequency cepstral coefficients of the current speech frame, and outputs the recognized text.

2. The online / offline collaborative multilingual speech recognition method according to claim 1, characterized in that, The step of obtaining the language confidence vector includes: concatenating the Mel frequency cepstral coefficients of the current speech frame with the Mel frequency cepstral coefficients of the previous N historical speech frames cached in the end-side circular buffer to generate a concatenated spectral feature matrix, where N is a positive integer; The spliced ​​spectral feature matrix is ​​input into the multi-head self-attention layer of the edge-side lightweight language classification network to extract the inter-frame correlation weights of the spliced ​​spectral feature matrix in the time dimension. The spliced ​​spectral feature matrix carrying the inter-frame correlation weights is input into the fully connected output layer of the edge-side lightweight language classification network, and mapped to generate the language confidence vector containing each preset language dimension.

3. The online / offline collaborative multilingual speech recognition method according to claim 1, characterized in that, The step of extracting the underlying representation vector includes: taking the mean pooling result of the convolutional feature map output by the first convolutional layer of the end-side acoustic model in the channel dimension as the initial representation vector; Perform scalar quantization operation based on a preset quantization codebook on the initial representation vector, and map the initial representation vector to the index sequence of the corresponding codeword in the preset quantization codebook; The index sequence is variable-length encoded to generate the underlying representation vector in binary bitstream form; Before concatenating the underlying representation vector with the Mel frequency cepstral coefficients of the current speech frame, the underlying representation vector in binary bitstream form is restored to a floating-point vector.

4. The online / offline collaborative multilingual speech recognition method according to claim 1, characterized in that, The step of uploading to the cloud server via the communication link includes: concatenating the underlying representation vector and the Mel frequency cepstral coefficients of the current speech frame in the feature dimension to generate a hybrid feature data packet; Obtain the current round-trip latency of the network link between the endpoint and the cloud server; When the round-trip time of the current network link is less than a preset delay threshold, the hybrid feature data packet is transmitted through the low-latency channel of the communication link. When the round-trip time of the current network link is greater than or equal to the preset delay threshold, the underlying representation vector in the hybrid feature data packet is subjected to secondary downsampling processing, and the hybrid feature data packet after secondary downsampling processing is transmitted through the ordinary data channel of the communication link.

5. The online / offline collaborative multilingual speech recognition method according to claim 1, characterized in that, The step of inputting the received low-level representation vector into the attention module of the cloud-side multilingual acoustic model as a positional prior bias includes: within the attention module of the cloud-side multilingual acoustic model, using the query vector generated by linear mapping of the Mel frequency cepstral coefficients of the current speech frame as the basic query matrix. The key vector generated by linear mapping of the underlying representation vector is multiplied by the basic query matrix to generate the bias attention score matrix. The biased attention score matrix is ​​superimposed on the original attention score matrix of the cloud-side multilingual acoustic model to generate the superimposed attention score matrix. The superimposed attention score matrix is ​​subjected to masking and Softmax normalization to generate the position prior bias.

6. The online / offline collaborative multilingual speech recognition method according to claim 1, characterized in that, The steps of the cloud-side multilingual acoustic model to narrow the search space of acoustic candidate states and perform decoding under the constraint of the position prior bias include: applying the position prior bias to the acoustic output posterior probability distribution of the cloud-side multilingual acoustic model, setting the probability values ​​corresponding to candidate phonemes below a preset probability threshold in the acoustic output posterior probability distribution to zero and generating a truncated posterior probability distribution. The truncated posterior probability distribution is input into a pre-constructed weighted finite-state converter, which integrates a multilingual pronunciation dictionary and a multilingual language model. Inside the weighted finite state converter, a Viterbi pruning search is performed along the active path on the truncated posterior probability distribution to output the recognized text.

7. The online / offline collaborative multilingual speech recognition method according to claim 2, characterized in that, In the step of extracting the inter-frame correlation weights of the spliced ​​spectral feature matrix in the time dimension, a language relative position encoding matrix is ​​introduced inside the multi-head self-attention layer of the end-side lightweight language classification network. Calculate the first element in the spliced ​​spectral feature matrix. Frame and the The absolute time difference between frames is mapped to the corresponding vector in the language relative position encoding matrix; Add the corresponding vector to the key vector of the spliced ​​spectral feature matrix to generate a bias key vector; An attention score is calculated using the bias key vector and the query vector of the concatenated spectral feature matrix. The inter-frame association weight is then determined using the attention score. and All are positive integers and Not equal to .

8. The online / offline collaborative multilingual speech recognition method according to claim 3, characterized in that, In the step of performing scalar quantization operation based on a preset quantization codebook on the initial representation vector, the preset quantization codebook is maintained on the terminal side. The preset quantization codebook is periodically updated through the following steps: obtaining multiple historical initial representation vectors corresponding to multiple historical speech frames in the current period; Calculate the cluster center of the multiple historical initial representation vectors, and remove the codewords in the current preset quantization codebook that are farthest from the cluster center; Insert the vector corresponding to the cluster center at the removal position to generate the updated preset quantization codebook; When performing scalar quantization, the Euclidean distance between the initial representation vector and each codeword in the updated preset quantization codebook is calculated, and the codeword with the smallest Euclidean distance is selected as the quantization result.

9. The online / offline collaborative multilingual speech recognition method according to claim 5, characterized in that, In the step of superimposing the bias attention score matrix onto the original attention score matrix of the cloud-side multilingual acoustic model to generate the superimposed attention score matrix, the historical attention entropy value output by the cloud-side multilingual acoustic model at the previous decoding time is obtained. A dynamic temperature scaling factor is calculated based on the historical attention entropy value, wherein the historical attention entropy value is negatively correlated with the dynamic temperature scaling factor. The elements in the bias attention score matrix are multiplied and scaled using the dynamic temperature scaling factor to generate a scaled bias matrix. The scaled bias matrix is ​​added element-wise to the original attention score matrix to output the superimposed attention score matrix.

10. The online / offline collaborative multilingual speech recognition method according to claim 6, characterized in that, In the step of performing Viterbi pruning search on the truncated posterior probability distribution along the active path inside the weighted finite state converter, a candidate word grid is constructed at each word-ending node of the weighted finite state converter. In the candidate word grid, candidate paths whose cumulative path score is greater than the difference between the current global maximum path score and the preset retention difference are retained, and candidate paths in the candidate word grid whose cumulative path score is less than the difference between the current global maximum path score and the preset retention difference are truncated. The word-ending hidden states of the retained candidate paths are used as the starting hidden states for Viterbi pruning search in the weighted finite state converter for the next speech frame until the speech stream ends, and the recognized text is generated by backtracking along the optimal retained path.

Images