Cantonese-english mixed language speech recognition method, system, device and storage medium

Abstract

This invention provides a method, system, device, and storage medium for Cantonese-English mixed speech recognition. The method includes: determining the language of an audio file to be recognized; when the determination result is Cantonese-English mixed audio, collecting the Cantonese-English mixed audio data; establishing a Cantonese-English mixed pronunciation dictionary; establishing a training sample set in conjunction with the Cantonese-English mixed audio data; and inputting the training sample set into a speech recognition model for training, and saving the trained Cantonese-English mixed speech recognition model. This invention can effectively improve the service experience for Cantonese users in online travel agency scenarios, quickly and accurately transcribe Cantonese-English mixed audio content, assist customer service and downstream tasks in timely identifying inherent defects in information, save operational and maintenance costs, and ensure the timeliness and accuracy of services.

Description

Technical Field

[0001] This invention relates to the field of speech recognition, and more specifically, to a method, system, device, and storage medium for Cantonese-English mixed speech recognition. Background Technology

[0002] With the rapid development of artificial intelligence technology, robots are becoming increasingly intelligent. In the OTA (Online Travel Agency) field, intelligent outbound call robots are particularly important. Voice interaction has strong advantages in tasks such as intelligent dialogue, automatic dialing, intelligent analysis, and 24 / 7 high service efficiency. This greatly reduces the workload of customer service personnel and improves service quality. Among these, speech recognition is a crucial part of voice interaction, and the performance of the speech recognition system determines the performance of downstream tasks. Currently, end-to-end speech recognition frameworks are the mainstream, leveraging the advantages of big data and deep learning to improve the recognition rate of models, such as speech recognition models based on Transformer and Conformer. However, the multi-head attention (MHA) modules they use share common drawbacks, including quadratic time complexity, high memory consumption, slow convergence speed, and excessive parameters during computation. Furthermore, there is relatively little research on end-to-end speech recognition for Cantonese, as Cantonese words often involve a mix of Cantonese and English.

[0003] Therefore, the present invention provides a method, system, device and storage medium for Cantonese-English mixed speech recognition. Summary of the Invention

[0004] To address the problems in existing technologies, the present invention aims to provide a Cantonese-English mixed speech recognition method, system, device, and storage medium, overcoming the difficulties of existing technologies. The present invention can effectively improve the service experience of Cantonese users in online travel agency scenarios, quickly and accurately transcribe Cantonese-English mixed audio content, assist customer service and downstream tasks in timely discovering inherent defects in information, save operation and maintenance costs, and ensure the timeliness and accuracy of services.

[0005] An embodiment of the present invention provides a Cantonese-English mixed speech recognition method, comprising the following steps:

[0006] The language of the audio to be identified is determined. When the determination result is a mixed Cantonese and English audio, the mixed Cantonese and English audio data is collected.

[0007] Establish a Cantonese-English mixed pronunciation dictionary, and jointly establish a training sample set with the aforementioned Cantonese-English mixed audio data; and

[0008] The training sample set is input into the speech recognition model for training, and the trained Cantonese-English mixed speech recognition model is saved.

[0009] Preferably, the step of determining the language of the audio to be identified, and when the determination result is a mixed Cantonese-English audio, collecting the mixed Cantonese-English audio data, includes:

[0010] The audio to be identified is segmented into words, and the language of each segment is determined.

[0011] Calculate the proportion of the audio corresponding to the specified language to the audio to be identified;

[0012] When the ratio of Cantonese and English simultaneously reaches a preset threshold, the audio to be identified is determined to be a mixed Cantonese-English audio; and

[0013] If the result is a mixed Cantonese-English audio, the mixed Cantonese-English audio data is collected; otherwise, the audio is discarded.

[0014] Preferably, the step of establishing a Cantonese-English mixed pronunciation dictionary and establishing a training sample set in conjunction with the Cantonese-English mixed audio data includes:

[0015] Remove the mixed Cantonese-English audio data that does not conform to the model training specifications, and label the mixed Cantonese-English audio data that conforms to the model training specifications;

[0016] The acoustic features of the Cantonese-English mixed audio dataset are extracted, and data augmentation is performed on the acoustic features; and

[0017] A training sample set is jointly established by the Cantonese-English mixed audio dataset and the Cantonese-English mixed pronunciation dictionary.

[0018] Preferably, the step of inputting the training sample set into the speech recognition model for training and saving the trained Cantonese-English mixed speech recognition model includes:

[0019] The training sample set is input into the Conformer speech recognition model based on the T-MHA algorithm to train the speech recognition model;

[0020] Save the trained speech recognition model to obtain a Cantonese-English mixed speech recognition model.

[0021] Preferably, the Conformer speech recognition model based on the T-MHA algorithm includes:

[0022] A Conformer encoder comprises several sub-modules, including a T-MHA module, a convolutional module, and a position-feedforward neural network module. Normalization and residual connections are performed before and after each sub-module, respectively. The Conformer encoder can be represented by the following formula:

[0023]

[0024]

[0025]

[0026]

[0027] Where i represents the i-th time. Let represent the speech information at time i, and FFN denote a feedforward neural network. This represents the information at time i of the input speech, and T-MHA represents a multi-head attention mechanism based on a time-delay neural network. Let represent the speech information at time i after T-MHA, and Conv represent the convolutional module. Let represent the speech information at time i after convolution, and LayerNorm represent the normalization function. This represents the speech information at time i after passing through LayerNorm;

[0028] A concatenated temporal classification decoder, comprising fully connected layers and a softmax layer; and

[0029] An Attention decoder, comprising a T-MHA module and a location-feedforward neural network model.

[0030] Preferably, in the Conformer speech recognition model based on the T-MHA algorithm, the T-MHA algorithm replaces the linear layer in the original multi-head attention structure with a time-delay neural network.

[0031] assumed , m represents the dimension of feature x, Let x represent the total number of frames for feature x. Then, the formula for calculating T-MHA is as follows:

[0032]

[0033] Where i represents the i-th neuron in the input layer. This represents the j-th neuron in the output layer. This represents the output of the i-th channel neuron corresponding to the feature position in frame t. Let represent the speech features from time t, with frame window length i, to time t+i. Represents the TDNN parameter matrix. Let W represent the weights of the W matrix that connects the i-th input neuron and outputs the j-th neuron.

[0034] The formula for calculating the self-attention mechanism is as follows:

[0035]

[0036]

[0037] in, Used for scale adjustment, where i represents the i-th head, X represents the input speech features, V represents the speech feature vector after passing through the TDNN, Q represents the speech feature vector after passing through the TDNN, and K represents the speech feature vector after passing through the TDNN. V represents the speech feature vector V of the i-th head after passing through the TDNN. Let Q represent the speech feature vector of the i-th head after passing through the TDNN. K represents the speech feature vector of the i-th head after passing through the TDNN. Represents the TDNN parameter matrix. This represents the weight matrix of the Q-feature vector corresponding to the i-th head. This represents the weight matrix of the K eigenvectors corresponding to the i-th head. This indicates that the i-th head speech feature vector is transposed, where T represents the meaning of transpose. This indicates that the i-th head speech feature vector is transposed, where T represents the meaning of transpose. This indicates that V is multiplied by a corresponding weight based on the level of attention received;

[0038] The formula for calculating multi-head attention is as follows, and multi-head attention information is then concatenated:

[0039]

[0040]

[0041] in, This represents the h-th head. This represents a fully connected vector matrix. Finally, the vector dimensions of the matrices after concatenation are calculated to ensure consistency. This indicates connecting the first header to the h-th header. This represents the result of the Multihead function calculation. This indicates the calculation method for the i-th head.

[0042] Preferably, the Conformer speech recognition model based on the T-MHA algorithm uses a joint objective loss function that connects the temporal classification loss function and the cross-entropy loss function. The model is iterated until the training loss converges and the test performance on the validation set reaches its optimal value, as shown in the following formula:

[0043]

[0044] Where λ represents the weight of the CTC loss value in the joint objective loss function loss value, and its value ranges from [0, 1]. This represents the loss value of the CTC loss function. This represents the loss value of the CE loss function. This represents the loss value of the joint objective loss function.

[0045] Embodiments of the present invention also provide a Cantonese-English mixed speech recognition system for implementing the above-described Cantonese-English mixed speech recognition method, wherein the Cantonese-English mixed speech recognition system includes:

[0046] The data acquisition module determines the language of an audio file to be identified. When the determination result is a mixed Cantonese and English audio file, it acquires the mixed Cantonese and English audio data.

[0047] The sample building module establishes a Cantonese-English mixed pronunciation dictionary and, together with the Cantonese-English mixed audio data, builds a training sample set.

[0048] The model training module inputs the training sample set into the speech recognition model for training and saves the trained Cantonese-English mixed speech recognition model.

[0049] Embodiments of the present invention also provide a Cantonese-English mixed speech recognition device, comprising:

[0050] processor;

[0051] A memory in which executable instructions of the processor are stored;

[0052] The processor is configured to execute the steps of the above-described Cantonese-English mixed speech recognition method by executing the executable instructions.

[0053] Embodiments of the present invention also provide a computer-readable storage medium for storing a program, which, when executed, implements the steps of the above-described Cantonese-English mixed speech recognition method.

[0054] The purpose of this invention is to provide a method, system, device, and storage medium for Cantonese-English mixed speech recognition, which can effectively improve the service experience of Cantonese users in online travel agency scenarios, quickly and accurately transcribe Cantonese-English mixed audio content, assist customer service and downstream tasks in timely discovering inherent defects in information, save operation and maintenance costs, and ensure the timeliness and accuracy of services. Attached Figure Description

[0055] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0056] Figure 1 This is a flowchart of the Cantonese-English mixed speech recognition method of the present invention.

[0057] Figure 2 This is a flowchart of data preparation according to an embodiment of the present invention.

[0058] Figure 3 This is a flowchart of audio feature extraction according to an embodiment of the present invention.

[0059] Figure 4 This is a flowchart of a Conformer speech recognition model based on T-MHA according to an embodiment of the present invention.

[0060] Figure 5 This is a schematic diagram of the modules of the Cantonese-English hybrid speech recognition system of the present invention.

[0061] Figure 6 This is a schematic diagram of the structure of the Cantonese-English hybrid speech recognition device of the present invention.

[0062] Figure 7 This is a schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention. Detailed Implementation

[0063] The following specific examples illustrate the implementation methods of this application. Those skilled in the art can easily understand the other advantages and effects of this application from the content disclosed herein. This application can also be implemented or applied through other different specific embodiments, and various details in this application can be modified or changed according to different viewpoints and application systems without departing from the spirit of this application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0064] The embodiments of this application will now be described in detail with reference to the accompanying drawings, so that those skilled in the art can easily implement the application. This application may be embodied in many different forms and is not limited to the embodiments described herein.

[0065] In this application, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics represented in connection with that embodiment or example, which are included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics represented may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate different embodiments or examples represented in this application, as well as features of different embodiments or examples.

[0066] Furthermore, the terms "first" and "second" are used for illustrative purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the representation of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0067] For the purpose of clearly describing this application, devices that are not relevant to the description are omitted, and the same or similar components throughout the specification are given the same reference numerals.

[0068] Throughout this specification, when it is said that a device is "connected" to another device, this includes not only "direct connection" but also "indirect connection" by placing other components in between. Furthermore, when it is said that a device "comprises" a certain constituent element, unless otherwise stated otherwise, this does not exclude other constituent elements, but rather implies that other constituent elements may be included.

[0069] When we say that a device is "above" another device, this can mean that it is directly above the other device, or it can mean that other devices are present in between. Conversely, when we say that a device is "directly" "above" another device, there are no other devices present in between.

[0070] Although the terms first, second, etc., are used in some instances herein to refer to various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, first interface and second interface, etc., are used. Furthermore, as used herein, the singular forms “a,” “an,” and “the” are intended to also include the plural forms unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of features, steps, operations, elements, components, items, kinds, and / or groups, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, kinds, and / or groups. The terms “or” and “and / or” as used herein are interpreted as inclusive, or mean any one or any combination thereof. Thus, “A, B, or C” or “A, B, and / or C” means “any one of: A; B; C; A and B; A and C; B and C; A, B, and C.” Exceptions to this definition will only occur if the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0071] The technical terms used herein are for reference only to specific embodiments and are not intended to limit the scope of this application. The singular form used herein includes the plural form unless the statement explicitly indicates otherwise. The word "comprising" as used in the specification means to specify a particular characteristic, region, integer, step, operation, element, and / or component, and does not exclude the presence or addition of other characteristics, regions, integers, steps, operations, elements, and / or components.

[0072] Although not explicitly defined, all terms, including technical and scientific terms used herein, shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. Terms defined in commonly used dictionaries shall be further interpreted as having a meaning consistent with the relevant technical literature and the content of this present application, and shall not be over-interpreted as having an ideal or overly formulaic meaning unless otherwise defined.

[0073] Figure 1 This is a flowchart of the Cantonese-English mixed speech recognition method of the present invention. Figure 1 As shown, the Cantonese-English mixed speech recognition method of the present invention includes the following steps:

[0074] S110. Determine the language of the audio to be identified. When the determination result is a mixed Cantonese and English audio, collect the mixed Cantonese and English audio data.

[0075] S120. Establish a Cantonese-English mixed pronunciation dictionary and create a training sample set using Cantonese-English mixed audio data; and

[0076] S130. Input the training sample set into the speech recognition model for training, and save the trained Cantonese-English mixed speech recognition model.

[0077] In a preferred embodiment, S110 includes:

[0078] S111. Segment the audio to be recognized into words and determine the language of each word segment;

[0079] S112. Calculate the proportion of audio corresponding to a particular language to the audio to be identified;

[0080] S113. When the ratio of Cantonese and English simultaneously reaches a preset threshold, the audio to be identified is determined to be a mixed Cantonese-English audio; and

[0081] S114. When the judgment result is a mixed Cantonese and English audio, collect the mixed Cantonese and English audio data; otherwise, discard the audio.

[0082] In a preferred embodiment, S120 includes:

[0083] S121. Remove mixed Cantonese and English audio data that does not conform to the model training specifications, and label mixed Cantonese and English audio data that conforms to the model training specifications.

[0084] S122. Extract acoustic features from the mixed Cantonese and English audio dataset and perform data augmentation on the acoustic features; and

[0085] S123. Jointly establish a training sample set by combining the Cantonese-English mixed audio dataset and the Cantonese-English mixed pronunciation dictionary.

[0086] In a preferred embodiment, S130 includes:

[0087] S131. Input the training sample set into the Conformer speech recognition model based on the T-MHA algorithm to train the speech recognition model;

[0088] S132. Save the trained speech recognition model to obtain the Cantonese-English mixed speech recognition model.

[0089] In a preferred embodiment, the Conformer speech recognition model based on the T-MHA algorithm includes:

[0090] A Conformer encoder consists of several sub-modules, including a T-MHA module, a convolutional module, and a position-feedforward neural network module. Normalization and residual connections are performed before and after each sub-module, respectively. The Conformer encoder can be represented by the following formula:

[0091]

[0092]

[0093]

[0094]

[0095] Where i represents the i-th time. Let represent the speech information at time i, and FFN denote a feedforward neural network. This represents the information at time i of the input speech, and T-MHA represents a multi-head attention mechanism based on a time-delay neural network. Let represent the speech information at time i after T-MHA, and Conv represent the convolutional module. Let represent the speech information at time i after convolution, and LayerNorm represent the normalization function. This represents the speech information at time i after passing through LayerNorm;

[0096] A concatenated temporal classification decoder, comprising fully connected layers and a softmax layer; and

[0097] An Attention decoder, comprising a T-MHA module and a location-feedforward neural network model.

[0098] In a preferred embodiment, in the Conformer speech recognition model based on the T-MHA algorithm, the T-MHA algorithm replaces the linear layer in the original multi-head attention structure with a time-delay neural network.

[0099] assumed , m represents the dimension of feature x, Let x represent the total number of frames for feature x. Then, the formula for calculating T-MHA is as follows:

[0100]

[0101] Where i represents the i-th neuron in the input layer. This represents the j-th neuron in the output layer. This represents the output of the i-th channel neuron corresponding to the feature position in frame t. Let represent the speech features from time t, with frame window length i, to time t+i. Represents the TDNN parameter matrix. Let W represent the weights of the W matrix that connects the i-th input neuron and outputs the j-th neuron.

[0102] The formula for calculating the self-attention mechanism is as follows:

[0103]

[0104]

[0105] in, Used for scale adjustment, where i represents the i-th head, X represents the input speech features, V represents the speech feature vector after passing through the TDNN, Q represents the speech feature vector after passing through the TDNN, and K represents the speech feature vector after passing through the TDNN. V represents the speech feature vector V of the i-th head after passing through the TDNN. Let Q represent the speech feature vector of the i-th head after passing through the TDNN. K represents the speech feature vector of the i-th head after passing through the TDNN. Represents the TDNN parameter matrix. This represents the weight matrix of the Q-feature vector corresponding to the i-th head. This represents the weight matrix of the K eigenvectors corresponding to the i-th head. This indicates that the i-th head speech feature vector is transposed, where T represents the meaning of transpose. This indicates that the i-th head speech feature vector is transposed, where T represents the meaning of transpose. This indicates that V is multiplied by a corresponding weight based on the level of attention received;

[0106] The formula for calculating multi-head attention is as follows, and multi-head attention information is then concatenated:

[0107]

[0108]

[0109] in, This represents the h-th head. This represents a fully connected vector matrix. Finally, the vector dimensions of the matrices after concatenation are calculated to ensure consistency. This indicates connecting the first header to the h-th header. This represents the result of the Multihead function calculation. This indicates the calculation method for the i-th head.

[0110] In a preferred embodiment, the Conformer speech recognition model based on the T-MHA algorithm uses a joint objective loss function that connects the temporal classification loss function and the cross-entropy loss function. The model is iterated until the training loss converges and the test performance on the validation set reaches its optimal value, as shown in the following formula:

[0111]

[0112] Where λ represents the weight of the CTC loss value in the joint objective loss function loss value, and its value ranges from [0, 1]. This represents the loss value of the CTC loss function. This represents the loss value of the CE loss function. This represents the loss value of the joint objective loss function.

[0113] One specific embodiment of the present invention is as follows:

[0114] First, during the data acquisition phase, the audio to be recognized is analyzed for language. When the result is a mix of Cantonese and English, the mixed Cantonese and English audio data is collected. In the online travel agency scenario, the audio of conversations between customer service personnel and Cantonese-speaking users is collected and recognized in real time to determine whether the user's language is primarily Cantonese, Mandarin, English, or a mix of both. This achieves the purpose of segmentation, avoiding the use of a mixed Cantonese and English speech recognition model for all customers, saving computing power, improving computational efficiency, and simultaneously increasing the quality of model data to improve the accuracy of model judgments.

[0115] Secondly, in the sample establishment phase, a Cantonese-English mixed pronunciation dictionary was built, and a training sample set was established in conjunction with Cantonese-English mixed audio data. A Cantonese-English mixed dictionary was constructed by combining a Cantonese Chinese character dictionary and an English BPE dictionary. A Cantonese speech data quality inspection model was established, including but not limited to audio extraction for Cantonese business scenarios, speech audio length filtering, and detection of whether audio tags conform to standards. A feature extraction model was established to extract Fbank features from the audio and enhance these Fbank features.

[0116] Finally, during the model training phase, the training sample set is input into the speech recognition model for training, and the trained Cantonese-English mixed speech recognition model is saved. This invention proposes an end-to-end speech recognition model based on T-MHA, replacing the linear layers in the original (Multi-Head Attention, MHA) structure with Time Delay Neural Networks (TDNNs). The main model framework includes, but is not limited to, Transformers and Conformers based on T-MHA. To accelerate model training convergence and improve recognition accuracy and robustness, a multi-task learning approach is adopted, introducing Connectionist Temporal Classification (CTC). The model is trained by jointly using the CTC loss function and the attention-based Cross Entropy (CE) loss function, with different loss weights. During decoding, the attention decoder re-scores multiple best predicted text sequences output along the CTC path, thereby improving the model's recognition rate. In a preferred embodiment, the invention will continue to optimize the model in subsequent use. Based on the massive amount of Cantonese audio data from outbound calling scenarios, a trained model was used to transcribe the data, and the accuracy and robustness of the model were tested. Errors in the transcription were analyzed, and the model was further optimized by increasing the amount of training data.

[0117] Figure 2 This is a flowchart of data preparation according to an embodiment of the present invention. Figure 2As shown, a Cantonese and English pronunciation dictionary is constructed to address the problem of the Cantonese speech recognition system's inability to recognize mixed English, and a Cantonese speech data quality inspection model is established to improve the quality of the dataset. The specific implementation is as follows: During data preparation, Cantonese and English pronunciation rules were formulated, and the Cantonese Chinese character and English BPE pronunciation dictionaries were combined as the model's dictionary. Since the dataset was created based on real-world scenarios, but the audio in the Cantonese business scenarios of outbound call centers is quite complex, mainly including four categories: silence, noise, non-human voices, and multiple speakers. Furthermore, Cantonese has many tones, complex pronunciation, and dialectal diversity, so rules need to be set to remove audio that does not conform to the training specifications. First, the audio is segmented using a silence detection algorithm. The segmented audio is then filtered to remove audio that is too short or too long. Then, a perplexity method is used to remove audio containing high noise or unclear speech. Finally, audio that conforms to the rules is labeled.

[0118] Figure 3 This is a flowchart of audio feature extraction according to an embodiment of the present invention. Figure 3 As shown, a feature extraction model is established to extract Fbank acoustic features from audio. The specific implementation is as follows: First, a speech rate perturbation is added to the original audio to increase its diversity. Then, pre-emphasis, framing, and windowing operations are performed, with frames set to 25ms and frame shifts of 10ms. Next, a Fast Fourier Transform (FFT) is used to convert the time domain to the frequency domain, and the frequency domain information is squared. Then, the logarithm is taken after passing through a Mel filter to obtain 80-dimensional Fbank acoustic features. Finally, the SpecAugment algorithm is used to augment the Fbank features. SpecAugment can transform the overfitting problem in model training into an underfitting problem, allowing for mitigation through large networks and long-term training strategies. Furthermore, by masking time and frequency domain information, it can increase the robustness of the model's learning samples, further improving speech recognition performance.

[0119] Figure 4 This is a flowchart of a Conformer speech recognition model based on T-MHA according to an embodiment of the present invention. Figure 4 As shown, this invention proposes an end-to-end speech recognition algorithm based on T-MHA in mixed Cantonese and English scenarios. It replaces the linear layers in the original MHA with TDNN to reduce the number of model parameters and accelerate model convergence. The specific implementation is as follows:

[0120] First, the T-MHA model algorithm is proposed. The T-MHA model algorithm replaces the linear layers in the original MHA structure with TDNN. Based on its unique time-delay module, TDNN can fully consider the potential temporal relationships between features at different time positions, more accurately describing and mapping the relationship between features at the current position and other features. In contrast, the linear layers in the original MHA only consider features from one frame. Furthermore, TDNN uses shared weights during the learning process, reducing parameter computation. Therefore, to achieve faster model training convergence and better improve model recognition accuracy and robustness.

[0121] Applying TDNN to MHA, assuming , m represents the dimension of feature x, Let x represent the total number of frames for feature x. Then, the formula for calculating T-MHA is as follows:

[0122]

[0123] Where i represents the i-th neuron in the input layer. This represents the j-th neuron in the output layer. This represents the output of the i-th channel neuron corresponding to the feature position in frame t. Let represent the speech features from time t, with frame window length i, to time t+i. Represents the TDNN parameter matrix. Let W represent the weights of the W matrix that connects the i-th input neuron and outputs the j-th neuron.

[0124] The formula for calculating the self-attention mechanism is as follows:

[0125]

[0126]

[0127] in, Used for scale adjustment, where i represents the i-th head, X represents the input speech features, V represents the speech feature vector after passing through the TDNN, Q represents the speech feature vector after passing through the TDNN, and K represents the speech feature vector after passing through the TDNN. V represents the speech feature vector V of the i-th head after passing through the TDNN. Let Q represent the speech feature vector of the i-th head after passing through the TDNN. K represents the speech feature vector of the i-th head after passing through the TDNN. Represents the TDNN parameter matrix. This represents the weight matrix of the Q-feature vector corresponding to the i-th head. This represents the weight matrix of the K eigenvectors corresponding to the i-th head. This indicates that the i-th head speech feature vector is transposed, where T represents the meaning of transpose. This indicates that the i-th head speech feature vector is transposed, where T represents the meaning of transpose. This indicates that V is multiplied by a corresponding weight based on the level of attention received;

[0128] The formula for calculating multi-head attention is as follows, and multi-head attention information is then concatenated:

[0129]

[0130]

[0131] in, This represents the h-th head. This represents a fully connected vector matrix. Finally, the vector dimensions of the matrices after concatenation are calculated to ensure consistency. This indicates connecting the first header to the h-th header. This represents the result of the Multihead function calculation. This indicates the calculation method for the i-th head.

[0132] Secondly, the model structure of this invention is proposed. The model framework structure of one embodiment of this invention includes, but is not limited to, a Transformer or Conformer based on T-MHA. Taking the Conformer model framework as an example, Conformer combines the advantages of Transformer and Convolutional Neural Networks (CNNs). Transformer models excel at capturing content-based global information, while CNN models effectively utilize local and edge features. The T-MHA-based Conformer end-to-end speech recognition model mainly includes a Conformer encoder, a CTC decoder, and an Attention decoder. The Conformer encoder contains several sub-modules, each consisting of a T-MHA module, a Convolution (Conv) module, and a Feed-forward Neural Network (FNN) module. Each sub-module undergoes Layer Norm normalization and residual connections before and after its execution.

[0133] The Conformer encoder can be represented by the following formula:

[0134]

[0135]

[0136]

[0137]

[0138] Where i represents the i-th time. Let represent the speech information at time i, and FFN denote a feedforward neural network. This represents the information at time i of the input speech, and T-MHA represents a multi-head attention mechanism based on a time-delay neural network. Let represent the speech information at time i after T-MHA, and Conv represent the convolutional module. Let represent the speech information at time i after convolution, and LayerNorm represent the normalization function. This represents the speech information at time i after passing through LayerNorm;

[0139] The CTC decoder consists of a fully connected layer and a softmax layer;

[0140] The Attention decoder consists of a T-MHA module and an FNN.

[0141] Finally, model training and decoding. This embodiment of the invention employs a multi-task learning approach to train the model, combining the CTC loss function and the CE loss function based on an Attention decoder. By leveraging the advantages of both, different loss weights are used for joint training. The CTC loss function aligns frames at the frame level, while the CE loss function aligns labels at the label level. Using the CTC loss function accelerates model convergence, while training the model using the CE loss function results in slower convergence but improves accuracy compared to CTC. The model is iterated using a joint objective loss function until the training loss converges and the optimal performance is achieved on the validation set, as shown in the following formula:

[0142]

[0143] Where λ represents the weight of the CTC loss value in the joint objective loss function loss value, and its value ranges from [0, 1]. This represents the loss value of the CTC loss function. This represents the loss value of the CE loss function. This represents the loss value of the joint objective loss function.

[0144] Because the model uses both frame-level decoder CTC and label-level decoder CE during training, it can use multiple different decoding methods during inference. The model supports four decoding algorithms:

[0145] (1) Greedy search decoding based on CTC: During the decoding process, the prefix is ​​not merged. The output with the highest probability in each frame is selected, and then the duplicate and "blank" characters are removed according to the rules to obtain the final sequence output;

[0146] (2) Prefix bundle search decoding based on CTC: During the decoding process, the same sequence prefixes are merged, multiple best paths are selected, and the path with the highest score is taken as the final output sequence;

[0147] (3) Attention-based beam search decoding: The Attention decoder is based on the encoder output and decodes in an autoregressive manner. This decoding method is label-level decoding, which is different from the above two frame-level decoding methods. There is no prefix merging process. The beam search algorithm selects multiple best paths and takes the path with the highest score as the output sequence.

[0148] (4) Two-pass decoding based on CTC and Attention: In the first decoding, n best prediction results are decoded through CTC prefix beam search. In the second decoding, the Attention decoder uses global Attention context information and, based on the n best prediction results decoded by CTC, re-scores them in a teacher-forcing manner to obtain the final prediction result. Compared with the third decoding method, this method avoids the autoregressive process and can reduce the model decoding latency through parallel computation. Based on the decoding score of Attention, the CTC decoding score is combined and summed with different weights to obtain the final score, thereby further improving the recognition effect.

[0149] In a preferred embodiment, the present invention will continue to optimize the model in subsequent use, addressing its shortcomings by analyzing the results of forward prediction errors. Specifically, the implementation involves: using a large amount of Cantonese audio data from outbound calling scenarios to transcribe the data, testing the model's accuracy and robustness, analyzing errors in the transcription, and further optimizing the model by increasing the amount of training data. In practical implementation, the present invention continuously collects feedback from customer service personnel and periodically repeats the above steps to improve the performance of the deep learning-based speech recognition model.

[0150] Figure 5 This is a schematic diagram of the modules of the Cantonese-English hybrid speech recognition system of the present invention. Figure 5 As shown, embodiments of the present invention also provide a Cantonese-English mixed speech recognition system for implementing the above-described Cantonese-English mixed speech recognition method. The Cantonese-English mixed speech recognition system includes:

[0151] The data acquisition module 51 performs language determination on an audio file to be identified. When the determination result is a mixed Cantonese and English audio file, it acquires the mixed Cantonese and English audio data.

[0152] Sample building module 52 builds a Cantonese-English mixed pronunciation dictionary and builds a training sample set by combining Cantonese-English mixed audio data.

[0153] The model training module 53 inputs the training sample set into the speech recognition model for training and saves the trained Cantonese-English mixed speech recognition model.

[0154] In a preferred embodiment, the data acquisition module 51 is configured to segment an audio to be identified into words and determine the language of each segment; calculate the proportion of audio corresponding to each language to the audio to be identified; when the proportions of Cantonese and English simultaneously reach a preset threshold, determine that the audio to be identified is a mixed Cantonese-English audio; and when the determination result is a mixed Cantonese-English audio, acquire the mixed Cantonese-English audio data, otherwise discard the audio.

[0155] In a preferred embodiment, the sample establishment module 52 is configured to remove Cantonese-English mixed audio data that does not conform to the model training specifications, label Cantonese-English mixed audio data that conforms to the model training specifications, extract acoustic features from the Cantonese-English mixed audio dataset, perform data augmentation on the acoustic features, and jointly establish a training sample set by combining the Cantonese-English mixed audio dataset and the Cantonese-English mixed pronunciation dictionary.

[0156] In a preferred embodiment, the model training module 53 is configured to input the training sample set into the Conformer speech recognition model based on the T-MHA algorithm to train the speech recognition model; and save the trained speech recognition model to obtain a Cantonese-English mixed speech recognition model.

[0157] The Cantonese-English hybrid speech recognition system of the present invention can effectively improve the service experience of Cantonese users in online travel agency scenarios, quickly and accurately transcribe Cantonese-English hybrid audio content, assist customer service and downstream tasks in timely discovering inherent defects in information, save operation and maintenance costs, and ensure the timeliness and accuracy of services.

[0158] This invention also provides a Cantonese-English mixed speech recognition device, including a processor and a memory storing executable instructions for the processor. The processor is configured to execute steps of a Cantonese-English mixed speech recognition method by executing the executable instructions.

[0159] As shown above, the Cantonese-English mixed speech recognition device of this embodiment of the invention can effectively improve the service experience of Cantonese users in online travel agency scenarios, quickly and accurately transcribe Cantonese-English mixed audio content, assist customer service and downstream tasks in timely discovering inherent defects in information, save operation and maintenance costs, and ensure the timeliness and accuracy of services.

[0160] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "platform."

[0161] Figure 6 This is a schematic diagram of the structure of the Cantonese-English hybrid speech recognition device of the present invention. See below for reference. Figure 6 To describe an electronic device 600 according to this embodiment of the present invention. Figure 6 The electronic device 600 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0162] like Figure 6 As shown, the electronic device 600 is presented in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0163] The storage unit stores program code, which can be executed by the processing unit 610 to perform the steps described in the above-described section on the electronic prescription transfer processing method according to various exemplary embodiments of the present invention. For example, the processing unit 610 can perform actions such as... Figure 1 The steps are shown in the figure.

[0164] Storage unit 620 may include readable media in the form of volatile storage units, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include read-only memory (ROM) 6203.

[0165] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0166] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0167] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0168] This invention also provides a computer-readable storage medium for storing a program that, when executed, implements the steps of a Cantonese-English mixed speech recognition method. In some possible implementations, various aspects of this invention can also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps described in the above-described electronic prescription processing method section of this specification according to various exemplary embodiments of the invention.

[0169] As shown above, when the program of the computer-readable storage medium of this embodiment is executed, it can effectively improve the service experience of Cantonese users in the online travel agency scenario, quickly and accurately transcribe mixed Cantonese and English audio content, assist customer service and downstream tasks in timely discovering the inherent defects of information, save operation and maintenance costs, and ensure the timeliness and accuracy of services.

[0170] Figure 7 This is a schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention. (Reference) Figure 7 As shown, a program product 800 for implementing the above-described method according to an embodiment of the present invention is described. It may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0171] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0172] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0173] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0174] In summary, the purpose of this invention is to provide a method, system, device, and storage medium for Cantonese-English mixed speech recognition, which can effectively improve the service experience of Cantonese users in online travel agency scenarios, quickly and accurately transcribe Cantonese-English mixed audio content, assist customer service and downstream tasks in timely discovering inherent defects in information, save operation and maintenance costs, and ensure the timeliness and accuracy of services.

[0175] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A Cantonese-English mixed speech recognition method, characterized in that, Includes the following steps: The language of the audio to be identified is determined. When the determination result is a mixed Cantonese and English audio, the mixed Cantonese and English audio data is collected. Establish a Cantonese-English mixed pronunciation dictionary, and combine the aforementioned Cantonese-English mixed audio data to create a training sample set; as well as The training sample set is input into the Conformer speech recognition model based on the T-MHA algorithm to train the speech recognition model; the trained speech recognition model is saved to obtain a Cantonese-English mixed speech recognition model. The Conformer speech recognition model based on the T-MHA algorithm includes: a Conformer encoder, comprising several sub-modules, including a T-MHA module, a convolutional module, and a positional feedforward neural network module. Normalization and residual connections are performed before and after each sub-module, respectively. The Conformer encoder can be represented by the following formula: Where i represents the i-th time. Let represent the speech information at time i, and FFN denote a feedforward neural network. This represents the information at time i of the input speech, and T-MHA represents a multi-head attention mechanism based on a time-delay neural network. Let represent the speech information at time i after T-MHA, and Conv represent the convolutional module. Let represent the speech information at time i after convolution, and LayerNorm represent the normalization function. This represents the speech information at time i after passing through LayerNorm; A concatenated temporal classification decoder, comprising fully connected layers and a softmax layer; and An Attention decoder, comprising a T-MHA module and a location-feedforward neural network model.

2. The Cantonese-English mixed speech recognition method as described in claim 1, characterized in that, The step of determining the language of an audio file to be identified, and when the determination result is a mixed Cantonese-English audio file, collecting the mixed Cantonese-English audio data, includes: The audio to be identified is segmented into words, and the language of each segment is determined. Calculate the proportion of the audio corresponding to the specified language to the audio to be identified; When the ratio of Cantonese and English simultaneously reaches a preset threshold, the audio to be identified is determined to be a mixed Cantonese-English audio; and If the result is a mixed Cantonese-English audio, the mixed Cantonese-English audio data is collected; otherwise, the audio is discarded.

3. The Cantonese-English mixed speech recognition method as described in claim 1, characterized in that, The establishment of the Cantonese-English mixed pronunciation dictionary, and the creation of a training sample set in conjunction with the Cantonese-English mixed audio data, includes: Remove the mixed Cantonese-English audio data that does not conform to the model training specifications, and label the mixed Cantonese-English audio data that conforms to the model training specifications; The acoustic features of the Cantonese-English mixed audio dataset are extracted, and data augmentation is performed on the acoustic features; and A training sample set is jointly established by the Cantonese-English mixed audio dataset and the Cantonese-English mixed pronunciation dictionary.

4. The Cantonese-English mixed speech recognition method as described in claim 1, characterized in that: In the Conformer speech recognition model based on the T-MHA algorithm, the T-MHA algorithm replaces the linear layer in the original multi-head attention structure with a time-delay neural network; assumed , m represents the dimension of feature x, Let x represent the total number of frames for feature x. Then, the formula for calculating T-MHA is as follows: Where i represents the i-th neuron in the input layer. This represents the j-th neuron in the output layer. This represents the output of the i-th channel neuron corresponding to the feature position in frame t. Let represent the speech features from time t, with frame window length i, to time t+i. Represents the TDNN parameter matrix. Let W represent the weights of the W matrix that connects the i-th input neuron and outputs the j-th neuron. The formula for calculating the self-attention mechanism is as follows: in, Used for scaling, where i represents the i-th head, X represents the input speech features, V represents the speech feature vector after passing through the TDNN, Q represents the speech feature vector after passing through the TDNN, and K represents the speech feature vector after passing through the TDNN. V represents the speech feature vector V of the i-th head after passing through the TDNN. Let Q represent the speech feature vector of the i-th head after passing through the TDNN. K represents the speech feature vector of the i-th head after passing through the TDNN. Represents the TDNN parameter matrix. This represents the weight matrix of the Q-feature vector corresponding to the i-th head. This represents the weight matrix of the K eigenvectors corresponding to the i-th head. This indicates that the i-th head speech feature vector is transposed, where T represents the meaning of transpose. This indicates that the i-th head speech feature vector is transposed, where T represents the meaning of transpose. This indicates that V is multiplied by a corresponding weight based on the level of attention received; The formula for calculating multi-head attention is as follows, and multi-head attention information is then concatenated: in, This represents the h-th head. This represents a fully connected vector matrix. Finally, the vector dimensions of the matrices after concatenation are calculated to ensure consistency. This indicates connecting the first header to the h-th header. This represents the result of the Multihead function calculation. This indicates the calculation method for the i-th head.

5. The Cantonese-English mixed speech recognition method as described in claim 1, characterized in that: The Conformer speech recognition model based on the T-MHA algorithm uses a joint objective loss function that connects the temporal classification loss function and the cross-entropy loss function. The model is iterated until the training loss converges and the test performance on the validation set reaches its optimal value, as shown in the following formula: Where λ represents the weight of the CTC loss value in the joint objective loss function loss value, and its value ranges from [0, 1]. This represents the loss value of the CTC loss function. This represents the loss value of the CE loss function. This represents the loss value of the joint objective loss function.

6. A Cantonese-English mixed speech recognition system, used to implement the Cantonese-English mixed speech recognition method according to claim 1, characterized in that, include: The data acquisition module determines the language of an audio file to be identified. When the determination result is a mixed Cantonese and English audio file, the module acquires the mixed Cantonese and English audio data. The sample building module establishes a Cantonese-English mixed pronunciation dictionary and, together with the Cantonese-English mixed audio data, builds a training sample set. The model training module inputs the training sample set into the Conformer speech recognition model based on the T-MHA algorithm to train the speech recognition model; the trained speech recognition model is saved to obtain a Cantonese-English mixed speech recognition model. The Conformer speech recognition model based on the T-MHA algorithm includes: a Conformer encoder, comprising several sub-modules, including a T-MHA module, a convolutional module, and a positional feedforward neural network module. Normalization and residual connections are performed before and after each sub-module. The Conformer encoder can be represented by the following formula: Where i represents the i-th time. Let represent the speech information at time i, and FFN denote a feedforward neural network. This represents the information at time i of the input speech, and T-MHA represents a multi-head attention mechanism based on a time-delay neural network. Let represent the speech information at time i after T-MHA, and Conv represent the convolutional module. Let represent the speech information at time i after convolution, and LayerNorm represent the normalization function. This represents the speech information at time i after passing through LayerNorm; A concatenated temporal classification decoder, comprising fully connected layers and a softmax layer; and An Attention decoder, comprising a T-MHA module and a location-feedforward neural network model.

7. A Cantonese-English hybrid speech recognition device, characterized in that, include: processor; A memory in which executable instructions of the processor are stored; The processor is configured to perform the steps of the Cantonese-English mixed speech recognition method according to any one of claims 1 to 5 by executing the executable instructions.

8. A computer-readable storage medium for storing a program, characterized in that, When the program is executed by the processor, it implements the steps of the Cantonese-English mixed speech recognition method according to any one of claims 1 to 5.

Images