Pronunciation correlation-based bias method and device, equipment, storage medium and product

Abstract

The invention relates to a bias method and device based on pronunciation correlation, equipment, a storage medium and a product, and relates to the field of speech recognition. The method comprises the following steps: receiving pre-selected characters inferred from a target voice; candidate hot words having pronunciation correlation with the pre-selected characters are determined from pre-constructed character correlation data based on pronunciation correlation with a condition higher than a specific pronunciation similarity threshold value as a screening condition; and offsetting the pre-selected characters according to the candidate hot words.

Description

Technical Field

[0001] This application relates to the field of speech recognition, and in particular to a biasing method, apparatus, device, storage medium, and product based on pronunciation correlation. Background Technology

[0002] In traditional speech recognition, the correspondence between characters and pronunciations mainly relies on a pre-built pronunciation dictionary. When the target speech to be recognized is received, the pre-selected characters are first inferred and used as the bias object. Then, the bias is applied according to the preset hot word weights to guide the recognition result.

[0003] However, with the rapid popularization of speech recognition in many fields and scenarios, different languages, dialects, and various professional terms and new internet words are constantly emerging, which makes it difficult to accurately identify many words with similar pronunciations but different meanings.

[0004] For example, words like "loan" and "bandwidth," or "secondary card" and "lucky card," are all words with similar pronunciations but different meanings. This poses a significant challenge to speech recognition methods that primarily rely on manually constructed pronunciation dictionaries. Due to their poor generalization ability and low efficiency, they are no longer able to meet the increasingly demanding requirements for speech recognition performance. Summary of the Invention

[0005] This application provides a biasing method, apparatus, device, storage medium, and product based on pronunciation similarity. In the process of speech recognition, it uses pronunciation correlation for biasing to improve the accuracy of character recognition and meet the higher demands of speech recognition.

[0006] On the one hand, a bias method based on pronunciation similarity is provided, the method comprising: Receive pre-selected characters inferred from the target speech; From pre-constructed character association data based on pronunciation relevance, candidate hot words that have pronunciation relevance to pre-selected characters are identified using a filtering condition that is higher than a certain pronunciation similarity threshold. Based on the candidate hot words, the pre-selected characters are biased.

[0007] In one implementation, a method for constructing character association data based on pronunciation relevance includes: Based on the first set of labeled speech data, a training dataset is constructed. The training dataset includes multiple sets of training audio pairs in character units, the phoneme sequences corresponding to the two audios in each pair, and the pronunciation similarity between the two audios in each pair. The function of training a pronunciation correlation model using a training dataset and outputting the similarity between audio samples; Based on the labeled second set of speech data, a prediction dataset is constructed, which includes multiple sets of predicted audio pairs in character units; Using a pronunciation correlation model, the pronunciation similarity between characters in the prediction dataset is generated; Based on pronunciation similarity, character association data based on pronunciation relevance is constructed.

[0008] In one implementation, the target speech and the second set of speech data belong to a specific domain of speech.

[0009] In one implementation, the second set contains less voice data than the first set.

[0010] In one implementation, a prediction dataset is constructed based on the labeled second set of speech data, including: The labeled second set of speech data is processed into predicted audio in character units; Construct a set of predicted audio for each character using multiple predicted audios corresponding to the same character; The predicted audio sets corresponding to each pair of characters are combined to construct multiple sets of predicted audio pairs based on characters.

[0011] In one implementation, a training dataset is constructed based on the labeled first set of speech data, including: The labeled first set of speech data is processed into multiple sets of training audio pairs in character units, and the phoneme sequences corresponding to the two audios in each pair. Based on multiple frames in the phoneme sequence, a similarity matrix is ​​generated for the two audio components in each pair, and the pronunciation similarity of each pair is determined based on the similarity matrix. A training dataset is constructed based on the pronunciation similarity of each pair.

[0012] On the other hand, a biasing device based on pronunciation relevance is provided. The device includes: a character receiving unit, a hot word determination unit, and a character biasing unit, wherein... A character receiving unit is used to receive pre-selected characters inferred from the target speech; The hot word determination unit is used to determine candidate hot words that have phonetic relevance to pre-selected characters from pre-constructed character association data based on pronunciation relevance, using a threshold higher than a specific pronunciation similarity threshold as a filtering condition. The character bias unit is used to bias the pre-selected characters based on the candidate hot words.

[0013] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the above-described pronunciation-related bias method.

[0014] On the other hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer program, which is loaded and executed by a processor to implement the above-described bias method based on phonetic relevance.

[0015] On the other hand, a computer program product is provided, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to perform to implement the pronunciation-related bias method provided in the various alternative implementations described above.

[0016] The technical solution provided in this application may include the following beneficial effects: This application provides a biasing method based on pronunciation relevance. It pre-constructs character association data based on pronunciation relevance. After inferring pre-selected characters from the target speech, it filters characters with pronunciation relevance from the character association data, using a threshold of pronunciation similarity as a condition, to select candidate hot words. The pre-selected characters are then biased based on these candidate hot words, thereby guiding the determination of the target speech recognition result. Compared to traditional speech recognition, which directly biases hot words with preset weights after inferring pre-selected characters, this application incorporates pronunciation relevance, filtering out candidate hot words with similar pronunciations and then biasing based on these candidate hot words, thus improving the accuracy of speech recognition.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] Figure 1 A flowchart illustrating a biasing method based on pronunciation similarity provided in an exemplary embodiment of this application is shown. Figure 2 This illustration shows a flowchart of the construction of character association data based on pronunciation relevance provided in an exemplary embodiment of this application; Figure 3A flowchart illustrating the process of constructing a similarity matrix provided in an exemplary embodiment of this application is shown; Figure 4 A schematic diagram of a training pronunciation correlation model provided in an exemplary embodiment of this application is shown; Figure 5 This illustration shows a flowchart of constructing predicted audio provided by an exemplary embodiment of this application; Figure 6 A block diagram of a biasing device based on phonetic similarity provided in an exemplary embodiment of this application is shown; Figure 7 A structural block diagram of a computer device illustrated in an exemplary embodiment of this application is shown. Detailed Implementation

[0020] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of methods and apparatus consistent with some aspects of this application as detailed in the appended claims.

[0021] This application provides a biasing scheme that utilizes pronunciation correlation during speech recognition to improve the accuracy of character recognition and meet the ever-increasing demand for speech recognition.

[0022] Figure 1 The flowchart illustrates a biasing method based on pronunciation similarity provided in an exemplary embodiment of this application. This method can be executed by a computer device, which can be implemented as a server or a terminal device, such as... Figure 1 As shown, the method may include the following steps.

[0023] Step 100: Receive the pre-selected characters inferred from the target speech.

[0024] As mentioned earlier, in the speech recognition process, after receiving the target speech to be recognized, pre-selected characters can be inferred first, and then biased based on hot word weights. For example, after receiving a segment of target speech, the possible characters corresponding to the speech can be inferred through steps such as acoustic feature extraction, deep neural network encoding, language model construction, and decoding optimization, and used as pre-selected characters.

[0025] Specifically, the acoustic feature extraction process can include framing, windowing, and MFCC (Mel-Frequency Cepstral Coefficients) feature extraction, ultimately yielding a multi-dimensional vector sequence, where each vector reflects the tract shape and spectral envelope of the current frame. This achieves the transformation from raw audio to MFCC, with the goal of converting continuous audio signals into computer-processable temporal features while preserving key information about the speech content.

[0026] Subsequently, a deep neural network encoder can be used to achieve a representation from features to abstractions. The goal can be to map MFCC features to a high-dimensional space and capture contextual semantic information. For example, Transformers, CNNs (Convolutional Neural Networks), or RNNs (Recurrent Neural Networks), whose core feature is self-attention, can be used. Of course, the traditional HMM+GMM (Hidden Markov Model + Gaussian Mixture Model) method can also be used to output the context vector at each time step, thereby representing the importance of that frame in the global semantics.

[0027] Next, a language model can be constructed to transform text into a probability distribution. The goal can be to model the statistical regularities of word sequences and constrain the semantic rationality of the decoding results. For example, an n-gram model based on the Markov assumption can be used, or a neural language model using RNN / LSTM (Recurrent Neural Network / Long Short-Term Memory and Transformer-XL (Transformer for Extra Long sequences)). After the above process, the inference result, i.e., the pre-selected characters, can be obtained.

[0028] Of course, in practical applications, character reasoning can also be performed on speech based on other methods.

[0029] Step 110: From the pre-constructed character association data based on pronunciation relevance, candidate hot words that have pronunciation relevance to pre-selected characters are determined using a filtering condition that is higher than a specific pronunciation similarity threshold.

[0030] As described above, with the rapid popularization of speech recognition, there are many words with similar pronunciations but different semantic meanings that are not easily recognized accurately. For example, "loan" and "bandwidth", "sub-card" and "lucky card", "expense" and "phone bill", etc.; also, there are dialects that are rarely heard in some remote areas where the signal coverage was difficult to achieve before, and they also appear in the network or various customer services as the signal coverage has been comprehensively improved. Since they are rarely heard or there is no dedicated person who can understand them, when using the pronunciation dictionary constructed before for recognition, it will affect the accuracy and cause serious inaccuracies.

[0031] Therefore, based on pronunciation correlation, the embodiments of this application pre-construct character association data. During speech recognition, the pronunciation correlation can be used to more accurately find the characters that may correspond. That is, using pronunciation correlation, try to comprehensively find the characters that the speech wants to express, so as to improve the accuracy of speech recognition. For example, as mentioned above, "loan" and "bandwidth", "sub" and "lucky", obviously have strong pronunciation correlations.

[0032] Specifically, character association data can be constructed based on characters that have already been marked with pronunciation features. For example, taking a single character as a unit, according to the pinyin and tones in the electronic dictionary, determine the pronunciation similarity value between characters. For example, for "loan" and "bandwidth", the pinyin and tones are the same, both are dài, so the pronunciation is the same. For "sub-card" and "lucky card", only the tones are different, which belongs to pronunciation similarity. For this, the pronunciation similarity value between each character can be determined through a quantitative method. The specific determination method can be through various distance formulas or neural network methods. For example, the pronunciation similarity value can be normalized to 0 to 1, or 0 to 100, etc. according to the actual situation.

[0033] Then, based on the pronunciation similarity value between each character, construct character association data. For example, it can be a database that stores the pronunciation similarity value between each character, or it can be an N×N matrix. The element in the i-th row and j-th column represents the pronunciation similarity value between the i-th and j-th characters, which can also be called the pronunciation distance. This application is not limited to any form of character association data, as long as it is based on pronunciation correlation.

[0034] Thus, a candidate hot word with pronunciation correlation to the preselected character can be determined by using a specific pronunciation similarity threshold as the screening condition.

[0035] Specifically, a pronunciation similarity threshold can be determined first to indicate the strictness of the screening condition, such as 9 or 95 corresponding to the interval introduced at the front end as the pronunciation similarity threshold, etc. Based on this condition, characters with pronunciation relevance can be screened out. For example, in the previous step, the inferred character is "loan", then in this step, "belt" with a pronunciation similarity value of 0 to it can be screened out. Another example is that in the previous step, the inferred character is "deputy", then in this step, "fortune" may be screened out. After that, in the preset hot word library, candidate hot words with relatively high relevance to the preselected characters can be found.

[0036] Of course, in actual applications, the number of screened candidates can also be preset. For example, using a specific pronunciation similarity threshold as the screening condition, 10 or 20 characters with pronunciation relevance are screened and determined, and then they are respectively matched with the preset hot word library, so that multiple candidate hot words can be determined to improve the recall rate and accuracy.

[0037] In actual applications, according to specific application scenarios, different hot word libraries can be used to determine candidate hot words. For example, in application scenarios such as online customer service, medical, and academic, different hot word libraries can be preset to achieve an accurate biasing effect, that is, to obtain a more accurate recognition result for the target speech. Therefore, in one implementation, the determined candidate hot words can belong to the same speech domain as the target speech. For example, if the target speech is the online customer service of the communication service, the hot word library can be adapted to various contents of customer service.

[0038] Step 120: Bias the preselected characters according to the determined candidate hot words.

[0039] Hotword Biasing can refer to a mechanism in speech recognition to improve the recognition accuracy and probability of a predefined set of specific keywords or phrases. In this step, through the hot word replacement and context guidance mechanism, it can be injected into the decoding path to bias the preselected characters, thereby guiding the recognition result.

[0040] As introduced in the previous step, multiple candidate hot words can be determined. Then in actual applications, multiple candidate hot words can also be used as biasing results to guide the speech recognition result.

[0041] In one implementation, the preselected characters can also be biased according to the determined candidate hot words in combination with the preset hot word biasing weight. Specifically, the single or multiple candidate hot words can be sorted first, and then according to the existing hot word weight preset, the preselected characters can be pre-biased. Finally, the final biasing result is determined by combining the coincidence degree of the pre-biasing result and the candidate hot words.

[0042] In one implementation, after the bias is completed, the preset hot word weights can be adjusted according to the results, thereby continuously improving the hot word weights and making them more dynamic in real time.

[0043] This application embodiment pre-constructs character association data based on pronunciation relevance. After inferring pre-selected characters from the target speech, it filters characters with pronunciation relevance from the character association data based on pronunciation relevance, using a threshold of higher than a specific pronunciation similarity as a condition, and then biases the pre-selected characters according to the candidate hot words, thereby guiding the determination of the recognition result of the target speech. Compared with traditional speech recognition, which directly biases hot words with preset hot word weights after inferring pre-selected characters, this application embodiment incorporates pronunciation relevance, which can filter out candidate hot words with similar pronunciations and then bias them, thereby improving the accuracy of speech recognition.

[0044] As can be seen from the above embodiments, the character association data pre-constructed based on pronunciation relevance is the key to improving character recall rate and speech recognition accuracy. Therefore, the comprehensiveness, accuracy, and real-time performance of character association data become crucial to improving speech recognition accuracy.

[0045] Therefore, to further improve the accuracy of bias and speech recognition, this application provides a method for constructing character association data based on pronunciation relevance. This method can improve the comprehensiveness, accuracy, and real-time performance of the character association data. This method can be executed by a computer device, which can be implemented as a server or a terminal device, such as... Figure 2 As shown, the method may include the following steps.

[0046] Step 200: Construct a training dataset based on the labeled first set of speech data. This training dataset may include multiple sets of training audio pairs (character-by-character), the phoneme sequences corresponding to the two audio elements in each pair, and the pronunciation similarity between the two audio elements in each pair.

[0047] In practical applications, a large amount of labeled speech data can be collected as the first set. The labeled data here can be the correct characters labeled on the speech data, i.e., text annotation, while the speech data can be speech data of different lengths, including characters, words, sentences, etc.

[0048] Phonemes are the basic units of speech. Different languages ​​have their own sets of phonemes. For example, English has 39 phonemes as defined in a pronunciation dictionary, while Chinese has pinyin phonemes (such as zh, ch, ong).

[0049] First, each speech data is transcribed into a corresponding static phoneme sequence. For example, the "bandwidth" mentioned earlier can be transcribed into a static phoneme sequence using Pinyin phonemes. This phoneme sequence includes not only pure phoneme information, i.e., static phoneme sequence, but also dynamic time series, i.e., the start time and end time of each phoneme in the audio.

[0050] Then, the MFA (Montreal Forced Aligner) can be used to align phoneme sequences and speech data. In practical applications, the entire speech data can be segmented at the boundary using characters (such as "words" or "characters") as the smallest unit, thereby obtaining a large number of phoneme sequences at the "word" or "character" level and their corresponding speech data, constructing a massive "speech data – phoneme sequence" alignment dataset.

[0051] Next, the pairwise speech data can be combined to form multiple training audio pairs based on characters. Since each pair of audio segments corresponds to its own phoneme sequence, the pronunciation similarity between the two audio segments in each pair can be determined. Specifically, the pronunciation similarity between the two audio segments can be calculated based on statistical models. For example, a Hidden Markov Model (HMM) can be used to calculate the phoneme sequence similarity through state transition probabilities and observation probabilities. The process may include aligning the phoneme sequence with the acoustic model states and calculating the joint probability of the state sequence as the similarity. Another example is using a Gaussian Mixture Model (GMM), compressing multiple templates into a GMM, and evaluating the similarity after alignment through probability density. Yet another example is using methods based on pronunciation rules to determine the pronunciation similarity between the two audio segments in each pair. This completes the construction of the training dataset.

[0052] In one implementation, to more accurately represent the pronunciation similarity between two audio files, a similarity matrix can be constructed for the two audio files, such as... Figure 3 The step may include: Step 201: Process the labeled first set of speech data into multiple sets of training audio pairs in character units, and the phoneme sequences corresponding to the two audios in each pair.

[0053] Step 202: Based on multiple frames in the phoneme sequence, generate a similarity matrix for the two audio components in each pair, and determine the pronunciation similarity of each pair based on the similarity matrix.

[0054] Step 203: Construct a training dataset based on the pronunciation similarity of each pair.

[0055] Specifically, as introduced earlier, the first set of labeled speech data can be used to generate phoneme sequences for each speech data using transcription and MFA. Then, these sequences can be freely combined in pairs to form multiple sets of training audio pairs based on characters.

[0056] Subsequently, phoneme sequences of each pair of training audio can be extracted to generate a phoneme-level similarity matrix. This allows us to determine the degree of pronunciation difference between two audio files. Specifically, we can set the embedding vector lengths of the models outputting training audios A and B to be respectively... Then the similarity matrix For a size of The matrix can be represented as ,in Indicates the first The system checks if the phonemes corresponding to the frames are the same; a value of 1 indicates that the phoneme labels of the two frames are the same, while a value of 0 indicates that they are different. Finally, it obtains two training audio pairs A and B aligned to the frame level, the phoneme sequences corresponding to A and B respectively, and the similarity matrix between A and B. (That is, the phoneme similarity matrix). Then, the pronunciation similarity can be calculated using methods such as average similarity, DTW path summation, and HMM.

[0057] Ultimately, a training dataset can be constructed based on the pronunciation similarity of each pair. In practical applications, the similarity matrix can also be used directly as the pronunciation similarity for each pair.

[0058] Step 210: Using the training dataset, train the pronunciation correlation model to output the function of similarity between audio files.

[0059] In this step, multiple training audio pairs from the training dataset can be used as input, with the phoneme sequences corresponding to the two audio items in each pair, and the pronunciation similarity between the two audio items in each pair serving as supervision, to train the model. In practical applications, various model architectures and technical solutions can be implemented: For example, contrastive learning frameworks (such as CLAP, Contrastive Language-Audio Pretraining) employ a dual-tower structure, encoding the phoneme sequences of two audio files separately and establishing pronunciation similarity associations through a contrastive learning mechanism. Specifically, the CLAP model, through joint alignment of the audio encoder and text encoder, can map phoneme sequences to a unified semantic space and calculate cosine similarity as a pronunciation similarity metric.

[0060] Another example is the phoneme-level triplet loss model, which, referencing audio similarity measurement methods, designs a triplet loss function: anchor audio, positive sample audio (high similarity), and negative sample audio (low similarity). Phoneme sequences are encoded using CNNs or Transformers, and the phoneme sequences are mapped to a multi-dimensional embedding space through projection layers, using Euclidean distance as the similarity measure.

[0061] Another example is the end-to-end speech feature alignment model, which is based on the Resemblyzer (Resemblance Analyzer) framework. It uses a Wav2Vec 2.0 (Waveform to Vector) pre-trained model to extract phoneme-level acoustic features and combines this with the Dynamic Time Warping (DTW) algorithm to align phoneme sequences. Similarity is calculated by comparing the differences in the aligned feature distributions (such as KL divergence, Kullback-Leibler divergence, and Kullback-Leibler divergence).

[0062] In this embodiment, XLS-R (Xtra Large Scale Speech Representation) can be used as the backbone model of the speech encoder, and a contrastive learning loss function can be added on it to achieve the ability to represent and infer the pronunciation similarity between audio data.

[0063] The encoder module uses the MFCC (Multi-Frequency Coefficient) of the speech data as input features. MFCC is an acoustic parameter widely used in speech analysis and recognition. It incorporates the Mel-scale to better reflect the distribution of human ear sensitivity to pitch across different frequency bands. This provides a stable and discriminative input representation for the subsequent speech recognition encoder, improving the model's recognition performance and semantic modeling capabilities.

[0064] In terms of model structure, XLS-R is a multilingual self-supervised speech pre-training model launched by Meta AI. Its structure is a stacked module based on Transformers, using the wav2vec 2.0 architecture to scale to 128 languages ​​for large-scale training. Its key structures include a front-end convolutional feature extractor and a Transformer encoder stack. The front-end convolutional feature extractor uses multiple 1D convolutional layers to extract local acoustic features from the original waveform; the Transformer encoder stack uses 12–24 Transformer modules, each layer containing a multi-head self-attention mechanism and a feedforward network; the model output has a hidden dimension of 768, enabling frame-by-frame context-sensitive representations.

[0065] Regarding the overall training objective of the model, the three types of losses—CTC (Connectionist Temporal Classification) loss, cross-entropy loss, and the contrastive cross-entropy loss based on the phoneme similarity matrix mentioned earlier—are jointly trained in a weighted combination, i.e., the total loss:

[0066] in These are the weights for the three types of loss functions, which can be adjusted according to actual needs. For CTC losses, For cross-entropy loss, This is a contrastive cross-entropy loss based on the similarity matrix. The weighted joint use of three loss functions ensures that the model possesses phoneme recognition capabilities while also internalizing the structural representation of pronunciation differences through contrastive learning, thereby obtaining an embedding space representation with cross-linguistic generalization capabilities. Let... Indicates the input speech recognition encoder MFCC acoustic features, the entire speech contains Frame, then , This represents the target phoneme sequence for model training. This indicates that the input audio data A is in the encoder's... Embedding vectors and probability distributions of frame output Indicates the first On-frame model predicts phonemes The probability of [the loss function]. The specific loss functions are as follows: (1) CTC Loss: Used to train the encoder to perform time-series prediction on the input audio data, maintaining sensitivity to temporal order and providing coarse-grained alignment constraints. This loss enables the model to learn a reasonable distribution of time-series features without precisely labeling the time of each frame, as follows:

[0067] in, This represents all possible mappings to time series. The path set is the set of all possible time-series aligned paths (including repeating characters and whitespace) given a given acoustic feature input. Indicates that the model output is for the path The conditional probability represents the model's output time series given the input acoustic features. The probability of CTC loss allows the model to automatically skip or repeat characters during decoding, while using whitespace to separate repeated characters, thus solving the problem of "unknown alignment path".

[0068] (2) Cross-entropy loss (CE Loss): This function supervises the difference between the phoneme sequence output by the model and the phoneme sequence in the training data, enhances recognition accuracy, and provides coarse-grained alignment constraints. This loss function is the standard supervision objective for phoneme classification tasks in speech recognition models.

[0069]

[0070] For each frame (common (frame), the model outputs a probability distribution over a set of phonemes. , indicating at time The model predicts phonemes The probability, the phoneme set contains all phonemes in the training set covering diverse pronunciations, scenarios, and noise; and This can be a phoneme distribution, typically one-hot encoded. Cross-entropy loss measures the model's predicted distribution. Phoneme distribution of training data The difference between them allows us to require that the prediction probability at the correct phoneme be as high as possible. Its function is to improve the phoneme classification accuracy of the model and strengthen the explicit recognition capability for each frame of input.

[0071] (3) Contrastive Cross-Entropy Loss Based on Similarity Matrix: This embodiment proposes a contrastive cross-entropy loss based on the similarity matrix. The similarity matrix of the training data constrains the speech recognition encoder's modeling of pronunciation representation relevance, enabling the trained model to recognize pronunciation relevance across languages. This loss function is based on the embedding sequences extracted from the encoder of the two input audio segments. It calculates the cosine similarity matrix between them and compares it with the similarity matrix constructed in the training dataset, allowing the model to learn how to encode pronunciation similarity in the embedding space. Specifically, it can be defined as follows:

[0072] in These are the phoneme similarity matrix elements at the phoneme level. , The first and second voice data are respectively the first voice data of A and B. The embedding vector of the frame. Indicates the phoneme position index of the candidate alignment. The temperature coefficient, which can range from 0.05 to 0.2, is fine-tuned on the validation set to balance distribution smoothness and discriminativity. Using this loss, the model learns to represent pronunciation differences in a structured way within the embedding space.

[0073] like Figure 4The diagram illustrates the training of a pronunciation relevance model, with speech data A and B as input. The upper encoding layer extracts features from the input speech (e.g., acoustic and semantic features), using techniques such as CNNs (convolutional neural networks for local feature extraction), RNNs / LSTMs (for capturing temporal dependencies), and Transformers (for global context modeling), outputting a high-dimensional speech feature representation. The upper feature layer can visualize the discrete or continuous features output by the encoder, providing a foundation for subsequent classification and loss calculation. The linear classification layer maps the high-dimensional features to a classification space (e.g., phonemes, number of character categories), and the Softmax layer transforms the output into a probability distribution (used for confidence scoring in classification tasks).

[0074] Joint optimization using multiple loss functions can, as described above, constrain model training from different angles through three loss functions: cross-entropy loss can optimize recognition accuracy; CTC loss can optimize the ability to predict time series data; and comparative cross-entropy loss based on the similarity matrix can enhance the similarity of similar features and the difference of dissimilar features, thereby optimizing feature discriminability.

[0075] The trained pronunciation correlation model has the ability to determine the similarity between different audio data, and can also combine CTC loss and cross-entropy loss to determine the similarity between different audio data more accurately.

[0076] Step 220: Based on the labeled second set of speech data, construct a prediction dataset, which includes multiple sets of predicted audio pairs in character units.

[0077] After training the pronunciation relevance model in the previous step, this step allows us to validate or predict the model using speech data. First, we need to construct a prediction dataset. Specifically, we can collect a second set of labeled speech data. This second set can be a portion of the first set, the entire set, or even different speech data. Next, we can use the MFA mentioned earlier to segment the entire speech data at the character level, obtaining the audio segment corresponding to each character as the predicted audio for that character. Finally, we can combine the predicted audio for each pair of characters to obtain multiple sets of predicted audio pairs based on individual characters.

[0078] In one implementation, to more accurately determine the similarity of predicted audio pairs, more fine-grained predicted audio pairs can be constructed. For example... Figure 5 Includes the following steps: Step 221: The labeled second set of speech data is processed into predicted audio in character units.

[0079] Specifically, first, for the second set of labeled speech data, use the MFA mentioned above to perform boundary segmentation on the entire speech data with characters as the smallest unit, and obtain the audio segments corresponding to each character as the predicted audio of the character.

[0080] Step 222: Construct a predicted audio set corresponding to a character from multiple predicted audios corresponding to the same character.

[0081] In the second set of speech data, there can be multiple predicted audios corresponding to the same character. For example, the character "loan" can correspond to 10 predicted audios. However, in actual scenarios, due to different scenarios, different context sentences, or different pronunciations of men, women, the elderly, and children, there may be slight differences in the pronunciation or tone of the same character.

[0082] Therefore, in order to more comprehensively and accurately determine the similarity between characters, in this step, a predicted audio set corresponding to a character can be constructed from multiple predicted audios corresponding to the same character. For example, if "loan" corresponds to 10 predicted audios, then all 10 predicted audios are selected. Similarly, for other characters, 10 or all less than 10 can be selected to complete the construction of the predicted audio set corresponding to the character.

[0083] Step 223: Combine the predicted audio sets corresponding to each pair of characters respectively to construct multiple groups of predicted audio pairs with characters as the unit.

[0084] In the previous step, a predicted audio set corresponding to a character and containing multiple predicted audios was obtained. In this step, multiple groups of predicted audio pairs with characters as the unit can be constructed by combining them in pairs. For example, both "loan" and "bring" correspond to 10 predicted audios, then they can be combined in pairs. Thus, in the predicted audio pairs of these two characters, there can be 100 sub-predicted audio pairs, and these 100 sub-predicted audio pairs are used as a set as the predicted audio pair of these two characters.

[0085] Step 230: Use the trained pronunciation correlation model to generate the pronunciation similarity between each character in the predicted data set.

[0086] In step 210, the model already has the ability to determine the similarity between audio data. Therefore, in this step, multiple groups of predicted audio pairs with characters as the unit can be used as the input to generate the pronunciation similarity between each character.

[0087] When introducing the construction of the prediction data set in step 220, among the predicted audio pairs constructed in steps 222 and 223, there are multiple sub-predicted audio pairs. Then in this step, the pronunciation similarity can be generated for each sub-predicted audio pair respectively, and the pronunciation similarity between characters can be determined in the way of average value. As an example mentioned above, there are 100 sub-predicted audio pairs for "loan" and "bring". These 100 sub-predicted audio pairs form a set as the predicted audio pair for these two characters. Then in this step, the 100 similarities of these 100 sub-predicted audio pairs can be determined respectively, and the similarity between the characters "loan" and "bring" can be determined in the way of average value, which also makes the determined similarity more accurate.

[0088] Step 240, based on the generated pronunciation similarity, construct character association data based on pronunciation correlation.

[0089] As introduced in step 110, character association data can be constructed according to the pronunciation similarity values between characters. This step is the pre-construction process.

[0090] As introduced above, the preset bias strategy and the target voice can belong to the same voice field. Among different majors and different dialects (which can be collectively referred to as different fields), there are their own different pronunciation correlations. Therefore, different character association data can be constructed for different fields. So, in order to make the speech recognition more accurate, in one implementation, the target voice and the second set can belong to the voices within a specific field.

[0091] For example, when training the pronunciation correlation model, the first set of voice data can be Chinese Mandarin with a large data volume. The trained pronunciation correlation model already has the ability to recognize pronunciation correlation. After that, a small dialect or a small business voice can be used as the second set of voice data. The character association data constructed thereby is more suitable for the target voice within the same specific field. Then the speech result recognized thereby is more accurate. That is, when the target voice and the second set belong to the same specific field, the character association data can include being adapted to the target voice.

[0092] Therefore, when constructing character association data based on pronunciation correlation, it can be based on the recognition requirement of a certain specific field of voice. By collecting the voices of this field as prediction data, when the target voice needs to be recognized, the corresponding adapted character association data can be retrieved to more accurately and comprehensively complete the recognition task.

[0093] For this purpose, in one implementation, the second set can also be smaller than the first set of speech data. As exemplified above, the first set could be Mandarin Chinese with a larger data volume, while the second set could be a dialect with a smaller data volume. Furthermore, in practical applications, it's difficult to collect a large amount of data for niche dialects. Similarly, the first set could be American English with a larger data volume, while the second set could be English with dialectal characteristics with a smaller data volume.

[0094] As can be seen from the above embodiments, the construction of character association data based on pronunciation relevance can be completed in parallel, independent of the speech recognition process. Therefore, the character association data can have high real-time performance. For example, for a certain dialect, as long as there is new annotated speech data, the pronunciation relevance model can be used to generate the pronunciation similarity between characters, which can gradually improve the speech recognition accuracy of the dialect.

[0095] As described above, the method of this application also has strong independence and can be "hot-swapped" with existing speech recognition systems. This feature allows for the addition of a pronunciation-related bias method between existing inference and hot word bias without modifying other structures of the system.

[0096] Figure 6 A block diagram of a phonetic correlation-based biasing device provided in an exemplary embodiment of this application is shown, the device being used to perform, as Figure 1 All or part of the steps in the illustrated embodiments, such as Figure 6 As shown, the device may include the following modules.

[0097] The character receiving unit 300 can be used to receive pre-selected characters inferred from the target speech; The hot word determination unit 310 can be used to determine candidate hot words that have a phonetic relevance to the pre-selected character from pre-constructed character association data based on phonetic relevance, using a threshold higher than a specific phonetic similarity threshold as a filtering condition; The character bias unit 320 can be used to bias the pre-selected characters according to the candidate hot words.

[0098] In one implementation, the hot word determination unit 310 can also be used to construct character association data based on pronunciation relevance, specifically for: Based on the first set of labeled speech data, a training dataset is constructed. The training dataset includes multiple sets of training audio pairs in character units, the phoneme sequences corresponding to the two audios in each pair, and the pronunciation similarity between the two audios in each pair. The function of training a pronunciation correlation model using a training dataset and outputting the similarity between audio samples; Based on the labeled second set of speech data, a prediction dataset is constructed, which includes multiple sets of predicted audio pairs in character units; Using a pronunciation correlation model, the pronunciation similarity between characters in the prediction dataset is generated; Based on pronunciation similarity, character association data based on pronunciation relevance is constructed.

[0099] In one implementation, the character bias unit 320 can specifically be used for: The labeled second set of speech data is processed into predicted audio in character units; Construct a set of predicted audio for each character using multiple predicted audios corresponding to the same character; The predicted audio sets corresponding to each pair of characters are combined to construct multiple sets of predicted audio pairs based on characters.

[0100] In one implementation, the character bias unit 320 can specifically be used for: The labeled first set of speech data is processed into multiple sets of training audio pairs in character units, and the phoneme sequences corresponding to the two audios in each pair. Based on multiple frames in the phoneme sequence, a similarity matrix is ​​generated for the two audio components in each pair, and the pronunciation similarity of each pair is determined based on the similarity matrix. A training dataset is constructed based on the pronunciation similarity of each pair.

[0101] In one implementation, the target speech and the second set of speech data belong to a specific domain of speech.

[0102] In one implementation, the second set contains less voice data than the first set.

[0103] Figure 7 A structural block diagram of a computer device illustrated in an exemplary embodiment of this application is shown. This computer device can be implemented as a server as described above in this application. The computer device includes a Central Processing Unit (CPU) 400, a system memory 410 including Random Access Memory (RAM) 411 and Read-Only Memory (ROM) 4412, and a system bus connecting the system memory 410 and the CPU 400. The computer device also includes a mass storage device 420 for storing an operating system 421, application programs 422, and other program modules 423.

[0104] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 410 and mass storage device 420 described above can be collectively referred to as memory.

[0105] According to various embodiments of this disclosure, the computer device can also be connected to a remote computer on a network, such as the Internet. That is, the computer device can be connected to a network 440 via a network interface unit 430 connected to the system bus, or it can use the network interface unit 440 to connect to other types of networks or remote computer systems (not shown).

[0106] The memory also includes at least one instruction, at least one program, code set, or instruction set, which are stored in the memory. The central processing unit 400 executes the at least one instruction, at least one program, code set, or instruction set to implement all or part of the steps in the server security configuration method shown in the above embodiments.

[0107] In one embodiment, a computer-readable storage medium is also provided, which stores at least one computer program that is loaded and executed by a processor to implement the above-described pronunciation-related biasing method. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, or optical data storage device, etc.

[0108] In one embodiment, a computer program product is also provided, comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to perform the above-described actions. Figure 1 The example illustrates a bias method based on pronunciation correlation.

[0109] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0110] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A biasing method based on pronunciation correlation, characterized in that, The method includes: Receive pre-selected characters inferred from the target speech; From pre-constructed character association data based on pronunciation relevance, candidate hot words that have pronunciation relevance to the pre-selected characters are determined using a filtering condition that is higher than a certain pronunciation similarity threshold. The pre-selected characters are biased based on the candidate hot words.

2. The method according to claim 1, characterized in that, The method for constructing character association data based on pronunciation relevance includes: Based on the labeled first set of speech data, a training dataset is constructed. The training dataset includes multiple sets of training audio pairs in character units, the phoneme sequences corresponding to the two audios in each pair, and the pronunciation similarity between the two audios in each pair. Using the aforementioned training dataset, a pronunciation correlation model is trained to output the similarity between audio samples. Based on the labeled second set of speech data, a prediction dataset is constructed, which includes multiple sets of predicted audio pairs in character units; Using the pronunciation correlation model, the pronunciation similarity between each character in the prediction dataset is generated; Based on the pronunciation similarity, character association data based on pronunciation relevance is constructed.

3. The method according to claim 2, characterized in that, The target speech and the second set of speech data belong to a specific domain of speech.

4. The method according to claim 3, characterized in that, The second set has less voice data than the first set.

5. The method according to claim 2, characterized in that, The construction of a prediction dataset based on the labeled second set of speech data includes: The labeled second set of speech data is processed into predicted audio in character units; Construct a set of predicted audio for each character using multiple predicted audios corresponding to the same character; The predicted audio sets corresponding to each pair of characters are combined to construct multiple sets of predicted audio pairs based on characters.

6. The method according to claim 2, characterized in that, The construction of a training data set based on the labeled first set of speech data includes: The labeled first set of speech data is processed into multiple sets of training audio pairs in character units, and the phoneme sequences corresponding to the two audios in each pair. Based on multiple frames in the phoneme sequence, a similarity matrix is ​​generated for the two audio components in each pair, and the pronunciation similarity of each pair is determined based on the similarity matrix. A training dataset is constructed based on the pronunciation similarity of each pair.

7. A biasing device based on phonetic correlation, characterized in that, The device includes: a character receiving unit, a hot word determination unit, and a character biasing unit, wherein, The character receiving unit is used to receive pre-selected characters inferred from the target speech; The hot word determination unit is used to determine candidate hot words that have a phonetic relevance to the pre-selected characters from pre-constructed character association data based on pronunciation relevance, using a threshold higher than a specific pronunciation similarity threshold as a filtering condition. The character biasing unit is used to bias the pre-selected characters according to the candidate hot words.

8. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer program, which is loaded and executed by the processor to implement the method as claimed in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions that, when executed by a computer device, cause the computer device to perform the method as described in any one of claims 1 to 6.

Images