Speech processing method and apparatus, electronic device, and storage medium

Abstract

The application provides a speech processing method and device, electronic equipment and storage medium. The method comprises the following steps: first, determining a prompt word of speech data, thereby generating a first generation probability of each candidate word in a candidate word library according to the prompt word, and selecting a candidate word from the candidate word library according to the first generation probability of each candidate word in the candidate word library to obtain a first text; then, determining a first similarity between the first text and the prompt word, and adjusting an influence parameter of the first generation probability of each candidate word according to the influence of the first similarity, thereby determining a second generation probability of each candidate word; then, selecting a candidate word from the candidate word library according to the second generation probability of each candidate word to obtain a second text; finally, generating a text corresponding to the speech data according to the first text and the second text. In this way, the accuracy of speech input recognition can be improved.

Description

Technical Field

[0001] This application relates to the field of general data processing technology, and in particular to a voice processing method, apparatus, electronic device and storage medium. Background Technology

[0002] In modern information processing systems, speech input recognition plays an increasingly important role, such as during conversations. However, in practical applications, speech input is often accompanied by various background noises (such as traffic noise, music, and crowds), which interfere with the speech signal, making it difficult for speech recognition systems to accurately capture the user's actual speech content. This can lead to missing user speech content and consequently, speech input recognition errors or inaccuracies. Therefore, improving the accuracy of speech input recognition is a pressing technical problem that needs to be solved in this field. Summary of the Invention

[0003] This application provides a speech processing method, apparatus, electronic device, and storage medium, which generates text based on prompt words in speech data and adjusts the generation probability of candidate words based on the similarity between the generated text and the prompt words, thereby improving the accuracy of speech input recognition.

[0004] In a first aspect, this application provides a speech processing method, the method comprising:

[0005] Determine the prompt words for the voice data;

[0006] Generate the first generation probability of each candidate word in the candidate word library based on the prompt words;

[0007] Based on the first generation probability of each candidate word in the candidate vocabulary, candidate words are selected from the candidate vocabulary to obtain the first text;

[0008] Determine the first similarity between the first text and the prompt word;

[0009] The second generation probability of each candidate word is determined by adjusting the influence parameter of the first generation probability on the first generation probability of each candidate word based on the first similarity.

[0010] Based on the second generation probability of each candidate word, candidate words are selected from the candidate word library to obtain the second text;

[0011] Generate the text corresponding to the speech data based on the first and second texts.

[0012] As can be seen, in this application, if there is a lack of voice input during the voice input recognition process, text can be generated based on the prompt words of the voice data to make up for the missing content in the voice input. Furthermore, during the text generation process, the generation probability of candidate words is adjusted based on the previously generated content, thereby improving the matching degree between the generated text and the user's intent.

[0013] Secondly, this application provides a voice processing apparatus, the apparatus comprising:

[0014] The first processing unit is used to determine the prompt words for the voice data;

[0015] The second processing unit is used to generate the first generation probability of each candidate word in the candidate word library based on the prompt words;

[0016] The third processing unit is used to select candidate words from the candidate word library based on the first generation probability of each candidate word in the candidate word library to obtain the first text;

[0017] The fourth processing unit is used to determine the first similarity between the first text and the prompt word;

[0018] The second processing unit is also used to adjust the influence parameter of the first generation probability of each candidate word based on the first similarity, and determine the second generation probability of each candidate word;

[0019] The third processing unit is also used to select candidate words from the candidate word library according to the second generation probability of each candidate word to obtain the second text, and to generate the text corresponding to the speech data according to the first text and the second text.

[0020] Thirdly, this application provides an electronic device including a processor, a memory, and a communication interface. The processor, memory, and communication interface are interconnected and perform communication with each other. The memory stores executable program code, the communication interface is used for wireless communication, and the processor is used to retrieve the executable program code stored in the memory and execute some or all of the steps described in any of the methods in the first aspect.

[0021] Fourthly, this application provides a computer-readable storage medium storing electronic data, which, when executed by a processor, is used to perform the electronic data to implement some or all of the steps described in the first aspect of this application.

[0022] Fifthly, this application provides a computer program product including a computer program operable to cause a computer to perform some or all of the steps described in the first aspect of this application. The computer program product may be a software installation package. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the structure of a voice processing system provided in an embodiment of this application;

[0025] Figure 2 A schematic flowchart of a speech processing method provided in an embodiment of this application;

[0026] Figure 3 A flowchart illustrating another speech processing method provided in an embodiment of this application;

[0027] Figure 4 A schematic flowchart illustrating a model training method provided in an embodiment of this application;

[0028] Figure 5 A flowchart illustrating a random masking operation provided in an embodiment of this application;

[0029] Figure 6 A functional unit block diagram of a voice processing device provided in an embodiment of this application;

[0030] Figure 7 A functional unit block diagram of another voice processing device provided in the embodiments of this application;

[0031] Figure 8 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0033] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps is not limited to the steps listed, but may optionally include steps not listed, or may optionally include other steps inherent to these processes, methods, products, or apparatuses.

[0034] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0035] Currently, voice input is often accompanied by various background noises, which interfere with the voice signal, making it difficult for voice recognition systems to accurately capture the user's actual speech content, thus leading to voice input recognition errors or inaccuracies. Existing methods for improving the accuracy of voice input recognition mainly use noise suppression techniques to reduce the impact of background noise on the voice signal and enhance the voice content in the voice data, thereby improving the accuracy of voice input recognition. However, simply using noise suppression or voice enhancement techniques still cannot achieve effective voice input recognition in situations with high noise levels or missing voice input.

[0036] Based on this, this application provides a speech processing method. After acquiring speech data, it determines the prompt words in the speech data, generates a first generation probability for each candidate word in a candidate word library based on the prompt words, and selects candidate words from the candidate word library based on the first generation probability of each candidate word to obtain a first text. Then, it determines a first similarity between the first text and the prompt words, and adjusts the influence parameter of the first generation probability of each candidate word based on the first similarity to determine a second generation probability for each candidate word. Subsequently, it selects candidate words from the candidate word library based on the second generation probability of each candidate word to obtain a second text. Finally, it generates the text corresponding to the speech data based on the first and second texts. In this way, for cases where speech input is missing, text generation can be performed based on the prompt words in the speech data, thereby compensating for the missing content in the speech input. Furthermore, during the text generation process, the generation probability of candidate words is adjusted based on the previously generated content, thereby improving the matching degree between the generated text and the user's intent.

[0037] Please see Figure 1 , Figure 1This is a schematic diagram of the structure of a voice processing system provided in an embodiment of this application, as shown below. Figure 1 As shown, the voice processing system 100 includes a voice acquisition device 101 and a voice processing device 102.

[0038] The voice acquisition device 101 is responsible for capturing the user's voice input and converting it into a digital audio signal for subsequent processing. Optionally, the voice acquisition device 101 can be a microphone or a device with microphone functionality. For example, the voice acquisition device 101 may also include a preamplifier, an analog-to-digital converter, a noise reduction module, etc.; the preamplifier is used to amplify the weak analog signal output from the microphone to a level suitable for subsequent processing; the analog-to-digital converter is used to convert the analog audio signal into a digital format for easy computer processing; the noise reduction module uses hardware-level noise suppression algorithms, such as spectral subtraction and Wiener filtering, to reduce background noise interference.

[0039] The speech processing device 102 is responsible for converting the digital audio signal acquired by the speech acquisition device 101 into text. Optionally, the speech processing device 102 may include two sub-devices, where a first sub-device is responsible for determining the prompt words for the speech data, and a second sub-device is responsible for generating text based on the prompt words. Optionally, the speech processing device 102 may be a processor or other device with processing capabilities.

[0040] Optionally, the voice acquisition device 101 and the voice processing device 102 can be integrated into one unit, such as a smartphone, tablet computer, or other device that has both voice acquisition and voice processing functions.

[0041] In this application, the voice acquisition device 101 acquires voice data, and the voice processing device 102 determines the prompt word in the voice data, thereby generating a first generation probability of each candidate word in the candidate word library based on the prompt word, and selecting candidate words from the candidate word library based on the first generation probability of each candidate word to obtain a first text; then, a first similarity between the first text and the prompt word is determined, and the influence parameter of the first generation probability of each candidate word is adjusted according to the first similarity to determine a second generation probability of each candidate word; then, candidate words are selected from the candidate word library based on the second generation probability of each candidate word to obtain a second text; finally, the text corresponding to the voice data is generated based on the first text and the second text.

[0042] It can be seen that in the process of voice input recognition, if there is a lack of voice input, text can be generated based on the prompt words of the voice data to make up for the missing content in the voice input. Furthermore, in the process of text generation, the generation probability of candidate words is adjusted based on the content generated earlier, thereby improving the matching degree between the generated text and the user's intent.

[0043] Based on this, the present application provides a voice processing method, and the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0044] Example 1, please refer to Figure 2 , Figure 2 This is a flowchart illustrating a speech processing method provided in an embodiment of this application. The method is applied to the aforementioned speech processing device, such as... Figure 2 As shown, the method includes the following steps:

[0045] Step S201: Determine the prompt words for the voice data.

[0046] The voice processing device acquires voice data through the voice acquisition device. Optionally, when the voice acquisition device acquires the voice data, it can perform preliminary processing on the collected audio signal, such as noise reduction and volume normalization. The prompt words are text data, and the voice data processing should include speech-to-text processing, based on which the prompt words are determined.

[0047] Specifically, in one feasible embodiment, determining the prompt words for the voice data includes: performing speech-to-text processing on the voice data to obtain a third text, where each text segment in the third text includes a corresponding confidence level; determining a first confidence level in the third text that is lower than a preset confidence level, replacing the first text segment corresponding to the first confidence level with a second text segment to obtain a fourth text, where the similarity between the second text segment and the third text is higher than the similarity between the first text segment and the third text; extracting keywords from the fourth text to determine the keywords corresponding to the fourth text; and determining the prompt words based on the keywords corresponding to the fourth text.

[0048] Speech-to-text processing of speech data can be based on deep learning models, such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). Specifically, it involves extracting features such as Mel-frequency cepstral coefficients (MFCCs) and Linear Predictive Coding (LPC) from the speech signal, and then using a trained ASR model to convert these features into text output. Each generated text segment is assigned a confidence score. The confidence score measures the degree of confidence in the correctness of a particular text segment, typically a value between 0 and 1. A text segment, as the name suggests, refers to a continuous sequence of characters or words extracted from a text. A text segment can be a single word, phrase, sentence, paragraph, etc.

[0049] For the first text segment with low confidence, a language model can be used to generate multiple candidate text segments. This can be further enhanced by incorporating a thesaurus relevant to the business domain. Then, using edit distance algorithms or semantic similarity calculations, the candidate text segment that best matches the context (the second text segment) is selected to replace the first text segment, thus forming a new fourth text. This improves the overall text quality, reduces errors caused by low confidence, and ensures that the generated text more closely reflects the user's true intent.

[0050] Keyword extraction from the fourth text primarily involves identifying representative and important words or phrases, such as through Term Frequency-Inverse Document Frequency (TF-IDF) and TextRank algorithms. Cue words can be key phrases or sentences that summarize the user's intent. Determining cue words based on at least one target keyword can combine contextual information and domain knowledge bases to transform the extracted keywords into more expressive cue words.

[0051] For example, if the user inputs the voice message "What is the recent price trend of gold?", the first input text obtained after speech-to-text conversion is "What is the recent price of yellow turbans?". If the confidence scores for "recently", "yellow turbans", "price", and "how" are 0.95, 0.45, 0.85, and 0.75 respectively, and the preset confidence score is 0.7, the candidate text segments generated for "yellow turbans" include "yellow turbans", "gold", and "yellow turbans". At this point, the candidate word "gold" that best matches the context is selected, resulting in the second input text "What is the recent price of gold?". Subsequently, the keywords extracted include "gold" and "price". The prompt words determined based on "gold" and "price" can include "gold price analysis".

[0052] In this application, by replacing low-confidence words and generating prompt words after keyword extraction, errors in subsequent text generation caused by low confidence can be reduced, ensuring that the generated text is closer to the user's true intent.

[0053] Step S202: Generate the first generation probability of each candidate word in the candidate word library based on the prompt words;

[0054] In the text generation process, the generation probability of each candidate word reflects the language model's confidence in predicting the next possible word within the current context. Specifically, given the generated preceding text (such as a sentence prefix), the model calculates a probability distribution P(next word | preceding text). The probability of each candidate word represents its likelihood of becoming the next word.

[0055] Step S203: Based on the first generation probability of each candidate word in the candidate word library, select candidate words from the candidate word library to obtain the first text.

[0056] The first text can be a complete text or text of a preset length. The first text is obtained by selecting candidate words from the candidate word library based on the probability of each candidate word's initial generation. This selection can be based on a specific decoding strategy, such as greedy search, random sampling, or Top-K sampling.

[0057] Step S204: Determine the first similarity between the first text and the prompt word.

[0058] The aforementioned similarity can be cosine similarity, Jaccard similarity, etc.

[0059] Step S205: Adjust the influence parameter of the first generation probability of each candidate word based on the first similarity to determine the second generation probability of each candidate word.

[0060] The generation probability of candidate words may be influenced by parameters such as decoding strategies and hyperparameters (e.g., temperature parameters, repetition penalties), as well as external knowledge and constraints (e.g., knowledge base enhancement, manual rule intervention). It is understandable that the first and second generation probabilities are not specific values, but rather refer to the generation probabilities under different influencing parameters.

[0061] Specifically, in a feasible embodiment, the influencing parameters of the first generation probability of each candidate word include a temperature parameter and a cue word embedding weight; the first generation probability of each candidate word is adjusted according to the first similarity to determine the second generation probability of each candidate word, including:

[0062] If the first similarity is less than the first threshold, the temperature parameter in the first generation probability of each candidate word is reduced, and / or the cue word embedding weight in the first generation probability of each candidate word is increased to obtain the second generation probability of each candidate word;

[0063] If the first similarity is greater than the second threshold, increase the temperature parameter in the first generation probability of each candidate word and / or decrease the prompt word embedding weight in the first generation probability of each candidate word to obtain the second generation probability of each candidate word, where the second threshold is not less than the first threshold.

[0064] In the text generation process, when generating the next word, not only the current context is considered, but the generation probability of candidate words is also adjusted based on their similarity to the prompt. If the generated content deviates from the topic corresponding to the prompt, the generation probability of words related to the prompt needs to be increased, making the generated content more likely to return to the topic of the prompt. The formula is: P′(next_token|context)=(1-α)·P(next_token|context)+α·Psimilarity, where P′(next_token|context) represents the adjusted generation probability distribution, reflecting the generation probability of the corresponding candidate word (next_token) when predicting the next text generation based on certain optimization or adjustment rules, given the context. P(next_token|context) refers to the original generation probability distribution, and the probability distribution for predicting the next word based on the context. Psimilarity is a probability distribution of similarity preferences based on prompt words, generated by calculating the semantic similarity between candidate words and prompt words. α is a weighting parameter that controls the balance between the original probability distribution and similarity preferences.

[0065] Optionally, the generation probability of candidate words can be adjusted based on a temperature parameter. The temperature parameter controls the degree of randomness in the generation process. Higher temperature values ​​amplify the weight of low-probability words, resulting in more random generation. Lower temperature values ​​amplify the weight of high-probability words, resulting in more predictable generation. The formula is: P T (ω∣context)=exp(logits ω / T) / ∑ i exp(logits i / T), where P T (ω|context) is the generation probability of candidate word ω after adjustment by the temperature parameter T; logits ω represents the original unnormalized score of candidate word ω, reflecting the model's initial preference for candidate word ω; T is the temperature parameter; logits i This represents the original normalized score of any candidate word i among all candidate words. Increasing the temperature T amplifies the weight of words with low original generation probability, thereby increasing the generation probability of words unrelated to the prompt word; conversely, decreasing the temperature T amplifies the weight of words with high original generation probability, thereby increasing the generation probability of words related to the prompt word.

[0066] Optionally, the semantic vector of the prompt word can be used as a dynamic bias term in the generation, directly affecting the probability of word generation in the next step. This method can strengthen the generation probability of the prompt word and its related words. The formula can include: P′(ω∣context)=P(ω∣context)+δ·similarity(Eprompt,Enext_token), where Eprompt is the semantic embedding vector of the prompt word, Enext_token is the semantic embedding vector of the candidate word, δ is a weight parameter used to control the bias strength, the similarity function calculates the similarity between two vectors (e.g., cosine similarity), and the aforementioned prompt word embedding weight can refer to δ. P(ω∣context) refers to the original generation probability of the candidate word ω, and P′(ω∣context) is the adjusted generation probability of the candidate word ω. Increasing the weight δ can increase the generation probability of candidate words related to the prompt word, while decreasing the weight δ can decrease the generation probability of candidate words related to the prompt word.

[0067] Therefore, when it is determined that the first similarity between the first text and the prompt word is low, the temperature parameter in the first generation probability of each candidate word can be reduced, and / or the prompt word embedding weight in the first generation probability of each candidate word can be increased, thereby obtaining the second generation probability of each candidate word; when it is determined that the first similarity between the first text and the prompt word is high, the temperature parameter in the first generation probability of each candidate word can be increased, and / or the prompt word embedding weight in the first generation probability of each candidate word can be reduced, thereby obtaining the second generation probability of each candidate word.

[0068] Furthermore, the system can be evaluated and corrected in stages. If the similarity between the text generated in a certain stage and the prompt is too low, the current generation can be terminated, the context reset, and the text generation restarted. Specifically, a portion of the generated text can be compared with the prompt, retaining the reasonable parts and replacing the deviating parts.

[0069] In this application, by adjusting the generation probability of candidate words, the generated text can be made to better match the semantics of the prompt words.

[0070] Step S206: Based on the second generation probability of each candidate word, select candidate words from the candidate word library to obtain the second text.

[0071] In this embodiment, the decoding strategy used in the second text generation process should be the same as that used in the first text generation process.

[0072] Step S207: Generate the text corresponding to the speech data based on the first text and the second text.

[0073] In this embodiment, during the text generation process, the generation probability of candidate words is adjusted based on the similarity between the previously generated text and the prompt words, until the text generation is completed and the text corresponding to the speech data is obtained.

[0074] Optionally, in a feasible embodiment, after determining the first similarity between the first text and the prompt word, the method further includes: if the first similarity is less than a third threshold, obtaining multiple candidate texts corresponding to the first text; and taking the candidate text with the highest similarity to the prompt word among the multiple candidate texts as the content of the first text.

[0075] Specifically, text correction can be performed on the first text when the first similarity to the prompt word is less than the third threshold. This requires obtaining multiple candidate texts corresponding to the first text. For example, given the context, possible words at specific positions can be predicted. Then, candidate texts corresponding to the first text can be queried from a homonym / synonym database. Alternatively, candidate texts corresponding to the first text can be queried from a thesaurus relevant to the business domain. Finally, the content of the first text is modified to the candidate text with the highest similarity to the prompt word. Understandably, the similarity between the modified first text and the prompt word should be higher than the original similarity.

[0076] Furthermore, self-attention mechanisms can be used to identify and correct words that do not fit the context in real time. Specifically, this can involve calculating attention scores between newly generated tokens and other tokens within the context window. These scores reflect the strength of the correlation between tokens. If the attention score between a newly generated token and its context tokens is significantly lower than average, or if the token has very little influence on other tokens, it may indicate that the token is not contextually appropriate or is unreasonable. In this case, a series of possible alternative words can be extracted as candidates based on the tokens within the context window, the next most likely token can be predicted using a language model, or a thesaurus can be selected. Besides focusing on the attention score of individual tokens, metrics such as perplexity can be used to measure the consistency and reasonableness of the entire sentence or phrase. High perplexity may indicate the presence of illogical or unreasonable parts.

[0077] In this application, by correcting text in the generated text that has a low similarity to the prompt word, the corrected text can be made to better match the semantics of the prompt word.

[0078] Optionally, in a feasible embodiment, specifically, in a feasible embodiment, selecting candidate words from the candidate word library based on the first generation probability of each candidate word in the candidate word library to obtain a first text includes: selecting candidate words from the candidate word library based on the first generation probability of each candidate word in the candidate word library to obtain a first candidate word and a second candidate word; updating the first generation probability of each candidate word based on the first candidate word, and selecting candidate words from the candidate word library based on the updated first generation probability of each candidate word to obtain a third candidate word, and determining a fifth text based on the first candidate word and the third candidate word; updating the first generation probability of each candidate word based on the second candidate word, and selecting candidate words from the candidate word library based on the updated first generation probability of each candidate word to obtain a fourth candidate word, and determining a sixth text based on the second candidate word and the fourth candidate word; and determining the first text from the fifth text and the sixth text.

[0079] In the process of text generation, multiple candidate words can be selected to obtain multiple candidate paths, so that text generation can be performed simultaneously based on multiple candidate paths until each candidate path reaches a preset termination condition. Finally, the text generated by each candidate path is selected to determine the final text.

[0080] Candidate paths refer to different decoding paths or sequences, each representing a possible text generation method. Specifically, this can be achieved through beam search, which retains multiple sequences with the highest probabilities as candidate paths during the decoding process. Alternatively, Top-K sampling can be used, selecting the top K high-probability words from the predicted probability distribution for the next generation step, forming multiple candidate paths. Nucleus sampling (Top-p) can also be employed, selecting the smallest set of words with a cumulative probability threshold p for the next generation step, ensuring diversity while maintaining a certain level of quality.

[0081] Understandably, for each generated candidate text, it's possible to check whether a preset termination condition is met to decide whether to continue generation. Preset termination conditions might include reaching the maximum generation length, specific punctuation marks, etc. When each candidate text meets the preset termination condition, the optimal text can be determined based on the multiple candidate texts. Furthermore, during text generation, the generation probability of each candidate word is determined based on the generation context; therefore, the generation probability of each candidate word is updated after it is generated.

[0082] In this application, the quality and diversity of the final output are improved by generating multiple candidate paths.

[0083] Based on the foregoing embodiments, in a feasible embodiment, determining the first text from the fifth text and the sixth text includes: determining the first text from the fifth text and the sixth text according to preset conditions; wherein the preset conditions include at least one of the following: the text with the highest relevance to the target business scenario; the text with the highest relevance to the speech features in the speech data; and the text with the highest similarity to the prompt words.

[0084] When selecting the first text from the fifth and sixth texts, this can be done based on the target business scenario. Specifically, a pre-trained language model can be used to calculate the semantic similarity between each text in the fifth and sixth texts and the prompt words or the description of the target business scenario. Furthermore, keywords from the fifth and sixth texts can be extracted and compared with those in the target business scenario, and statistical methods can be used to measure their degree of overlap.

[0085] Besides filtering based on target business scenarios, it's also possible to filter based on speech features in the speech data. Specifically, more complex speech features can be extracted using deep learning models (such as convolutional neural networks and recurrent neural networks), including but not limited to speech rate, pitch variations, and emotional features. Then, sentiment analysis models are used to identify emotional features in the speech (such as happiness, anger, anxiety, etc.), and these sentiment labels are applied to the selection of candidate texts. Specifically, sentiment analysis can be performed on each candidate text to calculate its sentiment similarity to the sentiment labels in the speech, and then candidate texts with high sentiment similarity can be selected. Furthermore, the similarity between the fifth and sixth texts and the prompt words can also be used to determine the selection. Understandably, if there are many candidate texts, different filtering orders can be set based on the aforementioned three preset conditions, thus combining the three preset conditions for filtering.

[0086] In this application, the relevance and accuracy of the first text and prompt words are improved through screening and evaluation under multiple preset conditions.

[0087] Optionally, the alignment module is used to determine the similarity between the text generated by the text generation module and the prompt word, including: the alignment module is used to determine the similarity between the target text and the prompt word, so that when the similarity between the target text and the prompt word is lower than a third threshold, the text generation module regenerates the text related to the prompt word.

[0088] The aforementioned generation of the target text can be performed iteratively. That is, after the text generation module, alignment module, error correction module, and optimization module complete a single round of text generation, alignment, error correction, and optimization, the alignment module will also evaluate the similarity between the currently obtained target text and the prompt word. If it is determined that the similarity between the currently obtained target text and the prompt word is low, the text generation module, alignment module, error correction module, and optimization module can iteratively regenerate until the similarity between the finally generated target text and the prompt word is higher than the third threshold. At this point, the target text is output.

[0089] Please see Figure 3 , Figure 3 A flowchart illustrating another speech processing method provided in this application embodiment is shown below. Figure 3 As shown, the inference method includes the following steps: First, input a prompt word, generating multiple candidate paths related to the prompt word. Then, based on the multiple candidate paths, generate initial text corresponding to each candidate path. Simultaneously, based on the similarity between the initial text corresponding to each candidate path and the prompt word, adjust the generation probability influence parameter of the candidate word for each candidate path. Based on the similarity between the initial text corresponding to each candidate path and the prompt word, perform error correction processing on the initial text. This gradually generates the generated text corresponding to each candidate path, and determines the text corresponding to the speech data from the generated text corresponding to multiple candidate paths. Finally, output the text corresponding to the speech data.

[0090] For example, suppose a user's voice input is "Any good movie recommendations lately? I'm interested in science fiction." If, due to missing speech, the initial recognized text is "Any good movies to recommend lately? I'm interested in science fiction," then natural language processing (NLP) can be used to extract keywords including "lately," "good," "movie," "recommend," and "science fiction." Based on these keywords, a more expressive prompt could be generated as "latest popular science fiction movies." Multiple candidate paths related to the prompt are first generated, such as "lately," "recently," and "recently." Then, initial text corresponding to each candidate path is generated, and the probability of generating subsequent candidate words is adjusted. Error correction is also performed on the generated text, resulting in the following generated texts: "What are some popular science fiction movies worth watching lately?", "What are some highly anticipated science fiction movies about to be released?", and "What are some science fiction movies worth watching recently?". The final text corresponding to the voice data can be "What are some popular science fiction movies worth watching lately?".

[0091] Furthermore, the aforementioned speech processing methods can be applied to the target model. The training process of the target model will be explained in detail below:

[0092] Example 2: Please refer to Figure 4 , Figure 4 This is a flowchart illustrating a model training method provided in an embodiment of this application. The method is applied to the aforementioned speech processing device, such as... Figure 4 As shown, the method includes the following steps:

[0093] Step S401: Obtain the first training data.

[0094] In acquiring the first training data, a large-scale Chinese corpus can be used, covering a variety of text sources, such as news, social media, and question-and-answer platforms, to ensure that the text content is extensive and diverse.

[0095] Optionally, the first training data includes the second training data and the third training data. The second training data is the correct text. The third training data is obtained by randomly eliminating and / or replacing homophones and synonyms in some correct text segments based on the fourth training data. The fourth training data is obtained by extracting keywords based on the second training data.

[0096] Understandably, the initial training data should include labeled correct text and its corresponding incorrect versions for model training and evaluation. Labeling refers to providing detailed annotations for error locations in the text, including spelling errors, semantic errors, and speech recognition errors. When acquiring data for incorrect versions, a Monte Carlo tree search strategy can be introduced to automatically generate partially incorrect versions of the correct text, enhancing the diversity of training samples. Specifically, Monte Carlo tree search simulates possible error scenarios, generating incorrect versions different from the original text. These errors could be spelling errors, semantic errors, or speech recognition errors, improving the model's robustness to different types of errors. For example, using the original correct text as the root node and randomly selecting tokens at certain positions as possible error locations, Monte Carlo tree search will expand the possible error candidates through random sampling; for example, replacing a character or word in case of a spelling error, or semantic errors.

[0097] Furthermore, random masking can be applied to erroneous versions. Random masking refers to randomly masking some correct tokens in the source sentence. This allows the target model to not only learn to correct the original errors but also predict the masked tokens based on context, improving the model's contextual understanding ability. Specifically, certain correct tokens in the source sentence are randomly masked; these tokens are key words or characters. During training, the model is required to fill in these missing parts. The model not only learns how to correct errors in the source sentence but also infers the masked correct tokens through contextual clues. This also facilitates handling situations where parts of the speech input are missing.

[0098] For example, please refer to Figure 5 , Figure 5 is a schematic flowchart of a random masking operation provided in the embodiments of the present application, as shown in Figure 5 in the figure, if the initial correct text is "please help me inquire about the flight from region A to region B", error text generation is performed first, for example, by replacing the homophone, replacing "inquire" with "train", to obtain a preliminary error text "please help me check and train about the flight from region A to region B", then random masking is performed, at this time, "bother" and "flight" are randomly located, and the text obtained after random masking is "ma help me check and train about the class from region A to region B".

[0099] Optionally, after performing model training, data preprocessing may also be performed, for example, text cleaning and annotation conversion. Text cleaning refers to removing redundant spaces, meaningless characters, repeated punctuation marks, and the like. Annotation conversion refers to encoding the annotated error categories, converting spelling errors, semantic errors, speech errors, etc. into specific error type codes for subsequent feature extraction.

[0100] Step S402: Train the target model based on the first training data to obtain a trained target model.

[0101] The target model processes speech data based on the logic of the aforementioned speech processing method. The target model may design different functional modules based on different procedures in the aforementioned speech processing method, so as to train the multiple functional modules respectively during the model training process. At this time, loss functions corresponding to the multiple modules may also be designed with different weights to form the final loss value. The loss function may use cross-entropy loss, contrastive loss, and BLEU (Bilingual Evaluation Understudy) / ROUGE (Recall-Oriented Understudy for Gisting Evaluation) evaluation, wherein the cross-entropy loss may be used for error correction tasks, the contrastive loss may be used for similarity evaluation, and the BLEU / ROUGE evaluation may be used for evaluating the semantic consistency between the generated text and the input prompt. It can be understood that the target model obtained based on training can perform the speech processing process shown in the foregoing embodiments, and the target model can also perform real-time parameter optimization and adjustment according to real-time feedback data during the reasoning process.

[0102] Consistent with the embodiments shown above, please refer to Figure 6 , Figure 6 is a block diagram of functional units of a speech processing apparatus provided in the embodiments of the present application, the speech processing apparatus is the aforementioned speech processing device or a part of the speech processing device, as shown in Figure 6 in the figure, the speech processing apparatus 60 includes:

[0103] The first processing unit 601 is used to determine the prompt words for the voice data;

[0104] The second processing unit 602 is used to generate a first generation probability for each candidate word in the candidate word library based on the prompt word;

[0105] The third processing unit 603 is used to select candidate words from the candidate word library based on the first generation probability of each candidate word in the candidate word library to obtain the first text;

[0106] The fourth processing unit 604 is used to determine the first similarity between the first text and the prompt word;

[0107] The second processing unit 602 is further configured to adjust the influence parameter of the first generation probability of each candidate word based on the first similarity, and determine the second generation probability of each candidate word.

[0108] The third processing unit 603 is further configured to select candidate words from the candidate word library according to the second generation probability of each candidate word to obtain the second text, and to generate the text corresponding to the speech data according to the first text and the second text.

[0109] In a feasible embodiment, in determining the prompt words for the voice data, the first processing unit 601 is specifically configured to: perform speech-to-text processing on the voice data to obtain a third text, wherein each text segment in the third text includes a corresponding confidence level; determine a first confidence level in the third text that is lower than a preset confidence level, replace the first text segment corresponding to the first confidence level with a second text segment to obtain a fourth text, wherein the similarity between the second text segment and the third text is higher than the similarity between the first text segment and the third text; extract keywords from the fourth text to determine the keywords corresponding to the fourth text; and determine the prompt words based on the keywords corresponding to the fourth text.

[0110] In a feasible embodiment, the influence parameters of the first generation probability of each candidate word include a temperature parameter and a cue word embedding weight; in determining the second generation probability of each candidate word by adjusting the first generation probability of each candidate word according to the first similarity, the second processing unit 602 is specifically used for:

[0111] If the first similarity is less than the first threshold, the temperature parameter in the first generation probability of each candidate word is reduced, and / or the cue word embedding weight in the first generation probability of each candidate word is increased to obtain the second generation probability of each candidate word;

[0112] If the first similarity is greater than the second threshold, increase the temperature parameter in the first generation probability of each candidate word and / or decrease the prompt word embedding weight in the first generation probability of each candidate word to obtain the second generation probability of each candidate word, where the second threshold is not less than the first threshold.

[0113] In a feasible embodiment, the third processing unit 603 is further configured to: if the first similarity is less than a third threshold, obtain multiple candidate texts corresponding to the first text; and take the candidate text with the highest similarity to the prompt word among the multiple candidate texts as the content of the first text.

[0114] In a feasible embodiment, specifically, in the aspect of selecting candidate words from the candidate word library based on the first generation probability of each candidate word in the candidate word library to obtain a first text, the third processing unit 603 is specifically configured to: select candidate words from the candidate word library based on the first generation probability of each candidate word in the candidate word library to obtain a first candidate word and a second candidate word; update the first generation probability of each candidate word based on the first candidate word, and select candidate words from the candidate word library based on the updated first generation probability of each candidate word to obtain a third candidate word, and determine a fifth text based on the first candidate word and the third candidate word; update the first generation probability of each candidate word based on the second candidate word, and select candidate words from the candidate word library based on the updated first generation probability of each candidate word to obtain a fourth candidate word, and determine a sixth text based on the second candidate word and the fourth candidate word; and determine the first text from the fifth text and the sixth text.

[0115] In a feasible embodiment, in determining the first text from the fifth and sixth texts, the third processing unit 603 is specifically configured to: determine the first text from the fifth and sixth texts according to preset conditions; wherein the preset conditions include at least one of the following: the text with the highest relevance to the target business scenario; the text with the highest relevance to the speech features in the speech data; and the text with the highest similarity to the prompt words.

[0116] It is understood that since the method embodiments and the device embodiments are different presentations of the same technical concept, the content of the method embodiment section in this application should be adapted to the device embodiment section in a synchronous manner, and will not be repeated here.

[0117] When using integrated units, such as Figure 7 As shown, Figure 7 This is a block diagram of the functional units of another voice processing device 60 provided in an embodiment of this application. Figure 7In this document, the voice processing device 60 includes a processing module 712 and a communication module 711. The processing module 712 controls and manages the operations of the voice processing device 60, such as the steps of the first processing unit 601, the second processing unit 602, the third processing unit 603, and the fourth processing unit 604, and / or other processes for executing the techniques described herein. The communication module 711 supports interaction between the voice processing device 60 and other devices. Figure 7 As shown, the voice processing device 60 may further include a storage module 713, which is used to store the program code and data of the voice processing device 60.

[0118] The processing module 712 can be a processor or controller, such as a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The communication module 711 can be a transceiver, RF circuitry, or a communication interface, etc. The storage module 713 can be a memory.

[0119] All relevant content in each scenario involved in the above method embodiments can be referenced from the functional descriptions of the corresponding functional modules, and will not be repeated here. Figure 2 The speech processing method shown.

[0120] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions according to the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. A computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. Semiconductor media can be solid-state drives.

[0121] Figure 8 This is a structural block diagram of an electronic device provided in an embodiment of this application. Figure 8 As shown, the electronic device 800 may include one or more of the following components: processor 801, memory 802 and communication interface 803. The processor 801, memory 802 and communication interface 803 are interconnected and perform communication between them. The memory 802 may store one or more computer programs, which may be configured to implement the methods described in the above embodiments when executed by one or more processors 801.

[0122] Processor 801 may include one or more processing cores. Processor 801 connects to various parts within the electronic device 800 using various interfaces and lines, and performs various functions and processes data of the electronic device 800 by running or executing instructions, programs, code sets, or instruction sets stored in memory 802, and by calling data stored in memory 802. Optionally, processor 801 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 801 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. It is understood that the aforementioned modem may also not be integrated into processor 801, but may be implemented separately through a communication chip.

[0123] The memory 802 may include random access memory (RAM) or read-only memory (ROM). The memory 802 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 802 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described above. The data storage area may also store data created by the electronic device 800 during use.

[0124] It is understood that the electronic device 800 may include more or fewer structural elements than those shown in the above block diagram, such as a power module, physical buttons, WiFi (Wireless Fidelity) module, speaker, Bluetooth module, sensor, etc., without limitation.

[0125] The aforementioned electronic device 800 may be a voice processing device or a part thereof.

[0126] This application provides a computer-readable storage medium storing program data, which, when executed by a processor, is used to perform some or all of the steps of any of the speech processing methods described in the above method embodiments.

[0127] This application also provides a computer program product, including a computer program operable to cause a computer to perform some or all of the steps of any of the speech processing methods described in the above method embodiments. The computer program product may be a software installation package.

[0128] It should be noted that, for the sake of simplicity, each of the aforementioned speech processing method embodiments is described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0129] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce a good effect.

[0130] Those skilled in the art will understand that all or part of the steps in the various method embodiments of any of the above-described voice processing methods can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0131] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of a speech processing method, apparatus, electronic device, and storage medium of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas of this application. At the same time, for those skilled in the art, based on the ideas of a speech processing method, apparatus, electronic device, and storage medium of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

[0132] This application is described with reference to flowchart illustrations and / or block diagrams of methods, hardware products, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0133] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0134] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0135] It is understood that any product that is controlled or configured to perform the processing method described in the flowchart of the method embodiment of the speech processing method of this application, such as the terminal and computer program product of the above flowchart, falls within the scope of the related products described in this application.

[0136] Obviously, those skilled in the art can make various modifications and variations to the voice processing method, apparatus, electronic device, and storage medium provided in this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

Claims

1. A speech processing method, characterized in that, The method includes: Determine the prompt words for the voice data; Based on the prompt words, generate the first generation probability of each candidate word in the candidate word library; Based on the first generation probability of each candidate word in the candidate word library, candidate words are selected from the candidate word library to obtain the first text; Determine the first similarity between the first text and the prompt word; The influence parameter of the first generation probability of each candidate word is adjusted based on the first similarity to determine the second generation probability of each candidate word; Based on the second generation probability of each candidate word, candidate words are selected from the candidate word library to obtain the second text; Generate the text corresponding to the speech data based on the first text and the second text.

2. The method according to claim 1, characterized in that, The prompt words used to determine the voice data include: The speech data is processed into text to obtain a third text, and each text segment in the third text includes a corresponding confidence level. A first confidence level lower than a preset confidence level is determined in the third text. The first text segment corresponding to the first confidence level is replaced with a second text segment to obtain a fourth text. The similarity between the second text segment and the third text is higher than the similarity between the first text segment and the third text. Keyword extraction is performed on the fourth text to determine the keywords corresponding to the fourth text; The prompt word is determined based on the keywords corresponding to the fourth text.

3. The method according to claim 1, characterized in that, The influencing parameters of the first generation probability of each candidate word include a temperature parameter and a cue word embedding weight. The step of adjusting the first generation probability of each candidate word based on the first similarity to determine the second generation probability of each candidate word includes: If the first similarity is less than the first threshold, the temperature parameter in the first generation probability of each candidate word is reduced, and / or the prompt word embedding weight in the first generation probability of each candidate word is increased to obtain the second generation probability of each candidate word; If the first similarity is greater than the second threshold, the temperature parameter in the first generation probability of each candidate word is increased, and / or the prompt word embedding weight in the first generation probability of each candidate word is decreased to obtain the second generation probability of each candidate word, wherein the second threshold is not less than the first threshold.

4. The method according to claim 1, characterized in that, After determining the first similarity between the first text and the prompt word, the method further includes: If the first similarity is less than the third threshold, obtain multiple candidate texts corresponding to the first text; The candidate text with the highest similarity to the prompt word among the plurality of candidate texts is taken as the content of the first text.

5. The method according to any one of claims 1-4, characterized in that, The step of selecting candidate words from the candidate word library based on the first generation probability of each candidate word in the candidate word library to obtain the first text includes: Based on the first generation probability of each candidate word in the candidate word library, candidate words are selected from the candidate word library to obtain the first candidate word and the second candidate word; The first generation probability of each candidate word is updated based on the first candidate word, and a candidate word is selected from the candidate word library according to the updated first generation probability of each candidate word to obtain a third candidate word, and a fifth text is determined based on the first candidate word and the third candidate word. The first generation probability of each candidate word is updated based on the second candidate word, and a candidate word is selected from the candidate word library according to the updated first generation probability of each candidate word to obtain a fourth candidate word, and a sixth text is determined based on the second candidate word and the fourth candidate word. The first text is determined from the fifth text and the sixth text.

6. The method according to claim 5, characterized in that, Determining the first text from the fifth text and the sixth text includes: Based on preset conditions, the first text is determined from the fifth text and the sixth text; The preset conditions include at least one of the following: The text most relevant to the target business scenario; The text that has the highest correlation with the speech features in the speech data; The text with the highest similarity to the aforementioned prompt word.

7. A voice processing device, characterized in that, The device includes: The first processing unit is used to determine the prompt words for the voice data; The second processing unit is used to generate a first generation probability for each candidate word in the candidate word library based on the prompt words; The third processing unit is used to select candidate words from the candidate word library according to the first generation probability of each candidate word in the candidate word library, and obtain the first text. The fourth processing unit is used to determine the first similarity between the first text and the prompt word; The second processing unit is further configured to adjust the influence parameter of the first generation probability of each candidate word based on the first similarity, and determine the second generation probability of each candidate word; The third processing unit is further configured to select candidate words from the candidate word library according to the second generation probability of each candidate word to obtain a second text, and to generate text corresponding to the speech data according to the first text and the second text.

8. An electronic device, the device comprising a processor, a memory, and executable program code stored in the memory, characterized in that, The processor is configured to retrieve the executable program code stored in the memory to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images