Voice data processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a voice data processing method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the feature extraction of to-be-processed voice data in response to the data processing of a voice data task, and obtaining a voice embedding vector; then, obtaining a preset bias list corresponding to the voice data task, and obtaining a preset phrase index dictionary based on the preset bias list; then, decoding and filtering the voice embedded vector to obtain a filtered text sequence containing at least one rare text vocabulary, and generating a rare vocabulary embedded vector based on a preset phrase index dictionary and the filtered text sequence; then, splicing the voice embedding vector and the rare vocabulary embedding vector to obtain a voice processing vector; and finally, inputting the voice processing vector into a large language processing model for data processing to obtain a target text sequence of the to-be-processed voice data, thereby effectively improving the accuracy of the context voice recognition.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a voice data processing method, device, electronic device and storage medium. Background Art

[0002] Contextual speech recognition processing refers to the use of real-time context signals to dynamically adjust the priorities in the pre-trained speech recognition system during the speech recognition process to improve the accuracy of the recognition results. Contextual speech recognition has important value in practical applications, such as in business systems where users need personalized recognition of specific names, places, and entities.

[0003] In the related art, when performing contextual speech recognition, a longer bias word list is usually added to guide the recognition results to better assist the contextual speech recognition process. However, as the accuracy of contextual speech recognition is improved, in order to cover more contextual scenarios, the number of bias words in the bias word list needs to be further increased, so that there are too many bias words in the contextual speech recognition, resulting in inaccurate actual recognition. Summary of the invention

[0004] The embodiments of the present application provide a voice data processing method, device, electronic device and storage medium, which can improve the accuracy of contextual voice recognition.

[0005] To achieve the above object, a first aspect of an embodiment of the present application provides a method for processing speech data, the method comprising: In response to data processing of the speech data task, feature extraction is performed on the speech data to be processed to obtain a speech embedding vector; Acquire a preset offset list corresponding to the voice data task, and obtain a preset phrase index dictionary based on the preset offset list; Decoding and filtering the speech embedding vector to obtain a filtered text sequence containing at least one rare text word, and generating a rare word embedding vector based on the preset phrase index dictionary and the filtered text sequence; Concatenating the speech embedding vector and the rare word embedding vector to obtain a speech processing vector; The speech processing vector is input into a large language processing model for data processing to obtain a target text sequence of the speech data to be processed.

[0006] In some embodiments, the decoding and filtering of the speech embedding vector to obtain a filtered text sequence containing at least one rare text word includes: Decoding the speech embedding vector to obtain a preliminary text sequence, wherein the preliminary text sequence includes a plurality of text words; Based on the common vocabulary data, common vocabulary filtering is performed on the plurality of text words to obtain the filtered text sequence.

[0007] In some embodiments, the rare text vocabulary is multiple, the preset phrase index dictionary includes multiple index text vocabulary, and the generating of the rare word embedding vector based on the preset phrase index dictionary and the filtered text sequence includes: Selecting a target rare text word from the plurality of rare text words one by one, calculating the edit distance between the target rare text word and each of the index text words, and selecting the index text word with the smallest edit distance from the plurality of index text words as the target index text word; The rare word embedding vector is generated based on all the target index text words.

[0008] In some embodiments, the step of extracting features from the speech data to be processed to obtain a speech embedding vector includes: Inputting the speech data to be processed into a speech feature extraction model for feature extraction to obtain a speech feature sequence; Downsampling the speech feature sequence to obtain a downsampled feature sequence; The down-sampled feature sequence is projected to obtain the speech embedding vector.

[0009] In some embodiments, the generation process of the large language processing model includes: Obtaining an initial large language processing model, a training speech sample, and a training rare vocabulary embedding vector corresponding to the training speech sample, wherein the initial large language processing model includes a projection layer; Calculating a cross entropy loss based on the training speech sample and the training rare vocabulary embedding vector, and updating a projection layer parameter of the projection layer based on the cross entropy loss; The large language processing model is obtained based on the updated initial large language processing model.

[0010] In some embodiments, before inputting the speech data to be processed into a speech feature extraction model for feature extraction to obtain a speech feature sequence, the method further includes: Obtaining an initial speech feature extraction model and training speech samples; Inputting the training speech sample into the initial speech feature extraction model to extract features, thereby obtaining a training speech feature sequence; Calculating a feature loss value based on the training speech feature sequence, and fine-tuning the initial speech feature extraction model based on the feature loss value; The speech feature extraction model is obtained based on the updated initial speech feature extraction model.

[0011] In some embodiments, the preset offset list includes at least one segmentation character, and obtaining a preset phrase index dictionary based on the preset offset list includes: Generate a 2-tuple corresponding to each of the segmentation characters; The preset phrase index dictionary is generated based on all the bigrams.

[0012] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a speech data processing device, the device comprising: A feature extraction module, for extracting features of the speech data to be processed in response to data processing of the speech data task, and obtaining a speech embedding vector; An index dictionary generation module, used to obtain the preset offset list corresponding to the voice data task, and obtain a preset phrase index dictionary based on the preset offset list; A rare embedding vector generation module, configured to perform decoding and filtering processing on the speech embedding vector to obtain a filtered text sequence containing at least one rare text word, and to generate the rare word embedding vector based on the preset phrase index dictionary and the filtered text sequence; An embedding vector concatenation module, used for concatenating the speech embedding vector and the rare word embedding vector to obtain a speech processing vector; The data processing module is used to input the speech processing vector into a large language processing model for data processing to obtain a target text sequence of the speech data to be processed.

[0013] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the voice data processing method described in the first aspect when executing the computer program.

[0014] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program. When the computer program is executed by a processor, the voice data processing method described in the first aspect is implemented.

[0015] The speech data processing method, device, electronic device and storage medium proposed in the embodiments of the present application include: first, in response to data processing of a speech data task, feature extraction is performed on the speech data to be processed to obtain a speech embedding vector; then, a preset bias list corresponding to the speech data task is obtained, and a preset phrase index dictionary is obtained based on the preset bias list; thereafter, the speech embedding vector is decoded and filtered to obtain a filtered text sequence containing at least one rare text word, and a rare word embedding vector is generated based on the preset phrase index dictionary and the filtered text sequence; then, the speech embedding vector and the rare word embedding vector are concatenated to obtain a speech processing vector; finally, the speech processing vector is input into a large language processing model for data processing to obtain a target text sequence of the speech data to be processed. When performing contextual speech recognition corresponding to a speech data task, the embodiment of the present application uses a preset phrase index dictionary corresponding to the speech data task as initial prompt information to assist in the execution of the contextual speech recognition task. At the same time, common words in the speech embedding vector are filtered to generate a filtered text sequence containing only rare text words, so as to increase the focus on rare words in the speech data task. The rare word embedding vector based on the association between the filtered text sequence and the preset phrase index dictionary ensures that it is closely related to the speech data task while avoiding the need to generate too many bias words. Therefore, when the rare word embedding vector is used to assist the speech data task in performing contextual speech recognition, the accuracy of the contextual speech recognition is effectively improved.

[0016] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a structural diagram of a voice data processing system provided in one embodiment of the present application.

[0018] Figure 2 This is a flowchart of a voice data processing method provided by another embodiment of the present application.

[0019] Figure 3 yes Figure 2 Flow chart of step 201 in FIG.

[0020] Figure 4 yes Figure 2 Flow chart of step 202 in FIG.

[0021] Figure 5 yes Figure 2 Flow chart of step 203 in FIG.

[0022] Figure 6 yes Figure 2 Another flow chart of step 203 in FIG.

[0023] Figure 7 This is a schematic diagram of a rare word embedding vector generation process provided by another embodiment of the present application.

[0024] Figure 8 This is a flowchart of a fine-tuning speech feature extraction model provided by another embodiment of the present application.

[0025] Fig. 9 This is a flowchart of generating a large language processing model provided by another embodiment of the present application.

[0026] Fig.10 It is a structural diagram of a voice data processing device provided in one embodiment of the present application.

[0027] Fig.11 It is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0029] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0031] Contextual speech recognition processing refers to the use of real-time context signals to dynamically adjust the priorities in the pre-trained speech recognition system during the speech recognition process to improve the accuracy of the recognition results. Contextual speech recognition has important value in practical applications, such as in business systems where users need personalized recognition of specific names, places, and entities.

[0032] In the related art, when performing contextual speech recognition, a longer bias word list is usually added to guide the recognition results to better assist the contextual speech recognition process. However, as the accuracy of contextual speech recognition is improved, in order to cover more contextual scenarios, the number of bias words in the bias word list needs to be further increased, so that there are too many bias words in the contextual speech recognition, resulting in inaccurate actual recognition.

[0033] In order to improve the accuracy of contextual speech recognition, when performing contextual speech recognition corresponding to a speech data task, the embodiment of the present application uses a preset phrase index dictionary corresponding to the speech data task as initial prompt information to assist in the execution of the contextual speech recognition task, and at the same time filters common words in the speech embedding vector to generate a filtered text sequence containing only rare text words, so as to increase the attention to rare words in the speech data task, and based on the rare word embedding vector of the association between the filtered text sequence and the preset phrase index dictionary, it is ensured that it is closely related to the speech data task without generating too many bias words, thereby effectively improving the accuracy of the contextual speech recognition when using the rare word embedding vector to assist the speech data task in performing contextual speech recognition.

[0034] In order to better describe the voice data processing method provided by the present application, the following first describes a voice data processing system applied to the voice data processing method. Figure 1 , is a structural diagram of a voice data processing system provided in an embodiment of the present application. Figure 1 As shown in , the speech data processing system includes a speech feature extraction module, a large language segmentation module and a large language processing model. After receiving the speech data to be processed, the speech data is first input into the speech feature extraction model to obtain a speech feature sequence, and further obtain a speech embedding vector, and then the speech embedding vector is input into the large language segmentation model to obtain a rare word embedding vector. Then, the speech embedding vector and the rare word embedding vector are concatenated and input into the large language processing model together to obtain a target text sequence corresponding to the speech data to be processed to complete the sub-context speech recognition task.

[0035] In this embodiment, the SSL model WavLM Large is used as the speech feature extraction model, which is a large-scale self-supervised pre-training model specifically for full-stack speech processing tasks. Vicuna 7B is used as the decoder of the large language processing model. It is an autoregressive language model based on the Transformer architecture. It is trained using a large amount of unlabeled data and transfer learning technology to enable it to have powerful language understanding capabilities. In the large language processing model, a lightweight, simple linear projection layer is set to align the extracted speech features with the LLM input space. The projection layer first downsamples the 50Hz speech feature sequence to 10Hz through a one-dimensional convolutional layer, and then projects it through two linear layers, with a hidden feature dimension of 2048.

[0036] In conjunction with the above-mentioned voice data processing system, the voice data processing method, device, electronic device and storage medium provided in the embodiments of the present application will be further described below. The voice data processing method provided in the embodiments of the present application can be applied to a client.

[0037] The following is a detailed description of the voice data processing method in the embodiment of the present application. Figure 2 , is an optional flow chart of the voice data processing method provided in the embodiment of the present application, Figure 2 The method in the embodiment may include but is not limited to steps 201 to 205. Figure 2 The order of step 201 to step 205 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0038] Step 201: In response to data processing of the speech data task, feature extraction is performed on the speech data to be processed to obtain a speech embedding vector.

[0039] The following is a detailed description of step 201.

[0040] In some embodiments, when responding to data processing of a speech data task, in order to fully analyze the context analysis of the speech data task, it is necessary to first perform feature extraction on the speech data to be processed corresponding to the speech data task to obtain a suitable speech embedding vector for describing the speech data to be processed, as described below.

[0041] Reference Figure 3 , feature extraction is performed on the speech data to be processed to obtain a speech embedding vector, including the following steps 301 to 303.

[0042] Step 301: Input the speech data to be processed into a speech feature extraction model for feature extraction to obtain a speech feature sequence.

[0043] Step 302: down-sample the speech feature sequence to obtain a down-sampled feature sequence.

[0044] Step 303: Project the downsampled feature sequence to obtain a speech embedding vector.

[0045] Steps 301 to 303 are described in detail below.

[0046] In some embodiments, the speech data to be processed is first input into a speech feature extraction model (MacLM encoder) for feature extraction to obtain a speech feature sequence corresponding to the speech data to be processed. For example, the 16kHz speech data to be processed is input into a WavLM encoder to generate a 50Hz speech feature sequence with a dimension of 1024.

[0047] Next, in the large language processing model, the speech feature data is downsampled using a one-dimensional convolutional layer to obtain a downsampled feature sequence with a reduced sampling frequency, such as reducing the sampling frequency of a 50 Hz speech feature sequence to 10 Hz to generate a downsampled feature sequence.

[0048] Finally, in the large language processing module, the downsampled feature sequence is mapped to a higher dimension through a linear projection layer, such as mapping the dimension of the 10Hz downsampled feature sequence to 1024, so as to generate a speech embedding vector for appropriately representing the speech data to be processed.

[0049] Through the above steps 301 to 303, a speech feature extraction model is used to obtain a speech feature sequence for appropriately describing the speech data to be processed, and then the convolution layer and projection layer in the large language processing model are used to downsample and project the speech feature sequence. The speech feature sequence can be converted into low-frequency, high-dimensional data that is more suitable for input into the large language processing model for processing. At the same time, the computational complexity and memory requirements can be reduced to improve the efficiency and accuracy of speech data processing.

[0050] Step 202: Obtain a preset offset list corresponding to the voice data task, and obtain a preset phrase index dictionary based on the preset offset list.

[0051] The following is a detailed description of step 202.

[0052] In some embodiments, in order to improve the accuracy of voice data processing tasks, while obtaining the voice embedding vector, it is also necessary to obtain a preset bias list corresponding to the voice data task, so as to use the preset bias list to generate a preset dictionary index dictionary for assisting in guiding the voice embedding vector for voice data processing.

[0053] For the Contextual ASR task, a bias list is added to the prompt to help large language processing models accurately identify rare words that are often misrecognized.

[0054] It is understandable that the preset bias list can be carried by the input voice data task, or it can be a pre-built bias list database, and then matched according to the generated voice embedding vector, which is closely related to the voice data task. The preset bias list can contain named entities or rare words, such as a contact list or a song playlist, etc.

[0055] It is understandable that during the model training process, the training bias list is randomly generated from the transcripts of the current training batch of speech. Specifically, each text has a probability Pkeep to be used to extract the bias vocabulary. For each retained text, k n-gram phrases are randomly selected from it, where k is uniformly sampled from [1, Nphrases] and n is uniformly sampled from [1, Norder], where Nphrases and Norder are hyperparameters that control the number and length of n-gram phrases. The test bias list consists of predefined bias lists containing hundreds to thousands of words, and hot words filtered out by the filtering algorithm.

[0056] How to generate a preset phrase index dictionary based on the preset offset list will be further described below.

[0057] Reference Figure 4 , obtaining a preset phrase index dictionary based on a preset offset list includes the following steps 401 to 402.

[0058] Step 401: Generate a bigram corresponding to each segmentation character.

[0059] Step 402: Generate a preset phrase index dictionary based on all bigrams.

[0060] Steps 401 to 402 are described in detail below.

[0061] Generally speaking, the preset offset list contains multiple segmentation characters, such as [Bob, Joe], and then generates a bigram corresponding to each segmentation character, such as {'bo': 'bob', 'ob': 'bob'} corresponding to 'Bob', and {'jo': 'joe', 'oe': 'joe'} corresponding to 'Joe'. Then, a preset phrase index dictionary (i.e., 2-gram phrase index dictionary) is generated based on all bigrams, such as ['bo': 'bob', 'ob': 'bob', 'jo': 'joe', 'oe': 'joe'] corresponding to [Bob, Joe].

[0062] Through the above steps 401 to 402, using the bigrams corresponding to each segmentation character in the preset bias list, a preset phrase index dictionary can be generated to better assist in understanding the collocation relationship of the segmentation characters, thereby facilitating the use of the preset phrase index dictionary to generate rare vocabulary embedding vectors that are more suitable for processing the speech data task.

[0063] Step 203: Decode and filter the speech embedding vector to obtain a filtered text sequence containing at least one rare text word, and generate a rare word embedding vector based on a preset phrase index dictionary and the filtered text sequence.

[0064] Step 203 is described in detail below.

[0065] In some embodiments, after obtaining the speech embedding vector and the preset phrase index dictionary, in order to avoid interference caused by too many popular words in the speech embedding vector in the processing of the speech data task, it is necessary to pre-filter the popular words in the speech embedding vector through a filtering mechanism to obtain a filtered text sequence containing only rare text words, as described below.

[0066] Reference Figure 5 , the speech embedding vector is decoded and filtered to obtain a filtered text sequence containing at least one rare text word, including the following steps 501 to 502.

[0067] Step 501: Decode the speech embedding vector to obtain a preliminary text sequence.

[0068] Step 502: Based on the common vocabulary data, a plurality of text words are filtered for common vocabulary to obtain a filtered text sequence.

[0069] Steps 501 to 502 are described in detail below.

[0070] In some embodiments, after obtaining the speech embedding vector, the speech embedding vector is input into the speech feature extraction model (WavLM Large model) again and decoded by the Viterbi algorithm to obtain a rough decoding result, that is, a preliminary text sequence containing multiple text words.

[0071] Next, based on the common vocabulary data, common vocabulary is filtered out from the multiple text vocabulary in the preliminary text sequence to obtain a filtered text sequence containing only rare text vocabulary. The common vocabulary data may be the 5000 most frequently appearing words in the Librispeech training corpus.

[0072] Through the above steps 501 to 502, multiple text words in the preliminary text sequence are filtered for common words to obtain a filtered text sequence that only contains rare text words, so as to avoid being interfered by common words, thus facilitating the subsequent generation of rare word embedding vectors for assisting in processing this voice data task by combining the filtered text sequence with a preset phrase index dictionary.

[0073] After obtaining the filtered text sequence that only contains rare text words, further generate rare word embedding vectors based on the preset phrase index dictionary and the filtered text sequence, which is described in detail as follows.

[0074] Refer to Figure 6 , generate rare word embedding vectors based on the preset phrase index dictionary and the filtered text sequence, including the following steps 601 to 602.

[0075] Step 601: Select a target rare text word one by one from multiple rare text words, calculate the edit distance between the target rare text word and each index text word, and select the index text word with the smallest edit distance from multiple index text words as the target index text word.

[0076] Step 602: Generate rare word embedding vectors based on all target index text words.

[0077] The following details steps 601 to 602.

[0078] In some embodiments, after obtaining the filtered text sequence and the preset phrase index dictionary, select a rare text word one by one from multiple rare text words in the filtered text sequence as the target rare text word, and then for each target rare text word, calculate its edit distance from each index text word in the preset phrase index dictionary, so as to select the index text word with the smallest edit distance from multiple index text words as the target index text word corresponding to the target rare text word.

[0079] Among them, calculating the edit distance between the target rare text word and each index text word in the preset phrase index dictionary means calculating the minimum number of single-character edit operations required to convert the target rare text word into each index text word, including insertion, deletion, and substitution. For example, if the target rare text word is "Beijin", and the index text words include {"Beijinbei" and "Beiduofen"}, then only one 'Be' needs to be deleted between "Beijin" and "Beijinbei", so its edit distance is 1; while between "Beijin" and "Beiduofen", 'duo' needs to be replaced with 'jin', and 'fen' needs to be deleted, so its edit distance is 3. Therefore, select "Beijinbei" as the target index text word corresponding to "Beijin".

[0080] After selecting the target index text vocabulary corresponding to each target rare text vocabulary one by one, all the target index text vocabulary are combined to obtain the rare vocabulary embedding vector prompt.

[0081] Through the above steps 601 to 602, the edit distance between the rare text vocabulary and multiple index text vocabulary is used to evaluate the similarity between the rare text vocabulary and multiple index text vocabulary, so as to select the index text vocabulary with the highest similarity to the rare text vocabulary to generate a rare vocabulary embedding vector, so that the rare vocabulary embedding vector is closer to the rare vocabulary part in the speech embedding vector corresponding to the current speech data task, thereby improving the accuracy of speech data processing.

[0082] Reference Figure 7 , is a schematic diagram of a rare word embedding vector generation process provided by an embodiment of the present application. Figure 7 As shown in , the input voice data to be processed is "hello bertie any good in your mind", and the preliminary text sequence obtained by decoding is "hello bertye any good in your mind", and the preset bias list corresponding to the voice data task is "achiet, anty's, ..., bertie, bords, ..., hello, ..."; next, a preset phrase index dictionary is generated based on the preset bias list as "'ae': {'micaceous', 'vacates', 'achiet'}, 'ch': {'muteh', 'achiet'}, 'hi': {'pethick', 'achiet}"; further, common vocabulary filtering is performed on multiple text words in the preliminary text sequence, that is, "anygood in your mind" in "hello bertye any good in your mind" is removed, leaving the filtered text sequence "hello bertye"; Next, the edit distance between each rare text word in the filtered text sequence and each index text word in the index dictionary is calculated to obtain the corresponding matching similar words (i.e., the target index text words) to generate the rare word embedding vector prompt{'hello', 'bertie'}.

[0083] Step 204: concatenate the speech embedding vector and the rare word embedding vector to obtain a speech processing vector.

[0084] Step 205: Input the speech processing vector into the large language processing model for data processing to obtain a target text sequence of the speech data to be processed.

[0085] Steps 204 to 205 are described in detail below.

[0086] In some embodiments, after obtaining the speech embedding vector and the rare word embedding vector prompt, the two text vectors are concatenated to obtain a speech processing vector with a unified representation, and then the speech processing vector is input into a large language processing model for decoding to obtain a target text sequence corresponding to the task data to be processed.

[0087] In order to further improve the accuracy of speech data processing, it is necessary to pre-train the speech data processing system (including speech feature extraction model and large language processing model), as described below.

[0088] Reference Figure 8 Before the speech data to be processed is input into the speech feature extraction model for feature extraction to obtain a speech feature sequence, the speech data processing method further includes the following steps 801 to 804.

[0089] Step 801: Obtain an initial speech feature extraction model and training speech samples.

[0090] Step 802: Input the training speech sample into the initial speech feature extraction model to extract features, and obtain a training speech feature sequence.

[0091] Step 803: Calculate a feature loss value based on the training speech feature sequence, and fine-tune the initial speech feature extraction model based on the feature loss value.

[0092] Step 804: Obtain a speech feature extraction model based on the updated initial speech feature extraction model.

[0093] Steps 801 to 804 are described in detail below.

[0094] In some embodiments, in the process of extracting features from the speech data to be processed in the context speech recognition processing task to obtain the corresponding speech feature sequence, the speech feature extraction model used is trained. It is necessary to pre-generate an initial speech feature extraction model (such as the SSL model WavLM Large) and a large number of training speech samples (such as speech data in the Librispeech database), and then input these training speech samples into the initial speech feature extraction model for feature extraction to obtain a training speech feature sequence that captures key information in the training speech samples (such as Mel spectrum features). Then, using the obtained training speech feature sequence and the label corresponding to the training speech sample (such as text transcription), calculate the feature loss value between the feature output by the initial speech feature extraction model and the target feature, which is usually achieved by defining a loss function, such as mean square error (MSE) or cross entropy loss. Then, based on the calculated feature loss value, use gradient descent or other optimization algorithms to adjust the model parameters to minimize the loss function. Finally, the initial speech feature extraction model updated after multiple trainings is used as the speech feature extraction model in practical applications.

[0095] In some embodiments, the speech feature extraction model used in this solution is obtained by training with 960 hours of Librispeech data.

[0096] Reference Fig. 9 , the generation process of the large language processing model includes the following steps 901 to 903.

[0097] Step 901: Obtain an initial large language processing model, a training speech sample, and a training rare word embedding vector corresponding to the training speech sample.

[0098] Step 902: Calculate a cross entropy loss based on the training speech samples and the training rare vocabulary embedding vectors, and update the projection layer parameters of the projection layer based on the cross entropy loss.

[0099] Step 903: Obtain a large language processing model based on the updated initial large language processing model.

[0100] Steps 901 to 903 are described in detail below.

[0101] In some embodiments, in the process of performing data processing on the speech processing vector in the context speech recognition processing task to obtain the target text sequence corresponding to the speech data task, in the training process of the large language processing model used, only the lightweight projection layer is updated, and the rest of the model parts remain frozen. Specifically, as described below, an initial large language processing model including a lightweight projection layer, a training speech sample, and a training rare vocabulary embedding vector corresponding to the training speech sample are pre-generated. Then, the training speech sample and the corresponding training rare vocabulary embedding vector are concatenated to input the initial large speech processing model to obtain the corresponding training text sequence, and then the cross entropy loss between the predicted output training text sequence and the true label corresponding to the training speech sample is calculated. Next, according to the calculated cross entropy loss, the parameters of the projection layer are adjusted using gradient descent or other optimization algorithms. Finally, the initial large language processing model updated after multiple trainings is used as the large language processing model in practical applications.

[0102] Through the above steps 801 to 804, and steps 901 to 903, the initial speech feature extraction model and the initial large language processing model are trained multiple times using training speech samples, thereby effectively improving the accuracy in actual speech data task processing.

[0103] In order to further verify the reliability of the voice data processing method provided in the present application, this embodiment performs simulation verification on it, as described in detail as follows.

[0104] The embodiment of the present application tests the performance of the proposed speech data processing system on the Librispeech dataset. WER (word error rate), B-WER (biased word error rate) and U-WER (unbiased word error rate) are used as evaluation indicators. B-WER only calculates the vocabulary in the preset bias list, while U-WER only calculates the vocabulary that is not in the preset bias list.

[0105] For the speech data processing system, when the size of the preset bias list is 100, the speech data processing system performs best, with WER / B-WER of the test-clean set being 1.27% / 3.67% and test-other set being 2.72% / 8.02%, respectively, which is a significant relative decrease of 39.81% / 63.37% and 35.24% / 61.37% compared to the baseline model, while U-WER remains basically unchanged. This demonstrates the effectiveness of the speech data processing method proposed in this application and the efficient use of the rare word embedding vector information in the prompt by the speech data processing system. As the size of the bias list increases, WER and B-WER increase slightly. However, even when the bias list size is increased to 2000, the WER / B-WER on the test-clean and test-other sets are 1.38% / 4.41% and 3.20% / 10.02%, respectively, which are still significant relative decreases of 34.60% / 55.99% and 23.81% / 51.73% respectively compared with the baseline model, which can effectively verify the effectiveness and accuracy of the speech data processing method proposed in this application.

[0106] The speech data processing method, device, electronic device and storage medium proposed in the embodiments of the present application include: first, in response to data processing of a speech data task, inputting the speech data to be processed into a speech feature extraction model for feature extraction to obtain a speech feature sequence, downsampling the speech feature sequence to obtain a downsampled feature sequence, projecting the downsampled feature sequence to obtain a speech embedding vector; then, obtaining a preset bias list corresponding to the speech data task, generating a bigram corresponding to each segmented character, and generating a preset phrase index dictionary based on all the bigrams; then, decoding the speech embedding vector to obtain a preliminary text sequence, the preliminary text sequence including Multiple text words, based on common word data, multiple text words are filtered by common words to obtain a filtered text sequence, and target rare text words are selected from multiple rare text words one by one, the edit distance between the target rare text word and each index text word is calculated, and the index text word with the smallest edit distance is selected from multiple index text words as the target index text word, and a rare word embedding vector is generated based on all target index text words; then, the speech embedding vector and the rare word embedding vector are concatenated to obtain a speech processing vector; finally, the speech processing vector is input into a large language processing model for data processing to obtain a target text sequence of the speech data to be processed.

[0107] When performing contextual speech recognition corresponding to a speech data task, the embodiment of the present application uses a speech feature extraction model to obtain a speech feature sequence for appropriately describing the speech data to be processed, and then uses the convolution layer and projection layer in the large language processing model to downsample and project the speech feature sequence, so that the speech feature sequence can be converted into low-frequency, high-dimensional data that is more suitable for input into the large language processing model for processing, while reducing computational complexity and memory requirements to improve the efficiency and accuracy of speech data processing; and, using the bigram corresponding to each segmentation character in the preset bias list, a preset phrase index dictionary can be generated to better assist in understanding the collocation relationship of the segmentation characters, thereby facilitating the use of the preset phrase index dictionary to generate a rare vocabulary embedding vector that is more suitable for processing the speech data task; then, multiple text words in the preliminary text sequence are filtered for common words to obtain a filtered text sequence containing only rare text words to avoid interference from common words, thereby facilitating the subsequent use of the filtered text sequence in combination with the preset phrase index dictionary to generate a rare vocabulary embedding vector for assisting in processing this speech data task vector; and, using the edit distance between rare text words and multiple index text words, the similarity between the rare text words and multiple index text words is evaluated, so as to select the index text words with the highest similarity to the rare text words to generate the rare word embedding vector, so that the rare word embedding vector is closer to the rare word part in the speech embedding vector corresponding to this speech data task, thereby improving the accuracy of speech data processing; in short, the preset phrase index dictionary corresponding to the speech data task is used as the initial prompt information to assist the execution of the task of contextual speech recognition, and at the same time, the common words in the speech embedding vector are filtered to generate a filtered text sequence containing only rare text words, so as to increase the attention to the rare words in the speech data task, and the rare word embedding vector based on the association between the filtered text sequence and the preset phrase index dictionary ensures that it is closely related to the speech data task without generating too many bias words, so as to effectively improve the accuracy of the contextual speech recognition when the rare word embedding vector is used to assist the contextual speech recognition of this speech data task.

[0108] The present application also provides a voice data processing device that can implement the above-mentioned voice data processing method. Fig.10 , the device 1000 comprises: A feature extraction module 1010 is used to extract features of the speech data to be processed in response to data processing of the speech data task, and obtain a speech embedding vector; An index dictionary generation module 1020, used to obtain a preset offset list corresponding to the voice data task, and obtain a preset phrase index dictionary based on the preset offset list; A rare embedding vector generating module 1030 is used to perform decoding and filtering processing on the speech embedding vector to obtain a filtered text sequence containing at least one rare text word, and generate a rare word embedding vector based on a preset phrase index dictionary and the filtered text sequence; An embedding vector concatenation module 1040 is used to concatenate the speech embedding vector and the rare word embedding vector to obtain a speech processing vector; The data processing module 1050 is used to input the speech processing vector into the large language processing model for data processing to obtain a target text sequence of the speech data to be processed.

[0109] In some embodiments, the rare embedding vector generation module 1030 is further configured to: Decoding the speech embedding vector to obtain a preliminary text sequence, where the preliminary text sequence includes multiple text words; Based on the common vocabulary data, common vocabulary filtering is performed on multiple text words to obtain a filtered text sequence.

[0110] In some embodiments, the rare embedding vector generation module 1030 is further configured to: Selecting target rare text words from multiple rare text words one by one, calculating the edit distance between the target rare text word and each index text word, and selecting the index text word with the smallest edit distance from the multiple index text words as the target index text word; Generate rare word embedding vectors based on all target index text words.

[0111] In some embodiments, the feature extraction module 1010 is further configured to: Inputting the speech data to be processed into the speech feature extraction model to extract features and obtain a speech feature sequence; Down-sampling the speech feature sequence to obtain a down-sampled feature sequence; The downsampled feature sequence is projected to obtain the speech embedding vector.

[0112] In some embodiments, the data processing module 1050 is further configured to: Obtaining an initial large language processing model, a training speech sample, and a training rare word embedding vector corresponding to the training speech sample, wherein the initial large language processing model includes a projection layer; Calculate the cross entropy loss based on the training speech samples and the training rare word embedding vectors, and update the projection layer parameters of the projection layer based on the cross entropy loss; A large language processing model is obtained based on the updated initial large language processing model.

[0113] In some embodiments, the data processing module 1050 is further configured to: Obtaining an initial speech feature extraction model and training speech samples; Input the training speech sample into the initial speech feature extraction model to extract features, and obtain a training speech feature sequence; Calculating a feature loss value based on the training speech feature sequence, and fine-tuning an initial speech feature extraction model based on the feature loss value; A speech feature extraction model is obtained based on the updated initial speech feature extraction model.

[0114] In some embodiments, the index dictionary generation module 1020 is further configured to: Generate a pair of tuples corresponding to each segmentation character; Generate a preset phrase index dictionary based on all bigrams.

[0115] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, the specific implementation of the voice data processing device is basically the same as the specific implementation of the above voice data processing method, and will not be repeated here.

[0116] In an embodiment of the present application, when performing contextual speech recognition corresponding to a speech data task, the speech data processing device uses a speech feature extraction model to obtain a speech feature sequence for appropriately describing the speech data to be processed, and then uses the convolution layer and projection layer in the large language processing model to downsample and project the speech feature sequence, so that the speech feature sequence can be converted into low-frequency, high-dimensional data that is more suitable for input into the large language processing model for processing, and at the same time can reduce the computational complexity and memory requirements to improve the efficiency and accuracy of speech data processing; and, using the bigram corresponding to each segmentation character in the preset bias list, a preset phrase index dictionary can be generated to better assist in understanding the collocation relationship of the segmentation characters, thereby facilitating the use of the preset phrase index dictionary to generate a rare vocabulary embedding vector that is more suitable for processing the speech data task; then, multiple text words in the preliminary text sequence are filtered for common words to obtain a filtered text sequence containing only rare text words to avoid interference from common words, thereby facilitating the subsequent use of the filtered text sequence in combination with the preset phrase index dictionary to generate a rare word embedding vector for assisting in processing this speech data task. See word embedding vector; and, using the edit distance between rare text words and multiple index text words, perform similarity evaluation between rare text words and multiple index text words, so as to select the index text words with the highest similarity to the rare text words to generate rare word embedding vectors, so that the rare word embedding vectors are closer to the rare word part in the speech embedding vector corresponding to this speech data task, thereby improving the accuracy of speech data processing; in short, use the preset phrase index dictionary corresponding to the speech data task as initial prompt information to assist the execution of the task of contextual speech recognition, and at the same time filter the common words in the speech embedding vector to generate a filtered text sequence containing only rare text words, so as to increase the attention to the rare words in the speech data task, and the rare word embedding vector based on the association between the filtered text sequence and the preset phrase index dictionary ensures that it is closely related to the speech data task without generating too many bias words, so that when the rare word embedding vector is used to assist the contextual speech recognition of this speech data task, the accuracy of this contextual speech recognition is effectively improved.

[0117] The present application also provides an electronic device, including: at least one memory; at least one processor; at least one program; The program is stored in the memory, and the processor executes the at least one program to implement the above-mentioned voice data processing method implemented in this application. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0118] See also Fig.11 , Fig.11 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes: The processor 1101 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application; The memory 1102 can be implemented in the form of ROM (Read Only Memory), static storage device, dynamic storage device or RAM (Random Access Memory). The memory 1102 can store operating systems and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1102, and the processor 1101 is used to call and execute the voice data processing method of the embodiments of this application; Input / output interface 1103, used to implement information input and output; The communication interface 1104 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.); A bus 1105 that transmits information between various components of the device (e.g., the processor 1101, the memory 1102, the input / output interface 1103, and the communication interface 1104); The processor 1101 , the memory 1102 , the input / output interface 1103 and the communication interface 1104 are connected to each other in communication within the device via the bus 1105 .

[0119] An embodiment of the present application further provides a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, the above-mentioned voice data processing method is implemented.

[0120] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0121] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0122] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0123] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0124] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0125] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0126] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0127] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0128] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0129] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0130] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0131] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for processing speech data, characterized in that: The method comprises: In response to data processing of the speech data task, feature extraction is performed on the speech data to be processed to obtain a speech embedding vector; Acquire a preset offset list corresponding to the voice data task, and obtain a preset phrase index dictionary based on the preset offset list; Decoding and filtering the speech embedding vector to obtain a filtered text sequence containing at least one rare text word, and generating a rare word embedding vector based on the preset phrase index dictionary and the filtered text sequence; Concatenating the speech embedding vector and the rare word embedding vector to obtain a speech processing vector; The speech processing vector is input into a large language processing model for data processing to obtain a target text sequence of the speech data to be processed.

2. The method for processing speech data according to claim 1, characterized in that: The decoding and filtering of the speech embedding vector to obtain a filtered text sequence containing at least one rare text word comprises: Decoding the speech embedding vector to obtain a preliminary text sequence, wherein the preliminary text sequence includes a plurality of text words; Based on the common vocabulary data, common vocabulary filtering is performed on the plurality of text words to obtain the filtered text sequence.

3. The method for processing speech data according to claim 1, characterized in that: There are multiple rare text words, the preset phrase index dictionary includes multiple index text words, and generating a rare word embedding vector based on the preset phrase index dictionary and the filtered text sequence includes: Selecting a target rare text word from the plurality of rare text words one by one, calculating the edit distance between the target rare text word and each of the index text words, and selecting the index text word with the smallest edit distance from the plurality of index text words as the target index text word; The rare word embedding vector is generated based on all the target index text words.

4. The method for processing speech data according to claim 1, characterized in that: The feature extraction of the speech data to be processed to obtain a speech embedding vector includes: Inputting the speech data to be processed into a speech feature extraction model for feature extraction to obtain a speech feature sequence; Downsampling the speech feature sequence to obtain a downsampled feature sequence; The down-sampled feature sequence is projected to obtain the speech embedding vector.

5. The method for processing speech data according to claim 4, characterized in that: The generation process of the large language processing model includes: Obtaining an initial large language processing model, a training speech sample, and a training rare vocabulary embedding vector corresponding to the training speech sample, wherein the initial large language processing model includes a projection layer; Calculating a cross entropy loss based on the training speech sample and the training rare vocabulary embedding vector, and updating a projection layer parameter of the projection layer based on the cross entropy loss; The large language processing model is obtained based on the updated initial large language processing model.

6. The method for processing speech data according to claim 4, characterized in that: Before inputting the speech data to be processed into a speech feature extraction model for feature extraction to obtain a speech feature sequence, the method further includes: Obtaining an initial speech feature extraction model and training speech samples; Inputting the training speech sample into the initial speech feature extraction model to extract features, thereby obtaining a training speech feature sequence; Calculating a feature loss value based on the training speech feature sequence, and fine-tuning the initial speech feature extraction model based on the feature loss value; The speech feature extraction model is obtained based on the updated initial speech feature extraction model.

7. The method for processing speech data according to claim 1, characterized in that: The preset offset list includes at least one segmentation character, and obtaining a preset phrase index dictionary based on the preset offset list includes: Generate a 2-tuple corresponding to each of the segmentation characters; The preset phrase index dictionary is generated based on all the bigrams.

8. A voice data processing device, characterized in that: The device comprises: A feature extraction module, for extracting features of the speech data to be processed in response to data processing of the speech data task, and obtaining a speech embedding vector; An index dictionary generation module, used to obtain the preset offset list corresponding to the voice data task, and obtain a preset phrase index dictionary based on the preset offset list; A rare embedding vector generation module, configured to perform decoding and filtering processing on the speech embedding vector to obtain a filtered text sequence containing at least one rare text word, and to generate the rare word embedding vector based on the preset phrase index dictionary and the filtered text sequence; An embedding vector concatenation module, used for concatenating the speech embedding vector and the rare word embedding vector to obtain a speech processing vector; The data processing module is used to input the speech processing vector into a large language processing model for data processing to obtain a target text sequence of the speech data to be processed.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the speech data processing method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speech data processing method according to any one of claims 1 to 7 is implemented.

Images